Multi-model selection for neural network-based tools in video coding

A neural network-based video coding system with hierarchical loop filtering and inter-prediction enhances compression efficiency by addressing complex motion in video data, reducing bandwidth and storage needs through selective model application.

JP7580874B2Active Publication Date: 2024-11-12TENCENT AMERICA LLC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2022560385
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-07-06
Filing Date
2021-08-23
Publication Date
2024-11-12
Estimated Expiration
2041-08-23

AI Technical Summary

Technical Problem

Traditional video coding standards struggle with efficiently compressing video data due to complex motion in non-translational scenes, leading to inefficiencies in bandwidth and storage requirements, and existing neural network-based methods are prone to errors, especially at object boundaries.

Method used

Implementing a neural network-based video coding system that utilizes a hierarchical temporal structure for loop filtering and inter-prediction, allowing for the generation of virtual reference frames and selective application of neural network models based on frame conditions, such as quantization parameters, hierarchy levels, and reference picture lists, to enhance compression efficiency.

Benefits of technology

The system achieves improved compression quality and reduced computational time by leveraging neural networks to handle complex motions and reduce redundancy in video data, offering better trade-offs in performance and execution time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007580874000012
    Figure 0007580874000012
  • Figure 0007580874000013
    Figure 0007580874000013
  • Figure 0007580874000014
    Figure 0007580874000014
Patent Text Reader

Abstract

A method, a computer program product, and a computer system for video coding and decoding are provided. Video data including one or more frames are received. One or more quantization parameters associated with the received video data are determined for frame generation or enhancement of a target frame from among the one or more frames. The video data is decoded based on the determined quantization parameters.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 136,062 (filed January 11, 2021) and U.S. Patent Application No. 17 / 368,069 (filed July 6, 2021) in the U.S. Patent and Trademark Office, both of which are incorporated by reference in their entireties herein.

[0002] FIELD OF THE DISCLOSURE The present disclosure relates generally to the field of data processing, and more specifically to video coding. [Background technology]

[0003] Video coding and decoding using inter-picture prediction with motion compensation has been known for several decades. Uncompressed digital video can consist of a sequence of pictures, each with spatial dimensions of, for example, 1920x1080 luminance samples and associated chrominance samples. The sequence of pictures can have a fixed or variable picture rate (also informally known as frame rate), for example, 60 pictures per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video (1920x1080 luminance sample resolution at a frame rate of 60 Hz) at 8 bits per sample requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires more than 600 GByte of storage space.

[0004] Traditional video coding standards such as H.264 / Advanced Video Coding (H.264 / AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC) share a similar (recursive) block-based hybrid prediction / transform framework, and individual coding tools such as intra / inter prediction, integer transform, and context-adaptive entropy coding are intensively hand-crafted to optimize overall efficiency. Summary of the Invention [Means for solving the problem]

[0005] Embodiments relate to methods, systems, and computer-readable media for video coding. According to one aspect, a method for video coding is provided. The method may include receiving video data including one or more frames. One or more quantization parameters associated with the received video data are determined for frame generation or enhancement of a target frame from among the one or more frames. Based on the determined quantization parameters, the video data is decoded.

[0006] According to another aspect, a computer system for video coding is provided. The computer system includes one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored in at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, such that the computer system is capable of performing a method. The method may include receiving video data including one or more frames. One or more quantization parameters associated with the received video data are determined for frame generation or enhancement of a target frame from among the one or more frames. The video data is decoded based on the determined quantization parameters.

[0007] According to yet another aspect, a computer-readable medium for video coding is provided. The computer-readable medium may include one or more computer-readable storage devices and program instructions stored in at least one of the one or more tangible storage devices, the program instructions being executable by a processor. The program instructions are executable by the processor to perform a method that may suitably include receiving video data including one or more frames. One or more quantization parameters associated with the received video data are determined for frame generation or enhancement of a target frame from among the one or more frames. Based on the determined quantization parameters, the video data is decoded.

[0008] These and other objects, features and advantages will become apparent from the following detailed description of illustrative embodiments, which is to be read in connection with the accompanying drawings, in which various features of the drawings are not to scale, as the figures are for clarity in conjunction with the detailed description to facilitate understanding by those skilled in the art. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates a networked computing environment in accordance with at least one embodiment. [Diagram 2] 1 is a hierarchical temporal structure for loop filter / inter prediction according to at least one embodiment. [Diagram 3] 1 is an operational flowchart illustrating steps performed by a program for encoding and decoding video data according to at least one embodiment. [Figure 4] FIG. 2 is a block diagram of internal and external components of the computer and server shown in FIG. 1 according to at least one embodiment. [Diagram 5] FIG. 2 is a block diagram of an exemplary cloud computing environment including the computer system shown in FIG. 1 according to at least one embodiment. [Figure 6]FIG. 6 is a block diagram of functional layers of the example cloud computing environment of FIG. 5 in accordance with at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Although detailed embodiments of the claimed structures and methods are disclosed herein, it can be understood that the disclosed embodiments are merely examples of the claimed structures and methods that can be embodied in various forms. However, these structures and methods can be embodied in many different forms and should not be construed as being limited to the exemplary embodiments described herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey its scope to those skilled in the art. In the description, details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the presented embodiments.

[0011] FIELD OF THE EMBODIMENTS The embodiments generally relate to the field of data processing, and more particularly to video coding. The exemplary embodiments described below provide, among other things, a system, method, and computer program for encoding and decoding video data based on a hierarchical temporal structure for loop filter / inter prediction. Thus, some embodiments have the ability to improve computational fields by enabling improved efficiency in video coding.

[0012] As mentioned before, video coding and decoding using inter-picture prediction with motion compensation has been known for several decades. Uncompressed digital video can consist of a sequence of pictures, each picture having spatial dimensions of, for example, 1920x1080 luminance samples and associated chrominance samples. The sequence of pictures can have a fixed or variable picture rate (also informally known as frame rate) of, for example, 60 pictures per second, i.e., 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video (1920x1080 luminance sample resolution at a frame rate of 60 Hz) at 8 bits per sample requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires more than 600 GByte of storage space. Traditional video coding standards such as H.264 / Advanced Video Coding (H.264 / AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC) share a similar (recursive) block-based hybrid prediction / transform framework, and individual coding tools such as intra / inter prediction, integer transform, and context-adaptive entropy coding are intensively hand-crafted to optimize overall efficiency.

[0013] One goal of video coding and decoding may be to reduce redundancy in the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage space requirements, in some cases by more than two orders of magnitude. Both lossless and lossy compression, as well as combinations thereof, may be used. Lossless compression refers to techniques where an exact replica of the original signal can be reconstructed from the compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for its intended application. For video, lossy compression is widely adopted. The amount of acceptable distortion depends on the application, for example, a user of a particular consumer streaming application may tolerate higher distortion than a user of a television distribution application. The achievable compression ratio may reflect that a higher acceptable / tolerable distortion may result in a higher compression ratio.

[0014] The spatiotemporal pixel neighborhood is utilized for predictive signal construction to obtain the corresponding residual for subsequent transformation, quantization, and entropy coding. Meanwhile, the nature of neural networks (NNs) is to extract different levels of spatiotemporal stimuli by analyzing spatiotemporal information from the receptive fields of neighboring pixels. The ability to explore highly nonlinear and nonlocal spatiotemporal correlations offers promising opportunities for significantly improving compression quality.

[0015] However, one caveat of leveraging information from multiple neighboring video frames is the complex motion caused by moving turtles and dynamic scenes. Traditional block-based motion vectors do not work well for non-translational motion. Learning-based optical flow methods can provide accurate motion information at the pixel level, but unfortunately are prone to errors, especially along the boundaries of moving objects. In some hybrid inter-frame predictions, NN-based models are applied to implicitly handle arbitrarily complex motion in a data-driven manner.

[0016] Therefore, in order to have a better trade-off between performance and coding execution time, when using a NN-based model as a loop filter (LF) or inter-prediction tool, it may be advantageous to select a different frame as a reference frame to apply the LF or generate intermediate frames.

[0017] Aspects are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer-readable media according to various embodiments. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0018] The exemplary embodiments described below provide a system, method, and computer program for encoding and decoding video data. Referring now to FIG. 1, a functional block diagram of a networked computing environment illustrates a video coding system 100 (hereinafter "system") for encoding and decoding video data. It should be understood that FIG. 1 provides only an example of one implementation and does not imply any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the illustrated environment may be made based on design and implementation requirements.

[0019] The system 100 may include a computer 102 and a server computer 114. The computer 102 may communicate with the server computer 114 via a communications network 110 (hereinafter "network"). The computer 102 may include a processor 104 and a software program 108 stored in a data storage device 106 capable of interfacing with a user and communicating with the server computer 114. As described below with reference to FIG. 4, the computer 102 may include internal components 800A and external components 900A, respectively, and the server computer 114 may include internal components 800B and external components 900B, respectively. The computer 102 may be, for example, a mobile device, a phone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device capable of running programs, accessing a network, and accessing a database.

[0020] The server computer 114 may also operate in a cloud computing service model, such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (laaS), as described below with respect to Figures 5 and 6. The server computer 114 may also be located in a cloud computing deployment model, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud.

[0021] The server computer 114, which may be used to encode and decode video data, is capable of executing a video coding program 116 (hereinafter, the "program") that may interact with the database 112. The video coding program method is described in more detail below with respect to FIG. 3. In one embodiment, the computer 102 may operate as an input device including a user interface, and the program 116 may operate primarily on the server computer 114. In an alternative embodiment, the program 116 may operate primarily on one or more computers 102, and the server computer 114 may be used to process and store data used by the program 116. It should be noted that the program 116 may be a stand-alone program or may be integrated into a larger video coding program.

[0022] However, it should be noted that processing for the program 116 may, in some cases, be shared between the computer 102 and the server computer 114 in any ratio. In other embodiments, the program 116 may run on more than one computer, server computer, or some combination of computers and server computers, e.g., multiple computers 102 communicating with a single server computer 114 over the network 110. In other embodiments, for example, the program 116 may run on multiple server computers 114 communicating with multiple client computers over the network 110. Alternatively, the program may run on a network server that communicates with the server and multiple client computers over the network.

[0023] The network 110 may include wired, wireless, or fiber optic connections, or some combination thereof. In general, the network 110 may be any combination of connections and protocols that support communication between the computer 102 and the server computer 114. The network 110 may include various types of networks, such as, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, a telecommunications network such as the Public Switched Telephone Network (PSTN), a wireless network, a public switched network, a satellite network, a cellular network (e.g., a fifth generation (5G) network, a long-term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a metropolitan area network (MAN), a private network, an ad-hoc network, an intranet, or a fiber optic-based network, etc., and / or a combination of these or other types of networks.

[0024] The number and arrangement of devices and networks shown in Figure 1 are provided as an example. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or different arrangements of devices and / or networks than those shown in Figure 1. Furthermore, two or more devices shown in Figure 1 may be implemented within a single device, or a single device shown in Figure 1 may be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (e.g., one or more devices) of system 100 may perform one or more functions that are described as being performed by other sets of devices of system 100.

[0025] 2, a hierarchical temporal structure 200 for loop filter / inter prediction is shown. The hierarchical structure 200 can apply NN-based loop filter or inter-frame prediction in video coding and decoding, and more specifically, can determine the number / index of a frame to be applied as a reference frame for using a NN model for frame generation or enhancement in video coding of I / P / B frames, respectively. 1 ,...,x T Assume an input video x containing 1 to 16 pixels (e.g., 1 to 16). In the first motion estimation step, the frame is divided into spatial blocks, and each block can be iteratively divided into smaller blocks, so that the current frame x t and the set of previous reconstructed frames

number

number

number

number

number

number

[0026] The quantization step gives the quantized transform block.

number

number

number

number

number

[0027] In HEVC, VVC, or other video coding frameworks or standards, the decoded picture may be included in a reference picture list (RPL) and may be used for motion compensated prediction as a reference picture and other parameter prediction for coding subsequent pictures in encoding or decoding order. Alternatively, the decoded portion of the current picture may be used for intra prediction or intra block copying for coding different regions or blocks of the current picture.

[0028] In one example, one or more virtual references may be generated and included in the RPL of both the encoder and the decoder, or only the decoder. The virtual reference picture may be generated by one or more processes including signal processing, spatial or temporal filtering, scaling, weighted averaging, up / down sampling, pooling, recursive processing with memory, linear system processing, nonlinear system processing, neural network processing, deep learning based processing, AI processing, pre-trained network processing, machine learning based processing, online learning network processing, or combinations thereof. The process for generating the virtual reference uses as input data zero or more forward reference pictures that are before the current picture in both output / display order and encoding / decoding order, and zero or more backward reference pictures that are after the current picture in both output / display order but before the current picture in encoding / decoding order. The output of the process is a virtual / generated picture that is used as a new reference picture. When this new reference picture is selected to predict a coding block in the current picture, a normal motion compensation technique may be applied.

[0029] In one example, the NN-based method may be applied to in-loop filter design at both slice / CTU level of each frame in combination with one or more of the above-mentioned additional components (e.g., DF, SAO, ALF, CCALF, etc.), or may be applied to replace one or more of the above-mentioned additional components (e.g., DF, SAO, ALF, CCALF, etc.). When applied, the reconstructed current picture is used as input data to the NN-based model to generate a NN-enhanced filtered picture. For each block or CTU, a decision may be made to select this NN-enhanced filtered picture as a post-filtering result or to use a conventional filtering method.

[0030] One or more NN models may be selected as the NN-based video coding tool for different scenarios to apply to pictures that meet certain conditions. In one embodiment, one or more NN-based models may be selected in the NN-based video coding tool for different QPs (quantization parameters). In other words, the NN-based video coding tool may use one model for all QPs, or separate models for each QP, or separate models with one model assigned to a set of QP ranges.

[0031] In one or more embodiments, for pictures under different hierarchical levels, one or more NN-based models are used in the NN-based video coding tool. In other words, the NN-based video coding tool can use one model for all frames, or can use separate models for different frames whose POCs meet certain conditions. For example, in FIG. 2 Figure 1 shows an example of a hierarchical structure of loop filters / inter prediction. One model is designed for frames with a hierarchy level ID equal to 1, another model is designed for frames with a hierarchy level ID equal to 2, etc. In another example, multiple models may be applied in a NN-based video coding tool for pictures with different hierarchy level IDs that are even or odd. In other words, one model is designed for frames with hierarchy level IDs that are even, and another model is designed for frames with hierarchy level IDs that are odd.

[0032] In one or more embodiments, for different frames having different types of reference picture lists (RPL), one or more NN-based models may be selected in the NN-based video coding tool. In other words, the NN-based video coding tool may use one model for all frames, or may use separate models for different frames whose reference picture lists (RPL) satisfy certain conditions. For example, one model is designed for frames whose length of their reference picture lists (RPL) is equal to 1, another model is designed for frames whose length of their reference picture lists (RPL) is equal to 2, and so on. In another example, multiple models may be applied in the NN-based video coding tool when reference pictures in the reference picture lists (RPL) come from different hierarchical levels. In other words, one model is designed for frames whose reference pictures in the reference picture lists (RPL) come from hierarchical levels 1 and 2, another model is designed for frames whose reference pictures in the reference picture lists (RPL) come from hierarchical levels 1, 2, and 3, and so on.

[0033] In one or more embodiments, one or more NN-based models are selected in the NN-based video coding tool for different frames having different numbers of reference frames as inputs for the NN-based models. In other words, the NN-based video coding tool can use one model for all frames, or can use separate models for different frames that use a certain number of reference frames as inputs for the NN-based models. For example, one model is designed for frames that select one unique reference frame in the RPL as inputs to the NN model, and another model is designed for frames that select two unique reference frames in the RPL as inputs to the NN model. In other examples, the reference frames may not need to be presented in the RPL, but may be stored in the DPB so that both the encoder and the decoder can access them.

[0034] In one or more embodiments, one or more NN-based models are selected in the NN-based video coding tool for different frames that have different temporal distances to their reference frames as inputs of the NN-based models. In other words, the NN-based video coding tool can use one model for all frames, or can use separate models for different frames whose reference frames can have different temporal distances to them. For example, one model is designed for a current frame that uses a reference frame whose temporal distance (to the current frame) is equal to 1 as input to the NN model, another model is designed for a current frame that has a reference frame whose temporal distance (to the current frame) is equal to 2 as input to the NN model, and so on.

[0035] In one or more embodiments, the NN-based video coding may be NN-based inter-prediction or loop filtering, or both.

[0036] The multiple model selection can be applied as a NN-based coding tool in video coding. The proposed method determines whether to select one or more NN-based models as a neural network-based coding tool for a picture according to different conditions. The NN-based coding tool can include, but is not limited to, NN-based loop filtering, NN-based virtual reference picture for inter prediction. The following are some examples to further elaborate the proposed method.

[0037] In one example, Fig. 2 Given the hierarchical GOP structure of, one model can be used for pictures of all levels, or for each level, separate models can be used for different hierarchical level IDs of frames (different conditions of POC). As an example, for a current picture (with POC=3), it has a different hierarchical level than other pictures (with POC=2), and with different scenarios or conditions, these two pictures can use one common model, or separate models in the NN-based coding tool.

[0038] In another example, FIG. 2 Considering the hierarchical GOP structure of, for a current picture (with POC=3), it has a reference picture list (RPL) of pictures (with POC=0,2,4,8), and for a current picture (with POC=10), it has a reference picture list (RPL) of pictures (with POC=8,12,16), and these two frames have different reference picture lists (RPL) whose reference frames are from different hierarchical levels, and such two pictures can use one common model or separate models as the NN-based coding tool.

[0039] In another example, FIG. 2Considering the hierarchical GOP structure of, for a current picture (with POC=3), we have a reference picture list (RPL) of pictures (with POC=0, 2, 4, 8), and different numbers of reference frames can be fed into the NN-based model. When different numbers of reference frames are used as input, it can use one common model, or separate models as the NN-based coding tool.

[0040] In another example, FIG. 2 Considering the hierarchical GOP structure of, for a current picture (with POC=3), it has a Reference Picture List (RPL) of pictures (with POC=0, 2, 4, 8), and for each reference frame, it may have a different temporal distance to the current picture, and for the different temporal distances of the reference frames relative to the current picture, it can be applied using one common model or using separate models of the NN-based coding tool.

[0041] Referring now to FIG. 3, there is shown an operational flow chart illustrating steps of a method 300 performed by a program for encoding and decoding video data.

[0042] At 302, the method 300 may include receiving video data including one or more frames.

[0043] At 304, the method 300 may include determining one or more quantization parameters associated with the received video data for frame generation or enhancement of a target frame from among the one or more frames.

[0044] At 306, the method 300 may include decoding the video data based on the determined quantization parameter.

[0045] It will be appreciated that Figure 3 is only intended to provide an illustration of one implementation and is not intended to suggest any limitations as to how different embodiments may be implemented. Many modifications to the depicted environment may be made based on design and implementation requirements.

[0046] Figure 4 is a block diagram 400 of the internal and external components of the computer shown in Figure 1, in accordance with an exemplary embodiment. It should be understood that Figure 4 provides only an illustration of one implementation and is not intended to imply any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made based on design and implementation requirements.

[0047] The computer 102 (FIG. 1) and the server computer 114 (FIG. 1) may include respective sets of internal components 800A, 800B and external components 900A, 900B shown in FIG. 5. Each of the set of internal components 800 includes one or more processors 820, one or more computer-readable RAMs 822 and one or more computer-readable ROMs 824 on one or more buses 826, one or more operating systems 828, and one or more computer-readable tangible storage devices 830.

[0048] The processor 820 is implemented in hardware, firmware, or a combination of hardware and software. The processor 820 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or other types of processing components. In some implementations, the processor 820 includes one or more processors that can be programmed to perform functions. The bus 826 includes components that enable communication between the internal components 800A, 800B.

[0049] One or more operating systems 828, software programs 108 (FIG. 1), and video coding programs 116 (FIG. 1) on the server computer 114 (FIG. 1) are stored in one or more of the respective computer-readable tangible storage devices 830 for execution by one or more of the respective processors 820 via one or more of the respective RAMs 822 (which typically include cache memory). In the embodiment shown in FIG. 4, each of the computer-readable tangible storage devices 830 is an internal hard drive magnetic disk storage device. Alternatively, each of the computer-readable tangible storage devices 830 is a semiconductor storage device such as a ROM 824, an EPROM, a flash memory, an optical disk, a magneto-optical disk, a solid-state disk, a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, a magnetic tape, and / or other types of non-transitory computer-readable tangible storage devices that may store computer programs and digital information.

[0050] Each set of internal components 800A, 800B also includes a R / W drive or interface 832 for reading from and writing to one or more portable computer-readable tangible storage devices 936, such as a CD-ROM, a DVD, a memory stick, a magnetic tape, a magnetic disk, an optical disk, or a semiconductor storage device. Software programs, such as software program 108 (FIG. 1) and video coding program 116 (FIG. 1), may be stored in one or more of the respective portable computer-readable tangible storage devices 936 and read out via the respective R / W drive or interface 832 and loaded onto the respective hard drive 830.

[0051] Each set of internal components 800A, 800B also includes a network adapter or interface 836, such as a TCP / IP adapter card, a wireless Wi-Fi interface card, or a 3G, 4G, or 5G wireless interface card or other wired or wireless communication link. The software program 108 (FIG. 1) and the video coding program 116 (FIG. 1) on the server computer 114 (FIG. 1) may be downloaded to the computer 102 (FIG. 1) and the server computer 114 from an external computer via a network (e.g., the Internet, a local area network, or other wide area network) and the respective network adapter or interface 836. From the network adapter or interface 836, the software program 108 and the video coding program 116 on the server computer 114 are loaded onto the respective hard drives 830. The network may include copper wire, optical fiber, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers.

[0052] Each of the sets of external components 900A, 900B may include a computer display monitor 920, a keyboard 930, and a computer mouse 934. The external components 900A, 900B may also include touch screens, virtual keyboards, touch pads, pointing devices, and other human interface devices. Each of the sets of internal components 800A, 800B also includes a device driver 840 for interfacing to the computer display monitor 920, the keyboard 930, and the computer mouse 934. The device driver 840, the R / W drive or interface 832, and the network adapter or interface 836 include hardware and software (stored in the storage device 830 and / or the ROM 824).

[0053] Although the present disclosure includes detailed descriptions of cloud computing, it is understood in advance that implementations of the teachings described herein are not limited to cloud computing environments. Rather, some embodiments may be implemented in conjunction with any other type of computing environment now known or later developed.

[0054] Cloud computing is a service delivery model that enables easy, on-demand network access to a shared pool of configurable computing resources (such as networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, services, etc.) that can be rapidly provisioned and released with minimal administration or interaction with a service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0055] The characteristics are as follows: On-demand Self-service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, as needed, without requiring human interaction with the provider of the service. Broad Network Access: Capabilities are available over the network and accessed via standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, and PDAs). Resource Pooling: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated depending on demand. Consumers generally have no control or knowledge over the exact location of the provided resources, but there is a sense of location independence in that they can specify location at a higher level of abstraction (e.g., country, state, or data center). Rapid Elasticity: Capabilities can be rapidly and elastically provisioned, sometimes automatically, to quickly scale out, and rapidly released to quickly scale in. To the consumer, the capabilities available for provisioning often appear unlimited and can be purchased in any quantity at any time. Measured Service: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported to provide transparency to both providers and consumers of the services utilized.

[0056] The service model is as follows: Software as a Service (SaaS): The ability offered to a consumer is to use a provider's applications that run on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application functions, with the possible exception of limited user-specific application configuration settings. Platform as a Service (PaaS): The functionality offered to the consumer is the deployment of consumer-created or acquired applications, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does control the deployed applications and possibly the application hosting environment configuration. Infrastructure as a Service (laaS): The functionality offered to the consumer is to provision processing, storage, network, and other basic computing resources, and the consumer may deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does control the operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).

[0057] The deployment model is as follows: Private Cloud: The cloud infrastructure is operated exclusively for an organization. The cloud infrastructure may be managed by the organization or a third party and may exist on-premise or off-premise. Community Cloud: The cloud infrastructure is shared by several organizations to support a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). It may be managed by the organization or a third party and may exist on-premise or off-premise. Public Cloud: Cloud infrastructure is made available to the general public or large industry groups and is owned by organizations that sell cloud services. Hybrid Cloud: A cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain unique entities but are tied together by standardized or proprietary technologies that allow for data and application portability (e.g., cloud bursting for load balancing between clouds).

[0058] Cloud computing environments are service-oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0059] Referring to FIG. 5, an exemplary cloud computing environment 500 is shown. As shown, the cloud computing environment 500 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers may communicate, such as, for example, a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, and / or an automobile computer system 54N. The cloud computing nodes 10 may communicate with each other. They may be physically or virtually grouped (not shown) in one or more networks, such as the private cloud, community cloud, public cloud, hybrid cloud, or combinations thereof described above. This allows the cloud computing environment 500 to provide infrastructure, platform, and / or software as a service without the cloud consumer having to maintain resources on the local computing device. It will be understood that the types of computing devices 54A-54N shown in FIG. 5 are intended to be exemplary only, and that the cloud computing nodes 10 and the cloud computing environment 500 may communicate with any type of computerized device over any type of network and / or network addressable connections (e.g., using a web browser).

[0060] Referring to Figure 6, a set of functional abstraction layers 600 provided by the cloud computing environment 500 (Figure 5) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 6 are intended to be illustrative only, and the embodiments are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0061] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, RISC (reduced instruction set computer) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0062] The virtualization layer 70 provides an abstraction layer over which the following examples of virtual entities may be provided: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.

[0063] In one example, management layer 80 may provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides allocation and management of cloud computing resources such that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides pre-positioning and procurement of cloud computing resources where future requirements are anticipated according to SLAs.

[0064] The workload tier 90 provides examples of functions for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this tier include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and video coding 96. Video coding 96 may encode and decode video data based on using multiple neural network models for prediction and loop filtering.

[0065] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of integration of technical details. The computer-readable media may include a computer-readable non-transitory storage medium having computer-readable program instructions for causing a processor to perform operations.

[0066] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves with instructions recorded on them, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0067] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium into each computing / processing device, or may be downloaded to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0068] The computer readable program code / instructions for performing the operations may be either assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk or C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or a connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, an electronic circuit, including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuit to perform an aspect or operation.

[0069] These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in the flowchart and / or block diagram blocks. These computer-readable program instructions may also be stored on a computer-readable storage medium that may direct a computer, programmable data processing apparatus, and / or other apparatus to function in a particular manner, such that the computer-readable storage medium having stored thereon instructions comprises an article of manufacture including instructions that implement aspects of the functions / acts specified in the flowchart and / or block diagram blocks.

[0070] The computer readable program instructions may also be loaded onto a computer, other programmable data processing device, or other device such that a series of operational steps are executed on the computer, other programmable device, or other device to generate a computer-implemented process such that the instructions, executing on the computer, other programmable device, or other device, perform the functions / acts specified in the flowchart and / or block diagram blocks.

[0071] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of an instruction that includes one or more executable instructions for performing a specified logical function. The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks compared to those shown in the figures. In some alternative implementations, the functions described in the blocks may be performed in a different order than described in the figures. For example, two blocks shown in succession may in fact be executed simultaneously or substantially simultaneously, or the blocks may be executed in reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, as well as combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or realizes a combination of dedicated hardware and computer instructions.

[0072] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0073] No element, act, or instruction used herein should be construed as critical or essential unless expressly described as such. Also, as used herein, the articles "a" and "an" are intended to include one or more and may be used interchangeably with "one or more." Furthermore, the term "set" as used herein includes one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with "one or more." When only one item is addressed, the term "one" or similar words are used. Also, as used herein, terms such as "has," "have," and "having" are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless otherwise specified.

[0074] The description of various aspects and embodiments is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Although combinations of features are recited in the claims and / or disclosed herein, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed herein. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with all other claims in the claim set. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used in this specification have been selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the marketplace, or to enable those skilled in the art to understand the embodiments disclosed herein. [Explanation of symbols]

[0075] 10 Cloud Computing Nodes 54A Personal digital assistant (PDA) or mobile phone 54B Desktop Computer 54C Laptop Computer 54N Automotive Computer System 60 Hardware and Software Layers 61 Mainframe 62 RISC architecture based servers 63 Server 64 Blade Server 65 Storage device 66 Networks and Networking Components 67 Network Application Server Software 68 Database Software 70 Virtualization Layer 71 Virtual Server 72 Virtual Storage 73 Virtual Network 74 Virtual Applications and Operating Systems 75 Virtual Clients 80 Management layer 81 Resource Provisioning 82 Measurement and Pricing 83 User Portal 84 Service Level Management 85 SLA Planning and Implementation 90 Workload Tier 91 Mapping and Navigation 92 Software Development and Lifecycle Management 93 Virtual Classroom Education Delivery 94 Data Analysis Processing 95 Transaction Processing 96 Video Coding 100 Video Coding System 102 Computer 104 processors 106 Data storage device 108 Software Programs 110 Communication Network 112 Database 114 Server Computer 116 Video Coding Programs 200 Hierarchical time structure, hierarchical structure 500 Cloud Computing Environments 600 Functional Abstraction Layer 800A, 800B Internal Components 820 Processor 822 Computer Readable RAM 824 Computer Readable ROM 826 Bus 828 Operating Systems 830 Computer-readable tangible storage device, hard drive 832 R / W drive or interface 836 Network Adapter or Interface 840 Device Drivers 900A, 900B External Components 920 computer display monitor 930 Keyboard 934 Computer Mouse 936 Portable computer-readable tangible storage device

Claims

1. 1. A processor-executable method for video coding, comprising: receiving video data including a plurality of frames corresponding to different hierarchical levels, the hierarchical level of each frame being determined based on the temporal position of each frame in the plurality of frames, and frames having the same hierarchical level being spaced apart from each other by a predetermined distance; determining one or more quantization parameters associated with the received video data for frame generation or enhancement of a target frame from among the plurality of frames; decoding the video data based on the determined quantization parameter, wherein a plurality of neural network models are selected for the frame generation or enhancement based on the determined quantization parameter, a first neural network model of the plurality of neural network models is designed for frames of a first hierarchical level, and a second neural network model of the plurality of neural network models, different from the first neural network model, is designed for frames of a second hierarchical level, the second hierarchical level being different from the first hierarchical level; A method comprising:

2. The method of claim 1 , wherein the target frame corresponds to one or more of an I-frame, a P-frame, and a B-frame.

3. The method of claim 1 , wherein the decoded video data is included in a reference picture list.

4. The method of claim 3 , wherein, based on the reference picture list, subsequent frames of the plurality of frames are decoded based on motion compensated prediction.

5. The method of claim 3 , wherein one or more virtual reference frames are generated and included in the reference picture list.

6. 6. The method of claim 5, wherein the virtual reference frame is generated based on one or more of signal processing, spatial or temporal filtering, scaling, weighted averaging, up / down sampling, pooling, recursive processing with memory, linear system processing, non-linear system processing, neural network processing, deep learning based processing, AI processing, pre-trained network processing, machine learning based processing, and online learning network processing.

7. A computer system for video coding arranged to perform the method according to any one of claims 1 to 6.

8. A computer program for causing one or more computer processors to carry out the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Image filter device, image decoding device, and image coding device

    WO2019031410A1