Video coding method and device, storage medium and computer device

Abstract

A video coding method, device, storage medium and computer device are disclosed. The method comprises receiving video data, the video data comprising a current picture; generating a virtual reference frame for the current picture based on a hierarchical level associated with the current picture and a most recently decoded picture; and decoding the video data based on the virtual reference frame.

Description

[0001] Incorporation by Reference

[0002] This application claims priority to U.S. Provisional Application No. 63 / 136,055, filed January 11, 2021, and U.S. Application No. 17 / 478,138, filed September 17, 2021, the entire contents of which are incorporated by reference herein. TECHNICAL FIELD

[0003] Embodiments of the present application relate to data processing technology, and in particular, to a video coding method and device, a storage medium and a computer device. BACKGROUND

[0004] Inter prediction with motion compensation has been used in video encoding and decoding for decades. Uncompressed digital video can consist of a series of pictures, each picture having a spatial dimension, for example 1920 x 1080 luminance samples and associated chrominance samples. The series of pictures can have a fixed or variable picture rate (also known as frame rate), for example 60 pictures per second or 60 Hz. Uncompressed video has a specific bitrate requirement. For example, an 1080p60 4:2:0 video (60 Hz frame rate, 1920 x 1080 luminance sample resolution) at 8 bit per sample requires close to 1.5 Gbit / s bandwidth. An hour of such video requires more than 600 GB of storage space.

[0005] Traditional video coding standards, such as H.264 / Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC), share a similar (recursive) block-based hybrid prediction / transform framework, where various coding tools, such as intra / inter prediction, integer transform and context adaptive entropy coding, are centrally handcrafted to optimize the overall efficiency.

[0006] Currently, neither the block-based motion vector nor the learning-based optical flow method can adapt to the complex motion of the video frame of the non-translational motion of the object, especially at the boundary of the moving object. Therefore, how to improve the coding efficiency of the video frame of the complex motion is a problem to be solved. SUMMARY

[0007] Embodiments of the present application relate to a video coding method and device, a storage medium and a computer device.

[0008] According to an aspect of the present application, a video coding method is provided, comprising:

[0009] receiving video data, the video data comprising a current picture;

[0010] generate a virtual reference frame for the current picture based on a hierarchy level associated with the current picture and a recently decoded picture; and

[0011] decode the video data based on the virtual reference frame.

[0012] According to another aspect of the present disclosure, a video coding device is provided, comprising:

[0013] a receiving module configured to receive video data, the video data comprising a current picture;

[0014] a generating module configured to generate a virtual reference frame for the current picture based on a hierarchy level associated with the current picture and a recently decoded picture; and

[0015] a decoding module configured to decode the video data based on the virtual reference frame.

[0016] According to another aspect of the present disclosure, a non-transitory computer readable medium having stored thereon a computer program for video coding is provided, the computer program configured to cause at least one computer processor to perform the above-mentioned video coding method.

[0017] According to another aspect of the present disclosure, a computer device is provided, comprising a processor and a memory, the memory having stored therein at least one instruction, the at least one instruction being loaded and executed by the processor to implement the above-mentioned video coding method.

[0018] According to embodiments of the present disclosure, a computer program product or computer program is also provided, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the above-mentioned video coding method.

[0019] As can be seen from the above technical solutions, according to the method provided by embodiments of the present disclosure, based on the hierarchical time structure, a neural network based loop filter or inter prediction can be used, and the number / index of frames to be used as reference frames can be determined, so that neural models can be used for frame generation or enhancement in the video coding of I / P / B frames, respectively, and thus the performance and coding running time can be better balanced, and the coding efficiency can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above objectives, as well as other objectives, features and advantages of the present application will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, wherein: various features of the drawings serve to better clarify one or more embodiments of the present application and, wherein:

[0021] Figure 1 is a networked computer environment in accordance with at least one embodiment;

[0022] Figure 2 is a hierarchical temporal structure for in-loop filter / inter prediction in accordance with at least one embodiment;

[0023] Figure 3 is an operational flow diagram of steps performed by a program to code video data in accordance with at least one embodiment;

[0024] Figure 4 is a block diagram of internal and external components of a computer and server shown in FIG. 1 1 1 in accordance with at least one embodiment; Figure 1

[0025] Figure 5 is a block diagram of a computer device shown in FIG. 1 1 1 in accordance with at least one embodiment; Figure 1

[0026] Figure 6 is a block diagram of functional layers of an example cloud computing environment shown in FIG. 1 1 1 in accordance with at least one embodiment. Figure 5 DETAILED DESCRIPTION

[0027] The present application discloses specific embodiments of the claimed structures and methods. However, it is understood that the disclosed embodiments are merely examples of the claimed structures and methods which can be embodied in various forms. The structures and methods, however, can be embodied in many different forms and should not be construed as limited to the exemplary embodiments described in this application. Rather, these exemplary embodiments are provided so that this application will be thorough and complete, and fully convey the scope of the disclosure to those skilled in the art. In the description of the specification, details of well-known features and techniques can be omitted to avoid unnecessarily obscuring the presented embodiments.

[0028] Embodiments relate generally to the field of data processing, and more particularly to video processing. The exemplary embodiments described below provide a system, method, and computer program to encode and / or decode video using in-loop filter / inter prediction based on a hierarchical temporal structure, among other things. Thus, some embodiments improve the efficiency of computation by improving the efficiency of video coding.

[0029] ​​​As noted above, inter prediction with motion compensation has been used in video encoding and decoding for decades. Uncompressed digital video can consist of a series of pictures, each picture having a spatial dimension, for example 1920 x 1080 luminance samples and associated chrominance samples. The series of pictures can have a fixed or variable picture rate (also known as frame rate), for example 60 pictures per second or 60 Hz. Uncompressed video has a specific bitrate requirement. For example, a 1080p60 4:2:0 video (60 Hz frame rate, 1920 x 1080 luminance sample resolution) at 8 bit per sample requires close to 1.5 Gbit / s bandwidth. An hour of such video requires more than 600 GB of storage space. Traditional video coding standards, such as H.264 / Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC), share a similar (recursive) block-based hybrid prediction / transform framework, where individual coding tools, such as intra / inter prediction, integer transform, and context adaptive entropy coding, are centrally handcrafted to optimize overall efficiency.

[0030] One goal of video encoding and decoding is to reduce the redundancy in the input video signal by compression. Compression can help reduce the bandwidth or storage space needs, in some cases, by two orders of magnitude or more. Both lossless and lossy compression, and combinations thereof, can be employed. Lossless compression refers to techniques where an exact copy of the original signal can be reconstructed from the compressed original signal. When using lossy compression, the reconstructed signal can not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for an intended application. For video, lossy compression is widely employed. The amount of distortion tolerated depends on the application; for example, some consumer streaming applications can tolerate more distortion than a television contribution application. The achievable compression ratio can reflect that a higher allowable / tolerable distortion can result in a higher compression ratio.

[0031] The construction of the prediction signal with spatio-temporal pixel neighborhoods to obtain the corresponding residual for subsequent transform, quantization, and entropy coding. On the other hand, the essence of a neural network (NN) is to extract different levels of spatio-temporal stimuli by analyzing the spatio-temporal information from the receptive field of neighboring pixels. The ability to explore highly nonlinear and non-local spatio-temporal correlations provides promising opportunities to greatly improve the compression quality.

[0032] However, when utilizing information from multiple neighboring video frames, one caveat is the complex motion caused by moving cameras and dynamic scenes. Traditional block-based motion vectors do not work well for non-translational motion. Learning-based optical flow methods can provide accurate motion information at the pixel level, unfortunately, this is error-prone, especially along the boundaries of moving objects. In certain hybrid inter-prediction, a NN-based model can be used to implicitly handle arbitrary complex motion in a data-driven manner.

[0033] Therefore, when using a NN-based model as a loop filter (LF) or for an inter-prediction tool, it is advantageous to select different frames as reference frames to apply the LF or to generate an intermediate frame, which can better trade-off performance and coding runtime.

[0034] The methods, apparatuses (systems) and computer readable storage media of various embodiments are described with reference to flowcharts and / or block diagrams illustrating the apparatuses (systems) and computer readable storage media according to various embodiments. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer readable program instructions.

[0035] The example embodiments described below provide a system, method, and computer program for video coding using reference picture samples instead of padding. Figure 1 is a functional block diagram of a networked computer environment showing a multimedia processing system 100 (hereinafter "system") for video coding. It should be understood that Figure 1 The illustration of only one implementation is provided for purposes of clarity and example. Various modifications to this environment can be made based on the design and implementation requirements.

[0036] The system 100 can include a computer 102 and a server computer 114. The computer 102 can communicate with the server computer 114 via a communication network 110 (hereinafter "network"). The computer 102 can include a processor 104 and a software program 108 stored in a data storage device 106 and capable of interfacing with a user and communicating with the server computer 114. As will be discussed below, Figure 4 As discussed, the computer 102 can include internal components 800A and external components 900A, respectively, and the server computer 114 can include internal components 800B and external components 900B, respectively. The computer 102 can be, for example, a mobile device, a phone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device capable of running a program, accessing a network, and accessing a database.

[0037] The server computer 114 can also operate in a cloud computing service model, such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (laaS), as discussed below with reference to Figure 5 and Figure 6 The server computer 114 can also be located in a cloud computing deployment model, such as a private cloud, community cloud, public cloud, or hybrid cloud.

[0038] The server computer 114, which can be used for video coding, is capable of running a DMVR video coding program 116 (hereinafter “program”) that can interact with the database 112. The following refers to Figure 3 The video coding program method is explained in more detail. In one embodiment, the computer 102 can operate as an input device that includes a user interface, while the program 116 can run primarily on the server computer 114. In an alternative embodiment, the program 116 can run primarily on at least one computer 102, while the server computer 114 can be used for processing and storing data used by the program 116. It should be noted that the program 116 can be a standalone program, or can be integrated into a larger video coding program.

[0039] It should be noted, however, that in some cases, processing of the program 116 can be shared between the computer 102 and the server computer 114 in any ratio. In another embodiment, the program 116 can run on more than one computer, server computer, or some combination of computers and server computers, for example, multiple computers 102 in communication with a single server computer 114 over the network 110. In another embodiment, for example, the program 116 can run on multiple server computers 114 in communication with multiple client computers over the network 110. Alternatively, the program can run on a web server in communication with a server and multiple client computers over a network.

[0040] The network 110 can include wired connections, wireless connections, fiber optic connections, or some combination thereof. In general, the network 110 can be any combination of connections and protocols that will support communications between the computer 102 and the server computer 114. The network 110 can include various types of networks, such as local area networks (LANs), wide area networks (WANs) such as the Internet, telecommunications networks such as a public switched telephone network (PSTN), wireless networks, public switched networks, satellite networks, cellular networks (e.g., fifth generation (5G) networks, long term evolution (LTE) networks, third generation (3G) networks, code division multiple access (CDMA) networks, etc.), public land mobile networks (PLMN), metropolitan area networks (MANs), private networks, ad hoc networks, intranets, fiber optic networks, etc., and / or combinations of these or other types of networks.

[0041] Figure 1 The number and arrangement of devices and networks shown is provided as an example. In practice, there can be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks than those shown. Figure 1 There can be more, fewer, or a different arrangement of devices and / or networks than those shown. Moreover, two or more devices shown can be implemented as a single device, and / or a single device shown can be implemented as multiple, distributed devices. Further, a group of devices (e.g., one or more devices) of system 100 can perform one or more functions described as being performed by another group of devices of system 100. Figure 1 Figure 1 The single devices shown can be implemented as multiple, distributed devices. Additionally or alternatively, a group of devices (e.g., at least one device) of system 100 can perform one or more functions described as being performed by another group of devices of system 100.

[0042] Reference is now made to Figure 2 depicts a hierarchical temporal structure 200 for in-loop filter / frame inter prediction. In video encoding and decoding, the hierarchical structure 200 can employ an NN-based in-loop filter or frame inter prediction, and more specifically, determine the number / index of frames to be used as reference frames for frame generation or enhancement using an NN model in video coding of I / P / B frames, respectively. Assume that an input video x comprises a number of image frames x1, x2,..., xT T (e.g., 1 to 16). In a first motion estimation step, the frames are partitioned into spatial blocks, each of which can be iteratively partitioned into smaller blocks, and for each block, a current frame x t is computed. A set of motion vectors m between the current frame x t and a set of previously reconstructed frames x contains frames from multiple previous coding periods. Then, in a second motion compensation step, for a current coding block in the current frame x , a prediction block is obtained by copying corresponding pixels from previous t frames based on the motion vectors m , and a residual r t between the original block and the prediction block can be obtained. In a third step, the residual r t is transformed, and then quantized. t .

[0043] The quantization step gives a quantized transform block. By entropy coding, the motion vectors m t and the quantized transform block are encoded into a bitstream and sent to a decoder. Then, on the decoder side, the decoded block will apply an inverse transform and dequantization (typically by an IDCT with dequantization coefficients) to obtain a recovered residual Then the ​to the predictor block to obtain a reconstructed block. The additional component is further used to improve the visual quality of the reconstructed In general, at least one of the following enhancement modules can be selected to process including Deblocking Filter (DF), Sample Adaptive Offset (SAO), Adaptive Loop Filter (ALF), etc.

[0044] In HEVC, VVC or other video coding framework or standard, decoded pictures can be included in a reference picture list (RPL) and can be used as reference pictures for motion-compensated prediction and other parametric prediction, so that at least one of the following pictures can be coded in the order of encoding or decoding. Alternatively, decoded parts of the current picture can be used for intra prediction or intra block copy to code different regions or blocks of the current picture.

[0045] In an example, at least one virtual reference can be generated and included in the RPL in both the encoder and the decoder, or only in the decoder. The virtual reference picture can be generated by at least one process, including signal processing, spatial or temporal filtering, scaling, weighted averaging, up / down sampling, pooling, recursive processing with memory, linear system processing, nonlinear system processing, neural network processing, deep learning based processing, AI processing, pre-trained network processing, machine learning based processing, online training network processing, or a combination thereof. For the process of generating at least one virtual reference, zero or more forward reference pictures, which are located before the current picture in the order of output / display and in the order of encoding / decoding, and zero or more backward reference pictures, which are located after the current picture in the order of output / display but before the current picture in the order of encoding / decoding, are used as input data. The output of the process is a virtual / generated picture, which will be used as a new reference picture. When this new reference picture is selected to predict a coded block in the current picture, conventional motion compensation techniques can be applied.

[0046] In an example, the NN-based approach can be applied to in-loop filter design, simultaneously on a per-frame basis in a slice / CTU level, in combination with or instead of at least one of the above-mentioned additional components (e.g., DF, SAO, ALF, CCALF, etc.). When applied, the reconstructed current picture will be used as input data for at least one NN-based model to generate a NN-enhanced filtered picture. For each block or CTU, it can be decided whether to select this NN-enhanced filtered picture as the post-filtering result or to use a traditional filtering method.

[0047] When the hierarchical level ID of a picture satisfies certain conditions, the NN-based video coding tools can be applied to the picture. In one embodiment, the NN-based video coding tools can be applied to pictures with temporal level ID less than or equal to a given threshold. In other words, the NN-based video coding tools can not be applied to pictures with temporal level ID greater than the given threshold. In another embodiment, the NN-based video coding tools can be applied to pictures with temporal level ID greater than a given threshold. In other words, the NN-based video coding tools can not be applied to pictures with temporal level ID less than or equal to a given threshold. In another embodiment, the NN-based video coding tools can be applied only to pictures with a specific temporal level ID. In another embodiment, the NN-based video coding can be NN-based inter prediction or loop filtering or both.

[0048] The hierarchical structure can be extended to a pre-defined prediction structure, some pictures in the sequence can be used as reference for other pictures, while some other pictures can not be used as reference at all. In other cases, some pictures in the sequence are considered more important than others. They are coded with a smaller QP. These pictures can be used as reference more frequently than others. In some cases, these pictures can be assigned with a specific hierarchical temporal level ID. The NN-based video coding tools can be applied to these pictures, while they can not be applied to the rest of the pictures in the sequence, in a similar sense as the above embodiments.

[0049] The hierarchical structure of the NN-based coding tools in video coding can determine whether to apply the neural network-based coding tools to a picture according to the hierarchical level it belongs to. The NN-based coding tools can include but are not limited to NN-based loop filtering, NN-based virtual reference pictures for inter prediction. The following are several examples to further describe the proposed methods in detail.

[0050] In one example, in the hierarchical temporal structure 200, when the picture order count (POC) of the current picture is equal to 3, generally, the decoded pictures with POC equal to 0, 2, 4, or 8 can be stored in the decoded picture buffer and some of them are included in the reference picture list (RPL) for decoding the current picture (POC = 3). For example, if the frame in level 4 is selected as the frame to apply the NN-based coding tool, in most NN-based inter prediction models, to generate the virtual reference frames for the current picture (with POC = 3), the latest decoded pictures with POC equal to 2 or 4 can be used as input data to feed into the virtual reference generation process using the NN-based model. The NN-based inter prediction model will be applied 8 times to generate all the virtual reference frames as the reference pictures for each picture at level 4. In the same or another embodiment, if the temporal level 4 is selected as the apply-filter frame, when the NN-based in-loop filter is used for detail enhancement or post-filtering, the NN-based in-loop filter generation will be activated 8 times to obtain the NN-enhanced filtered pictures from each picture at temporal level 4. In the same or another embodiment, all the pictures with temporal level ID less than or equal to 3 will not apply the NN-based video coding tools. In another example, all the pictures with temporal level ID less than or equal to 3 will apply the NN-based video coding tools, such as the NN-based in-loop filter or the NN-based inter prediction.

[0051] Referring now to the Figure 3 , an operational flow diagram illustrating steps of a method 300 of program execution for encoding video data by a hierarchical temporal structure for in-loop filter / inter prediction is depicted.

[0052] At 302, the method 300 can include receiving video data, the video data including a current picture.

[0053] At 304, the method 300 can include generating, based on a hierarchical level associated with the current picture and a latest decoded picture, a virtual reference frame for the current picture.

[0054] At 306, the method 300 can include decoding, based on the virtual reference frame, the video data.

[0055] It should be understood that Figure 3 Only a simplified diagram of one implementation is provided, and does not imply any limitation with regard to how different embodiments can be implemented. Various modifications to the described environments can be made based on design and implementation requirements.

[0056] Figure 4 are in accordance with the exemplary embodiments Figure 1The block diagram 400 depicts the internal and external components of the computer. It should be understood that... Figure 4 The illustrations provided are merely illustrative of one implementation method and do not imply any limitation on the environments in which different embodiments may be implemented. Various modifications may be made to the described environment based on design and implementation requirements.

[0057] Computer 102 ( Figure 1 ) and server computer 114 ( Figure 1 ) can include Figure 4 The corresponding sets of internal components 800A, 800B and external components 900A, 900B are shown. Each set of internal components 800 includes at least one processor 820 connected to at least one bus 826, at least one computer-readable random access memory (RAM) 822 and at least one computer-readable read-only memory (ROM) 824 including at least one operating system 828, and at least one computer-readable tangible storage device 830.

[0058] Processor 820 is implemented in hardware, firmware, or a combination of hardware and software. Processor 820 is a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or other type of processing component. In some embodiments, processor 820 includes at least one processor capable of being programmed to perform functions. Bus 826 includes components that allow communication between internal components 800A and 800B.

[0059] At least one operating system 828, and a server computer 114 ( Figure 1 Software program 108 on ) Figure 1 ) and DMVR video codec 116 ( Figure 1 All of these are stored on at least one corresponding computer-readable tangible storage device 830, for execution by at least one corresponding processor 820 via at least one corresponding RAM 822 (which typically includes a cache memory). Figure 4 In the illustrated embodiment, each computer-readable tangible storage device 830 is a disk storage device of an internal hard disk drive. Alternatively, each computer-readable tangible storage device 830 is a semiconductor storage device, such as ROM 824, erasable programmable read-only memory (EPROM), flash memory, optical disk, magneto-optical disk, solid-state disk, optical disc (CD), digital versatile optical disc (DVD), floppy disk, cassette tape, magnetic tape, and / or other types of non-volatile computer-readable tangible storage devices capable of storing computer programs and digital information.

[0060] Each set of internal components 800A, 800B also includes a read / write (R / W) drive or interface 832 to read from and write to at least one portable computer readable tangible storage device 936 such as a CD-ROM, DVD, memory stick, magnetic tape, magnetic disk, optical disk or semiconductor storage device. Software programs such as the software programs 108( Figure 1 ) and DMVR video codec program 116( Figure 1 ) can be stored on at least one respective portable computer readable tangible storage device 936, read via the respective R / W drive or interface 832 and loaded into the respective hard drive 830.

[0061] Each set of internal components 800A, 800B also includes a network adapter or interface 836 such as a TCP / IP adapter card, wireless Wi-Fi interface card, or a 3G, 4G, or 5G wireless interface card or other wired or wireless communication links. The software programs 108( Figure 1 ) and DMVR video codec program 116( Figure 1 ) on the server computer 114( Figure 1 ) can be downloaded to the computer 102( Figure 5 ) and server computer 114 from an external computer via a network (e.g., the Internet, a local area network or other network and a corresponding network adapter or interface 836. From the network adapter or interface 836, the software programs 108 and DMVR video codec program 116 on the server computer 114 are loaded into the respective hard drive 830. The network can comprise copper wires, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers.

[0062] Each set of external components 900A, 900B can include a computer display monitor 920, a keyboard 930, and a computer mouse 934. External components 900A, 900B can also include touch screens, virtual keyboards, touch pads, pointing devices, and other human interface devices. Each set of internal components 800A, 800B also includes device drivers 840 to interface to computer display monitor 920, keyboard 930, and computer mouse 934. Device drivers 840, R / W drive or interface 832 and network adapter or interface 836 comprise hardware and software (stored in storage device 830 and / or ROM 824).

[0063] It should be understood that, although this application includes detailed descriptions of cloud computing, embodiments of the application are not limited to cloud computing environments. Rather, some embodiments can be implemented in any of a variety of other types of computing environments now known or later developed.

[0064] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a provider of the service. This model can be composed of at least five characteristics, at least three service models, and at least four deployment models.

[0065] The features are as follows:

[0066] On-demand self-service: cloud users can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with the service's provider.

[0067] Broad network access: capabilities are available over a network and accessed through standard mechanisms that can promote heterogeneity among the thin or thick client platforms (e.g., mobile phones, laptops, and PDAs) that can be used to access the capabilities.

[0068] Resource pooling: the provider's computing resources are pooled to serve multiple users, with different physical and virtual resources dynamically assigned and reassigned according to demand. Location independence means that the user typically has no control or knowledge over the exact location of the provided resources but can be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).

[0069] Rapid elasticity: capabilities can be rapidly and elastically provisioned, in some cases automatically, to quickly scale out and rapidly released to quickly scale in. To the user, the capabilities available for provisioning often appear to be unlimited and can be purchased in any quantity at any time.

[0070] Measured service: cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both the provider and the user of the service.

[0071] The service models are as follows:

[0072] Software as a Service (SaaS): the functionality provided to the user is the use of the provider's applications, running on cloud infrastructure, through a thin client interface such as a web browser (e.g., web-based e-mail). The user does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0073] Platform as a Service (PaaS): the functionality provided to the user is the ability to deploy onto the cloud infrastructure user-created or acquired applications created using programming languages and tools supported by the provider. The user does not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but has control over the deployed applications and possibly application hosting environment configurations.

[0074] Infrastructure as a Service (laaS): the functionality provided to the user is the provision of processing, storage, networks, and other fundamental computing resources where the user is able to deploy and run arbitrary software, which can include operating systems and applications. The user does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).

[0075] Deployment models are as follows:

[0076] Private cloud: the cloud infrastructure is operated solely for an organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0077] Community cloud: the cloud infrastructure is shared by several organizations and supports specific interests of the community of which the organizations are a part. It can be managed by the organizations or a third party and can exist on-premises or off-premises.

[0078] Public cloud: the cloud infrastructure is made available to general public or a large industry group and is owned by an organization selling cloud services.

[0079] Hybrid cloud: the cloud infrastructure is a composition of two or more types of cloud (private, community, or public) that remain unique clouds but are bound together, creating the hybrid cloud using standardized or proprietary technologies that enable data and application portability.

[0080] A cloud computing environment is service oriented, with a focus on statelessness, low coupling, modularity, and semantic interoperability. The core of cloud computing is an infrastructure of interconnected nodes, including resources, organized to deliver virtual hosting.

[0081] Referring to Figure 5 illustrates an exemplary cloud computing environment 500. As shown, cloud computing environment 500 includes one or more cloud computing nodes 10 with which a cloud consumer can interact. For example, a cloud consumer can access cloud computing nodes via a network 14 using a computing device 50, such as an individual desktop, laptop computer, or cellular telephone. Cloud computing nodes 10 can communicate with one another. They can be grouped (not shown) physically or virtually, in Figure 6 The types of computing devices 54A-N shown are intended to be illustrative only and that cloud computing nodes 10 and cloud computing environment 500 can communicate with any type of computer devices over any type of network and / or network addressable connection (e.g., using a web browser).

[0082] Referring to Figure 5 illustrates a set of functional abstraction layers provided by cloud computing environment 500 Figure 6 It should be understood that ​ The components, layers, and functions shown are intended to be illustrative only and that embodiments are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0083] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include: mainframes 61; RISC (Reduced Instruction Set Computer) architecture based servers 62; servers 63; blade servers 64; storage devices 65; and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0084] Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71; virtual storage 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0085] In one example, management layer 80 can provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within the cloud computing environment. Resource scaling 82 provides dynamic scaling of resources and workloads assigned to a cloud computing environment. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management such that required service levels are met. Service Level Agreement (SLA) planning and fulfillment 85 provide pre-arrangement for, and procurement of, cloud computing resources for which a future requirement is anticipated in accordance with an SLA.

[0086] Workloads layer 90 provides examples of functionality for which the cloud computing environment can be utilized. Examples of workloads and functions which can be provided from this layer include: mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analytics processing 94, transaction processing 95, and DMVR video coding 96. DMVR video coding 96 can code a video using reference picture samples instead of padding.

[0087] Some embodiments can relate to systems, methods, and / or computer readable media at any possible technical detail level of integration. In embodiments of the present application, computer readable media can include non-transitory computer readable storage media (or medium) on which computer readable program instructions are stored, and that cause a processor to carry out operations when executed.

[0088] A computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0089] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0090] Computer readable program code / instructions for carrying out operations can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects or operations.

[0091] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including

[0092] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0093] The diagrams of the flowcharts and block diagrams in the drawings show the architecture, functionality, and operation of possible implementations of systems, methods and computer-readable media according to various embodiments. In this regard, each block in the flowcharts or block diagrams can represent a module, segment, or portion of instructions, which comprises at least one executable instruction for implementing the specified logical function(s). The methods, computer devices, and computer-readable media can include more or fewer blocks than shown in the diagrams, different blocks that are arranged in different arrangements, or different arrangements of blocks. In some optional implementations, the functions noted in the blocks can occur out of the order noted in the diagrams. For example, two blocks shown in succession can in fact be executed substantially concurrently or in the reverse order, depending on the functionality involved. Also, it is noted that each block of the block diagrams and / or flowcharts, and combinations thereof, can be implemented by dedicated hardware-based systems that perform the specified functions or combinations of special-purpose hardware and computer instructions.

[0094] It will be apparent that systems and / or methods, described herein, can be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not limiting of the implementations. Thus, the operation and behavior of the systems and / or methods were described herein without reference to specific software code — it being understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0095] No element, act, or instruction used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items, and can be used interchangeably with “one or more.” Furthermore, as used herein, the term “set” is intended to include at least one item (e.g., a related item, an unrelated item, a combination of related and unrelated items, or the like), and can be used interchangeably with “one or more.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based, at least in part, on” unless explicitly stated otherwise.

[0096] The description of the various aspects and embodiments has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the possible implementations. The terminology used is for the purpose of describing particular embodiments only and is not intended to be limiting of possible implementations. While each of the foregoing embodiments has been described with attention to particular combinations of features, it will be apparent that other combinations of features could be utilized and that various modifications and enhancements can be made to the described embodiments without departing from the scope of the possible implementations. In particular, those skilled in the art will recognize that elements from the various embodiments can be employed in combination with elements of other embodiments without departing from the scope of the possible implementations. It should be understood that every maximum numerical limitation given throughout this patent document will include every lower numerical limitation, but not higher numerical limitations, except that in context with measuring range, numerical limits of the lower range will not also imply a higher range.

Claims

1. A method of video decoding, the method comprising: comprising: receiving video data, the video data comprising a current picture; generating, by a neural network, a virtual reference frame for the current picture based on a hierarchical temporal level associated with the current picture and a most recently decoded picture, wherein some pictures in the video data are assigned a hierarchical temporal level identification (ID), encoded with a smaller quantization parameter (QP) than other pictures in the video data, and the neural network is applied to the some pictures but not to the other pictures; and decoding the video data based on the virtual reference frame.

2. The method of claim 1, wherein, The virtual reference frame corresponds to at least one from among an I-frame, a P-frame, and a B-frame.

3. The method according to claim 1 or 2, characterized in that, The decoded video data is included in a reference picture list.

4. The method of claim 3, wherein, Also comprising: decoding, based on the reference picture list, a subsequent frame from the video data based on motion-compensated prediction, intra prediction, or intra block copy.

5. The method of claim 3, wherein, The virtual reference frame is included in the reference picture list.

6. A video decoding apparatus, comprising: comprising: a receiving module configured to receive video data, the video data comprising a current picture; a generating module configured to generate, by a neural network, a virtual reference frame for the current picture based on a hierarchical temporal level associated with the current picture and a most recently decoded picture, wherein some pictures in the video data are assigned a hierarchical temporal level identification (ID), encoded with a smaller quantization parameter (QP) than other pictures in the video data, and the neural network is applied to the some pictures but not to the other pictures; and a decoding module configured to decode the video data based on the virtual reference frame. The virtual reference frame corresponds to at least one from among an I-frame, a P-frame, and a B-frame.

7. The apparatus of claim 6, wherein, The decoded video data is included in a reference picture list.

8. The apparatus of claim 6 or 7, wherein, The decoding module is further configured to decode, based on the reference picture list, a subsequent frame from the video data based on motion-compensated prediction, intra prediction, or intra block copy.

9. The apparatus of claim 8, wherein, The virtual reference frame is included in the reference picture list.

10. The apparatus of claim 8, wherein, A computer program stored thereon, the computer program for causing at least one computer processor to perform the method of any one of claims 1 to 5.

11. A non-transitory computer readable medium, characterized in that, comprising a processor and a memory having stored therein at least one instruction, the at least one instruction being loaded and executed by the processor to implement the method of any one of claims 1 to 5.

12. A computer device, comprising: ​

Images