Video coding method and device, storage medium and computer device

Abstract

This application discloses a video encoding / decoding method, apparatus, storage medium, and computer device. The method includes: receiving video data, the video data including at least one frame; determining at least one quantization parameter associated with the video data to perform frame generation or enhancement on a target frame from the at least one frame; and decoding the video data based on the quantization parameter.

Description

[0001] References merged

[0002] This application claims priority to U.S. Provisional Application No. 63 / 136,062, filed January 11, 2021, and U.S. Application No. 17 / 368,069, filed July 6, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to data processing technology, and more particularly to a video encoding / decoding method, apparatus, storage medium, and computer device. Background Technology

[0004] The use of motion-compensated inter-frame prediction in video encoding and decoding has been around for decades. Uncompressed digital video can consist of a series of pictures, each with a luminance sample and an associated chrominance sample in a spatial dimension of, for example, 1920x1080. This series of pictures can have a fixed or variable picture rate (also informally referred to as the frame rate), such as 60 pictures per second or 60 Hz. Uncompressed video has a defined bitrate requirement. For example, 1080p60 4:2:0 video with 8 bits per sample (at a frame rate of 60 Hz and a luminance sample resolution of 1920x1080) requires close to 1.5 Gbit / s of bandwidth. One hour of such video would require more than 600 GB of storage space.

[0005] Traditional video coding standards, such as H.264 / Advanced Video Coding (H.264 / AVC), High Efficiency Video Coding (HEVC), and Universal Video Coding (VVC), share a similar (recursive) block-based hybrid prediction / transform framework, in which individual codec tools, such as intra / inter-frame prediction, integer transform, and context-adaptive entropy coding, are centrally handcrafted to optimize overall efficiency. Summary of the Invention

[0006] This application relates to a video encoding / decoding method, apparatus, storage medium, and computer device.

[0007] According to one aspect of this application, a video encoding / decoding method is provided, comprising:

[0008] Receive video data, the video data including at least one frame;

[0009] Determine at least one quantization parameter associated with the video data to perform frame generation or enhancement on a target frame from the at least one frame; and,

[0010] The video data is decoded based on the quantization parameters.

[0011] According to another aspect of this application, a video encoding / decoding apparatus is provided, comprising:

[0012] A receiving module is configured to receive video data, the video data including at least one frame;

[0013] A determining module is configured to determine at least one quantization parameter associated with the video data for frame generation or enhancement of a target frame from the at least one frame; and,

[0014] The decoding module is used to decode the video data based on the quantization parameters.

[0015] According to another aspect of this application, a non-volatile computer-readable medium is provided, on which a computer program for video encoding and decoding is stored, the computer program being configured to cause at least one computer processor to execute the above-described video encoding and decoding method.

[0016] According to another aspect of this application, a computer device is provided, including a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described video encoding and decoding method.

[0017] According to an embodiment of this application, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned video encoding / decoding method. Attached Figure Description

[0018] The above and other objects, features, and advantages of this application will become apparent from the following detailed description of exemplary embodiments in conjunction with the accompanying drawings. Various features in the drawings are not drawn to scale because the illustrations are for clarity, to facilitate understanding of the technical solutions of this application by those skilled in the art in conjunction with the detailed description. In the drawings:

[0019] Figure 1 It is a networked computer environment according to at least one embodiment;

[0020] Figure 2 It is a hierarchical temporal structure for loop filter / inter-frame prediction according to at least one embodiment;

[0021] Figure 3 It is an operation flowchart of the steps for encoding and decoding video data executed by a program according to at least one embodiment;

[0022] Figure 4According to at least one embodiment Figure 1 A block diagram showing the internal and external components of the computer and server;

[0023] Figure 5 It includes, according to at least one embodiment Figure 1 A block diagram of an exemplary cloud computing environment for the computer equipment shown; and

[0024] Figure 6 According to at least one embodiment Figure 5 The diagram shows a block diagram of the functional layers of an exemplary cloud computing environment. Detailed Implementation

[0025] This application discloses specific embodiments of the claimed structures and methods. However, it should be understood that the disclosed embodiments are merely examples of the claimed structures and methods that can be embodied in various forms. These structures and methods can be embodied in many different forms and should not be construed as limited to the exemplary embodiments described herein. Rather, these exemplary embodiments are provided to make this application more comprehensive and complete, and to fully convey the scope to those skilled in the art. Details of well-known features and techniques may be omitted in the specification to avoid unnecessarily obscuring the presented embodiments.

[0026] The embodiments generally relate to the field of data processing, and more specifically, to video processing. The exemplary embodiments described below provide a system, method, and computer program that, among other things, encode and / or decode video based on a hierarchical temporal structure using loop filters / inter-frame prediction. Therefore, some embodiments improve computational efficiency by increasing the efficiency of video encoding and decoding.

[0027] One objective of video encoding and decoding is to reduce redundancy in the input video signal through compression. Compression can help reduce bandwidth or storage space requirements, in some cases by two orders of magnitude or more. Lossless compression, lossy compression, and combinations thereof can be employed. Lossless compression refers to the technique of reconstructing an exact copy of the original signal from the compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for the intended application. Lossy compression is widely used for video. The amount of distortion tolerated depends on the application; for example, users of some consumer streaming applications may tolerate higher distortion than users of television streaming applications. The achievable compression ratio can be reflected in the higher allowable / tolerable distortion, resulting in a higher compression ratio.

[0028] The predicted signal is constructed using spatiotemporal pixel neighborhoods to obtain the corresponding residuals, which are then used for subsequent transformation, quantization, and entropy coding. On the other hand, the essence of neural networks (NNs) is to extract different levels of spatiotemporal stimuli by analyzing spatiotemporal information from the receptive fields of neighboring pixels. The ability to explore highly nonlinear and nonlocal spatiotemporal correlations offers promising opportunities for significantly improving compression quality.

[0029] However, when utilizing information from multiple adjacent video frames, a caveat arises from complex motion caused by moving cameras and dynamic scenes. Traditional block-based motion vectors do not perform well for non-translational motion. Learning-based optical flow methods can provide accurate motion information at the pixel level, but unfortunately, this is error-prone, especially along the boundaries of moving objects. In some hybrid inter-frame prediction, neural network-based models can be used to implicitly handle arbitrarily complex motion in a data-driven manner.

[0030] Therefore, when using NN-based models as loop filters (LF) or for inter-frame prediction tools, it is advantageous to select different frames as reference frames to apply the LF or generate intermediate frames, which can better balance performance and encoding / decoding runtime.

[0031] Various aspects are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer-readable storage media of various embodiments. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0032] The exemplary embodiments described below provide a system, method, and computer program for video encoding and decoding using reference image samples instead of padding. Figure 1 This is a functional block diagram of a networked computer environment, illustrating a multimedia processing system 100 (hereinafter referred to as the "System") for video encoding and decoding. It should be understood that... Figure 1 The illustrations provided are merely for illustrative purposes and do not imply any limitation on the environments in which different embodiments may be implemented. Various modifications may be made to the described environment based on design and implementation requirements.

[0033] System 100 may include computer 102 and server computer 114. Computer 102 may communicate with server computer 114 via communication network 110 (hereinafter referred to as the "network"). Computer 102 may include processor 104 and software program 108, which is stored in data storage device 106 and is capable of interfacing with a user and communicating with server computer 114. Reference will be made below. Figure 4The computer 102 discussed may include internal component 800A and external component 900A, and the server computer 114 may include internal component 800B and external component 900B. The computer 102 may be, for example, a mobile device, telephone, personal digital assistant, netbook, laptop computer, tablet computer, desktop computer, or any type of computing device capable of running programs, accessing networks, and accessing databases.

[0034] Server computer 114 can also run in cloud computing service models, such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS), as described below. Figure 5 and Figure 6 The server computer 114, as discussed, can also be located in a cloud computing deployment model, such as a private cloud, community cloud, public cloud, or hybrid cloud.

[0035] The server computer 114, which can be used for video encoding and decoding, is capable of running a DMVR video encoding and decoding program 116 (hereinafter referred to as the "program"), which can interact with the database 112. See below for further details. Figure 3 The video codec program method will be explained in more detail below. In one embodiment, computer 102 may operate as an input device including a user interface, while program 116 may run primarily on server computer 114. In an alternative embodiment, program 116 may run primarily on at least one computer 102, while server computer 114 may be used to process and store the data used by program 116. It should be noted that program 116 may be a standalone program or may be integrated into a larger video codec program.

[0036] However, it should be noted that in some cases, the processing of program 116 can be shared between computer 102 and server computer 114 at any ratio. In another embodiment, program 116 can run on more than one computer, server computer, or some combination of computers and server computers, for example, multiple computers 102 communicating with a single server computer 114 via network 110. In another embodiment, for example, program 116 can run on multiple server computers 114 communicating with multiple client computers via network 110. Alternatively, the program can run on a network server communicating with a server and multiple client computers via a network.

[0037] Network 110 may include wired connections, wireless connections, fiber optic connections, or some combination thereof. Typically, network 110 may be any combination of connections and protocols supporting communication between computer 102 and server computer 114. Network 110 may include various types of networks, such as local area networks (LANs), wide area networks (WANs) such as the Internet, telecommunications networks such as the Public Switched Telephone Network (PSTN), wireless networks, public switched networks, satellite networks, cellular networks (e.g., fifth-generation (5G), Long Term Evolution (LTE), third-generation (3G), Code Division Multiple Access (CDMA), etc.), public land mobile networks (PLMNs), metropolitan area networks (MANs), private networks, ad hoc networks, intranets, fiber optic-based networks, etc., and / or combinations of these or other types of networks.

[0038] Figure 1 The number and arrangement of devices and networks shown are provided as examples. In reality, with... Figure 1 Compared to the devices and networks shown, there can be more devices and / or networks, fewer devices and / or networks, different devices and / or networks, or different arrangements of devices and / or networks. Furthermore, Figure 1 The two or more devices shown can be implemented in a single device, or Figure 1 The single device shown can be implemented as multiple distributed devices. Alternatively or additionally, a group of devices in system 100 (e.g., at least one device) can perform at least one function described as being performed by another group of devices in system 100.

[0039] Now for reference Figure 2 The paper describes a hierarchical temporal structure 200 for loop filters / inter-frame prediction. In video encoding and decoding, the hierarchical structure 200 can employ neural network-based loop filters or inter-frame prediction, and more specifically, determines the number / index of frames to be used as reference frames for frame generation or enhancement using neural network models in video encoding and decoding of I / P / B frames respectively. Assume the input video x comprises multiple image frames. (For example, 1 to 16). In the first motion estimation step, the frame is partitioned into spatial blocks, each block can be iteratively partitioned into smaller blocks, and the current frame x is calculated for each block. t With a set of previously reconstructed frames A set of motion vectors m between t Note that the subscript t indicates the current t-th encoding cycle, which may not match the display order (timestamp) of the image frames. Furthermore, It contains frames from multiple previous encoding cycles. Then, in the second motion compensation step, for the current frame... The current codec block in the memory is based on motion vector m. t Copy previous The corresponding pixels are used to obtain the prediction block, and the residual r between the original block and the prediction block can be obtained. t In the third step, the residual r is transformed. t Then quantize the residual r t .

[0040] The quantization step provides the quantized transform block. Through entropy encoding, the motion vector m is... t The quantized transform blocks are encoded into a bitstream and sent to the decoder. Then, on the decoder side, the decoded blocks are subjected to inverse transform and dequantization (typically via inverse transform, such as IDCT with dequantization coefficients) to obtain the recovered residuals. Then Add it back to the predictor block to obtain the reconstruction block. Additional components are further used to improve the reconstruction. Visual quality. Typically, at least one of the following enhancement modules can be selected to address this. These include deblocking filters (DF), sample adaptive offset (SAO), and adaptive loop filters (ALF).

[0041] In HEVC, VVC, or other video coding frameworks or standards, decoded images can be included in a Reference Picture List (RPL) and used as reference images for motion compensation prediction and other parametric prediction, allowing at least one of these images to be encoded or decoded in the order of encoding or decoding. Alternatively, decoded portions of the current image can be used for intra-frame prediction or intra-frame block copying to encode or decode different regions or blocks of the current image.

[0042] In the example, at least one virtual reference can be generated and included in the RPL in both the encoder and decoder, or only in the decoder. The virtual reference image can be generated by at least one process, including signal processing, spatial or temporal filtering, scaling, weighted averaging, upsampling / downsampling, pooling, recursive processing with memory, linear system processing, nonlinear system processing, neural network processing, deep learning-based processing, AI processing, pre-trained network processing, machine learning-based processing, online trained network processing, or combinations thereof. For the process generating at least one virtual reference, zero or more forward reference images, which precede the current image in output / display order and encoding / decoding order, and zero or more backward reference images, which follow the current image in output / display order but precede the current image in encoding / decoding order, are used as input data. The output of this process is a virtual / generated image, which will be used as a new reference image. When this new reference image is selected to predict the encoded / decoded blocks in the current image, conventional motion compensation techniques can be applied.

[0043] In the example, the NN-based method can be applied to in-loop filter design, simultaneously at the strip / CTU level on each frame, combined with at least one of the aforementioned additional components (e.g., DF, SAO, ALF, CCALF, etc.), or replacing at least one of the aforementioned additional components (e.g., DF, SAO, ALF, CCALF, etc.). In application, the reconstructed current image is used as input data for at least one NN-based model to generate an NN-enhanced filtered image. For each block or CTU, it can be decided whether to select this NN-enhanced filtered image as the post-filtering result or to use a conventional filtering method.

[0044] For different scenarios, at least one neural network (NN) model can be selected as the NN-based video codec tool to be applied to images that meet certain conditions. In one embodiment, for different quantization parameters (QPs), at least one NN-based model can be selected in the NN-based video codec tool. In other words, the NN-based video codec tool can use a single model for all QPs, or a separate model for each QP, or it can use separate models, one of which is assigned to a set of QP ranges.

[0045] In at least one embodiment, for images at different classification levels, at least one NN-based model will be used in the NN-based video codec tool. In other words, the NN-based video codec tool can use a single model for all frames, or use separate models for different frames where the Proof of Concept (POC) meets certain conditions. For example, Figure 1 The illustration shows an example of a hierarchical structure used for loop filters / inter-frame prediction. One model is designed for frames with a hierarchical level ID of 1; another model is designed for frames with a hierarchical level ID of 2; and so on. In another example, multiple models can be applied in a neural network-based video codec tool for images with different hierarchical level IDs, either even or odd. In other words, one model is designed for frames with even hierarchical level IDs; another model is designed for frames with odd hierarchical level IDs.

[0046] In at least one embodiment, for different frames with different types of Reference Picture Lists (RPLs), at least one NN-based model can be selected in the NN-based video codec tool. In other words, the NN-based video codec tool can use a single model for all frames, or use separate models for different frames whose Reference Picture Lists (RPLs) meet certain conditions. For example, one model is designed for frames whose Reference Picture Lists (RPLs) have a length of 1; another model is designed for frames whose Reference Picture Lists (RPLs) have a length of 2; and so on. In another example, when the reference pictures in the Reference Picture Lists (RPLs) come from different classification levels, multiple models can be applied in the NN-based video codec tool. In other words, one model is designed for frames whose Reference Picture Lists (RPLs) have reference pictures from classification levels 1 and 2; another model is designed for frames whose Reference Picture Lists (RPLs) have reference pictures from classification levels 1, 2, and 3; and so on.

[0047] In at least one embodiment, when different frames have different numbers of reference frames and are input to a neural network (NN)-based model, at least one NN-based model is selected in the NN-based video codec tool. In other words, the NN-based video codec tool can use a single model for all frames, or use separate models for different frames, where these different frames use a specific number of reference frames as input to the NN-based model. For example, one model is designed to select a unique reference frame from the RPL as input to the NN model; another model is designed to select two unique reference frames from the RPL as input to the NN model; and so on. In another example, the reference frames may not need to be rendered in the RPL; however, they can be stored in the DPB, making them accessible to both the encoder and decoder.

[0048] In at least one embodiment, for different frames that take different temporal distances to their reference frames as input to the NN-based model, at least one NN-based model will be selected in the NN-based video codec tool. In other words, the NN-based video codec tool can use a single model for all frames, or use separate models for different frames where the reference frames to these different frames have different temporal distances. For example, one model is designed to use a reference frame (to the current frame) with a temporal distance of 1 as input to the NN model; another model is designed to use a reference frame (to the current frame) with a temporal distance of 2 as input to the NN model; and so on.

[0049] In at least one embodiment, the NN-based video encoding and decoding can be NN-based inter-frame prediction, loop filtering, or both.

[0050] Multi-model selection can be applied to neural network-based encoding / decoding tools in video encoding and decoding. The proposed method determines whether to select at least one neural network-based model as a neural network-based encoding / decoding tool for images under different conditions. The neural network-based encoding / decoding tool can include, but is not limited to, neural network-based loop filtering and neural network-based virtual reference images for inter-frame prediction. Several examples further detailing the proposed method are provided below.

[0051] In one example, given Figure 1 In the hierarchical GOP structure, a single model can be used for all levels of images, or for each level, a separate model can be used for frames with different hierarchical level IDs (different POC conditions). As an example, for the current image (POC=3), which has different hierarchical levels and different scenes or conditions than another image (POC=2), these two images can use a common model from a NN-based encoding / decoding tool or a separate model.

[0052] In another example, given Figure 1 The hierarchical GOP structure in the model provides a reference picture list (RPL) for the current image (POC=3) and a reference picture list (RPL) for the images (POC=0, 2, 4, 8). Similarly, for the current image (POC=10), it provides a reference picture list (RPL) for the images (POC=8, 12, 16). These two frames have different reference picture lists (RPLs) from different hierarchical levels, and these two images can be used as a common model or separate models in a neural network-based encoding / decoding tool.

[0053] In another example, given Figure 1 The hierarchical GOP structure in the model provides a list of reference images (RPL) for the current image (POC=3) (POC=0, 2, 4, 8), allowing different numbers of reference frames to be fed into the NN-based model. When using different numbers of reference frames as input, it can be used as a common model or a separate model as an NN-based encoding / decoding tool.

[0054] In another example, given Figure 1 The hierarchical GOP structure in the image has a list of reference images (RPL) for the current image (POC=3) (POC=0, 2, 4, 8); for each reference frame, it can have different temporal distances to the current image; for different temporal distances of reference frames relative to the current image, it can be applied using a common model or a separate model from a neural network-based encoding / decoding tool.

[0055] Now for reference Figure 3The flowchart illustrates the steps of method 300 performed by a program that encodes and decodes video data.

[0056] At 302, method 300 may include receiving video data, the video data including at least one frame.

[0057] At 304, method 300 may include determining at least one quantization parameter associated with the video data to perform frame generation or enhancement on a target frame from the at least one frame.

[0058] At 306, method 300 may include decoding the video data based on the quantization parameters.

[0059] It should be understood that Figure 3 The illustrations provided are merely illustrative of one implementation and do not imply any limitations on how different embodiments may be implemented. Various modifications may be made to the described environment based on design and implementation requirements.

[0060] Figure 4 This is according to an exemplary embodiment. Figure 1 The block diagram 400 depicts the internal and external components of the computer. It should be understood that... Figure 4 The illustrations provided are merely illustrative of one implementation method and do not imply any limitation on the environments in which different embodiments may be implemented. Various modifications may be made to the described environment based on design and implementation requirements.

[0061] Computer 102 ( Figure 1 ) and server computer 114 ( Figure 1 ) can include Figure 4 The corresponding sets of internal components 800A, 800B and external components 900A, 900B are shown. Each set of internal components 800 includes at least one processor 820 connected to at least one bus 826, at least one computer-readable random access memory (RAM) 822 and at least one computer-readable read-only memory (ROM) 824 including at least one operating system 828, and at least one computer-readable tangible storage device 830.

[0062] Processor 820 is implemented in hardware, firmware, or a combination of hardware and software. Processor 820 is a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or other type of processing component. In some embodiments, processor 820 includes at least one processor capable of being programmed to perform functions. Bus 826 includes components that allow communication between internal components 800A and 800B.

[0063] At least one operating system 828, and a server computer 114 ( Figure 1 Software program 108 on ) Figure 1 ) and DMVR video codec 116 ( Figure 1 All of these are stored on at least one corresponding computer-readable tangible storage device 830, for execution by at least one corresponding processor 820 via at least one corresponding RAM 822 (which typically includes a cache memory). Figure 4 In the illustrated embodiment, each computer-readable tangible storage device 830 is a disk storage device of an internal hard disk drive. Alternatively, each computer-readable tangible storage device 830 is a semiconductor storage device, such as ROM 824, erasable programmable read-only memory (EPROM), flash memory, optical disk, magneto-optical disk, solid-state disk, optical disc (CD), digital versatile optical disc (DVD), floppy disk, cassette tape, magnetic tape, and / or other types of non-volatile computer-readable tangible storage devices capable of storing computer programs and digital information.

[0064] Each set of internal components 800A, 800B also includes a read / write (R / W) drive or interface 832 for reading from or writing to at least one portable computer-readable tangible storage device 936 (e.g., CD-ROM, DVD, Memory Stick, magnetic tape, disk, optical disc, or semiconductor storage device). Such as software program 108 ( Figure 1 ) and DMVR video codec 116 ( Figure 1 The software program can be stored on at least one corresponding portable computer-readable tangible storage device 936, read and loaded into the corresponding hard disk drive 830 via the corresponding R / W drive or interface 832.

[0065] Each group of internal components 800A and 800B also includes a network adapter or interface 836, such as a TCP / IP adapter card, a wireless Wi-Fi interface card, or a 3G, 4G, or 5G wireless interface card, or other wired or wireless communication links. Server computer 114 ( Figure 1 Software program 108 on ) Figure 1 ) and DMVR video codec 116 ( Figure 1 The data can be downloaded from an external computer to computer 102 via a network (e.g., the Internet, a local area network or other networks, a wide area network) and a corresponding network adapter or interface 836. Figure 1The network includes a network adapter or interface 836 and a server computer 114. Software program 108 and DMVR video codec program 116 on server computer 114 are loaded into the corresponding hard disk drive 830 from the network adapter or interface 836. The network may include copper wire, fiber optic, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers.

[0066] Each set of external components 900A, 900B may include a computer monitor 920, a keyboard 930, and a computer mouse 934. External components 900A, 900B may also include a touchscreen, virtual keyboard, touchpad, pointing device, and other human-machine interface devices. Each set of internal components 800A, 800B also includes a device driver 840 for interfacing with the computer monitor 920, keyboard 930, and computer mouse 934. Device driver 840, R / W driver or interface 832, and network adapter or interface 836 include hardware and software (stored in storage device 830 and / or ROM 824).

[0067] It should be understood that although this application includes a detailed description of cloud computing, the embodiments described herein are not limited to a cloud computing environment. Rather, some embodiments can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0068] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing power, memory, storage, applications, virtual machines, and services) that can be rapidly configured and deployed with minimal management effort or interaction with service providers. A cloud model can include at least five features, at least three service models, and at least four deployment models.

[0069] The characteristics are as follows:

[0070] On-demand self-service: Cloud users can automatically and unilaterally provide computing functions, such as server time and network storage, as needed, without having to interact manually with the service provider.

[0071] Broad network access: Functionality is available through the network and accessed via standard mechanisms that facilitate the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and personal digital assistants).

[0072] Resource pooling: This uses a multi-tenant model to pool a provider's computing resources to serve multiple users, dynamically allocating and reallocating different physical and virtual resources based on demand. Location independence means that users typically do not have control or knowledge of the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0073] Rapid elasticity: Features that can be configured quickly and elastically, automatically configuring to scale out quickly in certain situations and releasing to scale in quickly in others. To the user, the available configuration options often appear unlimited, and can be purchased in any quantity at any time.

[0074] Measurable services: Cloud systems automatically control and optimize resource usage by leveraging metering at some level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be viewed, controlled, and reported, providing transparency to both the service provider and the user.

[0075] The service model is as follows:

[0076] Software as a Service (SaaS): This provides users with the functionality to use applications from a provider that run on cloud infrastructure. The applications can be accessed from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Users do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application functionality, but may be limited by user-specific application configuration settings.

[0077] Platform as a Service (PaaS): This provides users with the ability to deploy user-created or acquired applications onto cloud infrastructure using provider-supported programming languages ​​and tools. Users do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage; instead, they control the deployed applications and the configuration of any possible application hosting environments.

[0078] Infrastructure as a Service (IaaS): This provides users with processing, storage, networking, and other basic computing resources, enabling them to deploy and run any software, including operating systems and applications. Users do not manage or control the underlying cloud infrastructure; instead, they control the operating system, storage, deployed applications, and possibly limited control over selected network components (e.g., host firewalls).

[0079] The deployment model is as follows:

[0080] Private cloud: A cloud infrastructure that operates solely for an organization. It can be managed by the organization or a third party, and can exist internally or externally.

[0081] Community cloud: A cloud infrastructure shared by multiple organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist internally or externally.

[0082] Public cloud: Cloud infrastructure available to the general public or large industrial groups and owned by organizations that sell cloud services.

[0083] Hybrid cloud: A cloud infrastructure consisting of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies to enable data and application portability (e.g., cloud bursts for load balancing between clouds).

[0084] Cloud computing environments are service-oriented, emphasizing statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure of a network of interconnected nodes.

[0085] Reference Figure 5 The figure illustrates an exemplary cloud computing environment 500. As shown, the cloud computing environment 500 includes at least one cloud computing node 10, to which local computing devices used by cloud users (e.g., personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer devices 54N) can communicate. The cloud computing nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in at least one network, such as the private cloud, community cloud, public cloud, hybrid cloud, or a combination thereof described above. This allows the cloud computing environment 500 to provide infrastructure, platform, and / or software as a service without requiring cloud users to maintain resources for these services on their local computing devices. It should be understood that... Figure 5 The types of computing devices 54A-N shown are merely exemplary, and cloud computing node 10 and cloud computing environment 500 can communicate with any type of computer device via any type of network and / or network-addressable connectivity (e.g., using a web browser).

[0086] Reference Figure 6 It demonstrates the 500 (cloud computing environment) Figure 5 This provides a set of functional abstraction layers, 600. It should be understood that... Figure 6 The components, layers, and functions shown are merely exemplary, and the embodiments are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0087] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a host 61, servers 62 and 63 based on a RISC (Reduced Instruction Set Computer) architecture, a blade server 64, a storage device 65, and a network and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0088] The virtual layer 70 provides an abstraction layer from which examples of the following virtual entities can be provided: virtual server 71, virtual storage 72, virtual network including virtual private network 73, virtual application and operating system 74, and virtual client 75.

[0089] In one example, management layer 80 can provide the following functions: Resource Provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks in the cloud computing environment. Metering and Pricing 82 provides cost detection and billing or invoicing for the consumption of these resources when they are used in the cloud computing environment. In one example, these resources may include application software licenses. Security provides authentication for cloud users and tasks and protection for data and other resources. User Access 83 provides access to the cloud computing environment for users and system administrators. Service Level Management 84 provides allocation and management of cloud computing resources to meet the required service level. Service Level Agreement (SLA) Planning and Implementation 85 provides pre-scheduling and acquisition of cloud computing resources for future needs anticipated according to the SLA.

[0090] Workload layer 90 provides examples of functionalities that can leverage cloud computing environments. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 91, software development and lifecycle management 92, virtual classroom teaching implementation 93, data analysis and processing 94, transaction processing 95, and DMVR video encoding and decoding 96. DMVR video encoding and decoding 96 can encode and decode video using reference image samples instead of padding.

[0091] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of technical detail integration. In embodiments of this application, a computer-readable medium may include a non-volatile computer-readable storage medium (or medium) storing computer-readable program instructions thereon that cause a processor to perform operations.

[0092] Computer-readable storage media can be tangible devices that can retain and store instructions for use by instruction execution devices. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile optical disc (DVD), memory sticks, floppy disks, mechanical encoding devices (such as punch-cards or slotted structures on which instructions are recorded), and any suitable combination of the foregoing. The computer-readable storage media used in this application should not be construed as being itself a volatile signal, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0093] The computer-readable program instructions described in this application can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the corresponding computing / processing device.

[0094] Computer-readable program code / instructions used to perform operations can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​(such as the "C" programming language) or similar programming languages. The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network (including local area network (LAN) or wide area network (WAN)) or can be connected to an external computer (e.g., via the Internet through an Internet service provider). In some embodiments, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can execute computer-readable program instructions by utilizing state information of computer-readable program instructions to personalize the electronic circuits and thereby perform aspects or operations.

[0095] These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions are executed by the processor of the computer or other programmable data processing apparatus to create means for implementing the function / action specified in at least one block of a flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored comprises an article of writing including instructions for implementing aspects of the function / action specified in at least one block of a flowchart and / or block diagram.

[0096] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device, thereby producing a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the function / action specified in at least one block of a flowchart and / or block diagram.

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including at least one executable instruction for implementing a specified logical function. The methods, computer devices, and computer-readable media may include more blocks, fewer blocks, different blocks, or blocks arranged differently compared to those depicted in the figures. In some alternative implementations, the functions marked in the blocks may occur in a non-linear order as indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed simultaneously or substantially simultaneously, or the blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action, or executes a combination of dedicated hardware and computer instructions.

[0098] It is evident that the systems and / or methods described in this application can be implemented in various forms of hardware, firmware, or combinations of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limited to these implementations. Therefore, this application describes the operation and behavior of these systems and / or methods without reference to specific software code—it should be understood that software and hardware can be designed based on the description in this application to implement these systems and / or methods.

[0099] Unless explicitly stated otherwise, no element, action, or instruction used in this application shall be construed as critical or essential. Furthermore, as used in this application, the articles “a” and “an” are intended to include at least one item and may be used interchangeably with “at least one.” Additionally, as used in this application, the term “set” is intended to include at least one item (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with “at least one.” The term “an” or similar language is used where only one item is desired. Furthermore, as used in this application, the terms “has,” “have,” “having,” etc., are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “at least partially based on,” unless otherwise explicitly stated.

[0100] Various aspects and embodiments have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Even though combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically described in the claims and / or not disclosed in the specification. While each dependent claim listed below may be directly subordinated to only one claim, the disclosure of possible implementations includes combinations of each dependent claim with each other claim in the claim set. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments of this application, a practical application of technology found in the market or a technical improvement thereof, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A video decoding method, characterized in that, include: Receive video data, the video data including multiple frames, the multiple frames corresponding to multiple rating levels; Determine at least one quantization parameter associated with the video data; Based on the quantization parameters, at least one neural network model is selected to generate or enhance the target frame from the plurality of frames. and, The video data is decoded based on the quantization parameters. Among the multiple frames, the first frame and the second frame correspond to different classification levels, and different neural network models are used for the first frame and the second frame; The first reference image list of the first frame contains multiple reference frames from different hierarchical levels; The second reference image list of the second frame contains multiple reference frames from different hierarchical levels, and the first reference image list is different from the second reference image list; Within the same classification level, the difference in the Picture Order Count (POC) between any two adjacent frames is the same.

2. The method according to claim 1, characterized in that, The target frame corresponds to at least one of I-frames, P-frames, and B-frames.

3. The method according to claim 1, characterized in that, Decoded video data is included in the list of reference images.

4. The method according to claim 3, characterized in that, Also includes: Based on the reference image list, subsequent frames from the plurality of frames are decoded using motion compensation prediction, intra-frame prediction, or intra-frame block copying.

5. The method according to claim 3, characterized in that, Also includes: Generate at least one virtual reference frame and include the at least one virtual reference frame in the list of reference images.

6. The method according to claim 5, characterized in that, The generation of at least one virtual reference frame includes: The virtual reference frame is generated based on at least one of the following: signal processing, spatial or temporal filtering, scaling, weighted averaging, upsampling / downsampling, pooling, recursive processing with memory, linear system processing, nonlinear system processing, neural network processing, deep learning-based processing, artificial intelligence (AI) processing, pre-trained network processing, machine learning-based processing, and online trained network processing.

7. A video decoding device, characterized in that, include: A receiving module is used to receive video data, the video data including multiple frames, the multiple frames corresponding to multiple rating levels; A determining module is used to determine at least one quantization parameter associated with the video data; The selection module is used to select at least one neural network model based on the quantization parameters to generate or enhance the target frame from the plurality of frames; and, A decoding module is used to decode the video data based on the quantization parameters; Among the multiple frames, the first frame and the second frame correspond to different classification levels, and different neural network models are used for the first frame and the second frame; The first reference image list of the first frame contains multiple reference frames from different hierarchical levels; The second reference image list of the second frame contains multiple reference frames from different hierarchical levels, and the first reference image list is different from the second reference image list; Within the same classification level, the difference in the Picture Order Count (POC) between any two adjacent frames is the same.

8. The apparatus according to claim 7, characterized in that, The target frame corresponds to at least one of I-frames, P-frames, and B-frames.

9. The apparatus according to claim 7, characterized in that, Decoded video data is included in the list of reference images.

10. The apparatus according to claim 9, characterized in that, The decoding module is further configured to decode subsequent frames from the plurality of frames based on the reference image list, using motion compensation prediction, intra-frame prediction, or intra-frame block copying.

11. The apparatus according to claim 9, characterized in that, Also includes: A generation module is used to generate at least one virtual reference frame and include the at least one virtual reference frame in the reference image list.

12. The apparatus according to claim 11, characterized in that, The generation module is used to generate the virtual reference frame based on at least one of the following: signal processing, spatial or temporal filtering, scaling, weighted averaging, upsampling / downsampling, pooling, recursive processing with memory, linear system processing, nonlinear system processing, neural network processing, deep learning-based processing, artificial intelligence (AI) processing, pre-trained network processing, machine learning-based processing, and online trained network processing.

13. A video encoding method, characterized in that, include: Generate video data, which includes multiple frames and the multiple frames correspond to multiple rating levels; Determine at least one quantization parameter associated with the video data; Based on the quantization parameters, at least one neural network model is selected to generate or enhance the target frame from the plurality of frames. and, The video data is encoded based on the quantization parameters; Among the multiple frames, the first frame and the second frame correspond to different classification levels, and different neural network models are used for the first frame and the second frame; The first reference image list of the first frame contains multiple reference frames from different hierarchical levels; The second reference image list of the second frame contains multiple reference frames from different hierarchical levels, and the first reference image list is different from the second reference image list; Within the same classification level, the difference in the Picture Order Count (POC) between any two adjacent frames is the same.

14. A video encoding device, characterized in that, include: A generation module is used to generate video data, the video data including multiple frames, the multiple frames corresponding to multiple rating levels; A determining module is used to determine at least one quantization parameter associated with the video data; The selection module is used to select at least one neural network model based on the quantization parameters to generate or enhance the target frame from the plurality of frames; and, An encoding module is used to encode the video data based on the quantization parameters; Among the multiple frames, the first frame and the second frame correspond to different classification levels, and different neural network models are used for the first frame and the second frame; The first reference image list of the first frame contains multiple reference frames from different hierarchical levels; The second reference image list of the second frame contains multiple reference frames from different hierarchical levels, and the first reference image list is different from the second reference image list; Within the same classification level, the difference in the Picture Order Count (POC) between any two adjacent frames is the same.

15. A method for transmitting a code stream, characterized in that, Perform the video encoding method of claim 13 to generate the bitstream; and transmit the bitstream.

16. A non-volatile computer-readable medium, characterized in that, It stores a computer program for video encoding and decoding, the computer program being used to cause at least one computer processor to perform the method as described in any one of claims 1 to 6, 13, and 15.

17. A computer device, characterized in that, The device includes a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the method as described in any one of claims 1 to 6, 13, and 15.

Images