Video encoding method, computer system and medium for encoding video data

Abstract

A method, computer program and computer system for encoding video data are provided. Reference samples and magnitudes of transform coefficients corresponding to a current block of video data are identified, the current block of video data being input to a neural network. Symbol values ​​associated with the transform coefficients are predicted based on at least the identified reference samples. The video data is encoded or decoded based on the predicted symbol values.

Description

[0001] Priority information

[0002] This application claims priority to U.S. Application No. 17 / 071,582, filed on October 15, 2020, the entire contents of which are incorporated herein by reference Technical Field

[0003] The present disclosure relates generally to the field of data processing, and more particularly, to video encoding and decoding. Background Art

[0004] AOMedia Video 1 (AV1) is an open video coding format designed for transmitting video over the Internet. It was developed as a successor to VP9 by the Alliance for Open Media (AOMedia), which was established in 2015 and includes semiconductor companies, video on demand providers, video content producers, software development companies, and web browser vendors.

[0005] Deep learning is a group of learning methods that try to model data using complex architectures that combine different nonlinear transformations. The cornerstone of deep learning is neural networks, which are combined together to form deep neural networks. An artificial neural network is an application that is nonlinear with respect to its parameters θ, which relate the input x to the output y = f(x,θ). The parameters θ are estimated from the learning samples. Neural networks can be used for regression or classification. There are many types of neural network architectures. Summary of the invention

[0006] Embodiments relate to methods, systems, and computer-readable media for encoding video data. According to one aspect, a method for encoding video data is provided. The method may include: identifying reference samples and transform coefficients corresponding to a current block of video data; identifying reference samples and magnitudes of transform coefficients corresponding to the current block of video data, the current block of video data being input to a neural network; predicting a symbol value associated with the transform coefficient based on at least the identified reference samples; and encoding or decoding the video data based on the predicted symbol value.

[0007] According to another aspect, a computer system for encoding video data is provided. The computer system may include one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored in at least one of the one or more storage devices, the program instructions being executed by at least one of the one or more processors via at least one of the one or more memories, whereby the computer system is capable of performing a method. The method may include: identifying reference samples and transform coefficients corresponding to a current block of video data; identifying reference samples and magnitudes of transform coefficients corresponding to a current block of video data, the current block of video data being from an input of a neural network; predicting a symbol value associated with the transform coefficient based on at least the identified reference sample; and encoding or decoding the video data based on the predicted symbol value.

[0008] According to another aspect, a computer-readable medium for encoding video data is provided. The computer-readable medium may include one or more computer-readable storage devices and program instructions stored on at least one of the one or more tangible storage devices, the program instructions being executable by a processor. The program instructions are executable by a processor to implement a method, which accordingly includes: identifying reference samples and transform coefficients corresponding to a current block of video data; identifying reference samples and magnitudes of transform coefficients corresponding to a current block of video data, the current block of video data being from an input of a neural network; predicting a symbol value associated with the transform coefficient based on at least the identified reference samples; and encoding or decoding the video data based on the predicted symbol value. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] These and other objects, features and advantages will become apparent from the following detailed description of illustrative embodiments read in conjunction with the accompanying drawings. In order to facilitate a clear understanding of the present invention by those skilled in the art in conjunction with the detailed description, the various features of the drawings are not drawn to scale. In the drawings:

[0010] Figure 1 illustrates a networked computer environment according to at least one embodiment;

[0011] Figure 2 is a diagram of an exemplary convolutional neural network according to at least one embodiment;

[0012] Figure 3 is a block diagram of a dense residual network according to at least one embodiment;

[0013] Figure 4 According to at least one embodiment Figure 3 The structural block diagram of the dense residual unit;

[0014] Figure 5A and Figure 5B is an intra prediction mode according to at least one embodiment;

[0015] Figure 6 is a cost calculation according to at least one embodiment;

[0016] Figure 7 is an operational flow diagram illustrating steps performed by a program for encoding video data according to at least one embodiment;

[0017] Figure 8 According to at least one embodiment Figure 1 block diagrams of the internal and external components of the computers and servers shown;

[0018] Fig. 9 According to at least one embodiment, Figure 1 A block diagram of an illustrative cloud computing environment of a computer system is shown; and

[0019] Fig.10 According to at least one embodiment Fig. 9 A block diagram of the functional layers of an illustrative cloud computing environment. DETAILED DESCRIPTION

[0020] The present application discloses detailed embodiments of the claimed structures and methods. However, it is to be understood that the disclosed embodiments are merely exemplary illustrations of the claimed structures and methods that may be implemented in various forms. However, these structures and methods may be embodied in many different forms and should not be construed as being limited to the exemplary embodiments described herein. On the contrary, these exemplary embodiments are provided to make the present disclosure detailed and complete, and will fully convey the scope of protection to those skilled in the art. In the specification, the details of well-known features and technologies are omitted to avoid unnecessarily obscuring the embodiments presented.

[0021] Embodiments relate generally to the field of data processing, and more specifically to video encoding and decoding. The exemplary embodiments described below provide a system, method, and computer program for encoding video data based on predicting the sign values ​​of transform coefficients of the video data. Thus, some embodiments have the ability to improve the computing field by improving coding efficiency based on predicting the signs of transform coefficients of video data using neural networks.

[0022] As mentioned before, deep learning is a group of learning methods that try to model data using complex architectures that combine different nonlinear transformations. The cornerstone of deep learning is neural networks, which are combined together to form deep neural networks. An artificial neural network is an application that is nonlinear with respect to its parameters θ, which relate the input x to the output y = f(x,θ). The parameters θ are estimated from the learning samples. Neural networks can be used for regression or classification. There are many types of neural network architectures. However, deep neural networks are only used for loop filtering and intra-frame prediction. Therefore, in order to improve coding efficiency, it may be advantageous to use neural networks for symbol prediction.

[0023] According to various embodiments, the present application describes various aspects with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer-readable media. It should be understood that each block of the flowchart illustration and / or block diagram and the combination of blocks in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions.

[0024] Reference now Figure 1 , a functional block diagram of a network computer environment shows a video encoding system 100 (hereinafter referred to as "system") for encoding video data based on the sign values ​​of the transform coefficients of the predicted video data. It should be understood that Figure 1 This merely provides an illustration of one implementation and does not imply any limitation with regard to the environments in which different embodiments may be implemented. Many modifications may be made to the described environment depending on design and implementation requirements.

[0025] System 100 may include computer 102 and server computer 114. Computer 102 may communicate with server computer 114 via communication network 110 (hereinafter referred to as "network"). Computer 102 may include processor 104 and software program 108, which is stored on data storage device 106 and is capable of interacting with a user and communicating with server computer 114. Figure 8 As described, the computer 102 may include internal components 800A and external components 900A, respectively, and the server computer 114 may include internal components 800B and external components 900B, respectively. The computer 102 may be, for example, a mobile device, a phone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device capable of running programs, accessing a network, and accessing a database.

[0026] The server computer 114 may also operate in a cloud computing service model, such as software as a service (SaaS), platform as a service (PaaS), or infrastructure as a service (IaaS), as described below with respect to Fig. 9 and Fig.10The server computer 114 may also be located in a cloud computing deployment model, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud.

[0027] The server computer 114, which can be used to encode video data, can run a video encoding program 116 (hereinafter referred to as "the program") that can interact with the database 112. Figure 3 The video encoding program method is described in more detail. In one embodiment, the computer 102 can operate as an input device including a user interface, and the program 116 can be primarily operated on the server computer 114. In an alternative embodiment, the program 116 can be primarily operated on one or more computers 102, and the server computer 114 can be used to process and store data used by the program 116. It should be noted that the program 116 can be a stand-alone program, or the program 116 can be integrated into a larger video encoding program.

[0028] However, it should be noted that in some cases, processing for program 116 may be shared in any proportion between computer 102 and server computer 114. In another embodiment, program 116 may be executed on more than one computer, server computer, or some combination of computers and server computers, such as multiple computers 102 communicating with a single server computer 114 via network 110. In another embodiment, for example, program 116 may be executed on multiple server computers 114 communicating with multiple client computers via network 110. Alternatively, the program may be executed on a network server that communicates with the server and multiple client computers via the network.

[0029] The network 110 may include a wired connection, a wireless connection, a fiber optic connection, or some combination thereof. In general, the network 110 may be any combination of connections and protocols that support communication between the computer 102 and the server computer 114. The network 110 may include various types of networks, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, a telecommunications network such as a public switched telephone network (PSTN), a wireless network, a public switched network, a satellite network, a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a metropolitan area network (MAN), a private network, an ad hoc network, an intranet, a fiber-based network, etc., and / or a combination of these or other types of networks.

[0030] supply Figure 1 The number and configuration of devices and networks shown are examples. Figure 1There may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or Figure 1 In addition, the devices and / or networks shown in FIG. Figure 1 Two or more of the devices shown may be implemented in a single device, or Figure 1 The single device shown may be implemented as multiple distributed devices. Additionally or alternatively, one set of devices (eg, one or more devices) of system 100 may perform one or more functions described as being performed by another set of devices of system 100.

[0031] Reference now Figure 2 , which shows an exemplary convolutional neural network 200. As an example and not a limitation, a neural network architecture is described here. However, it is understood that any neural network architecture with layers different from the examples described in this application can be used. Among other things, the convolutional neural network 200 can receive two inputs: a reconstruction parameter 202 and a quantization parameter (QP) 204, which allows a single set of parameters to be used to accommodate reconstructions with different qualities. For better convergence during training, the two inputs can be normalized. In order to reduce complexity, a simple CNN with 10 layers can be used. The simple CNN may include a cascade layer 206, 7 convolutional layers 208A-208G (each convolutional layer is followed by a rectified linear unit layer), a convolutional layer 210 and a summation layer 212. These layers may be connected layer by layer to form a network. It is understood that the above-mentioned layer parameters may be included in the convolutional layer. By connecting the reconstructed Y value, U value or V value to the summation layer, the network can be regularized to learn the characteristics of the residual between the reconstructed image and the original image.

[0032] According to one or more embodiments, among other things, the input of a neural network may include one or more reference samples of a current block and the magnitude (level or absolute value) of the transform coefficient values of the current block. The neural network may output a predicted sign value of the transform coefficient. In one embodiment, the reference sample may include a reconstructed sample from a spatially adjacent block. In another embodiment, the reference sample may include a reconstructed sample specified by a motion vector from a reference picture. In one embodiment, when reconstructed samples specified by motion vectors from multiple reference pictures may be involved, the input of the neural network is also related to the POC distance. The POC distance generally refers to the POC difference between the current picture and the reference picture. In one embodiment, the transform coefficient that can be used as the input of the neural network may be a de - quantized transform coefficient. In one embodiment, the output of the neural network may be the predicted sign value of a finite set of transform coefficients. In one example, the finite set of transform coefficients includes low - frequency coefficients. That is, the transform coefficients are located in a predefined upper - left region of the coefficient block (e.g., upper - left 1x1, 2x2, 4x4, 8x8, 16x16). In one example, the finite set of transform coefficients includes the first N coefficients in the forward scan order, where N may be a predefined number.

[0033] In one embodiment, the input of the neural network further includes an indicator of the main transform type (kernel). In one embodiment, the input of the neural network further includes an indicator of the secondary transform type or kernel (including whether the secondary transform can be applied). In one embodiment, an estimate of the reconstructed sample of the current block may be obtained using a hypothesis of the sign value, and the estimated reconstructed sample of the current block together with neighboring reconstructed values may be used as the input of the neural network. The output of the network may be a score, where a higher score means a higher likelihood that the hypothesis of the sign value is selected by the current block, and vice versa. In one embodiment, if the secondary transform is applied, the number of transform coefficients used as the input of the neural network will change. In one example, if the secondary transform can also be applied to the current block, a smaller number (<N) of transform coefficients in the forward scan order will be used.

[0034] According to one or more embodiments, the input of the neural network may include, among other things, reference samples of the current block and reconstructed (or residual) samples of the current block generated using at least one of the hypotheses of symbol values. The neural network may output predicted symbol values ​​for the transform coefficients. In one embodiment, for the reconstructed (or residual) samples of the current block generated using at least one of the hypotheses of symbol values ​​(which may be used as input to the neural network), only a subset of the reconstructed (or residual) samples of the current block may be used. The subset may include boundary samples of the current block, for example, M rows above and / or N columns to the left. Example values ​​of M and N may include 1, 2, 3, and 4. The values ​​of M and N may depend on encoding information, for example, block size. For different main transform types and / or auxiliary transform types, different neural networks may be applied to predict coefficient symbol values.

[0035] Reference now Figure 3 , which shows a block diagram of an exemplary dense residual network (DRN) 300. The network structure may include N dense residual units (DRUs) 302A-302N, and M may represent the number of convolution kernels. For example, as a trade-off between computational efficiency and performance, N may be set to 4 and M may be set to 32. The normalized QP map 304 may be concatenated with the reconstructed frame as an input to the DRN 300.

[0036] Reference now Figure 4 , which shows Figure 3 302N. The DRU can directly propagate the input to the subsequent unit through a shortcut. To further reduce the computational cost, a 3×3 depthwise separable convolution (DSC) layer is used in the DRU. The output of the network can have 3 channels, corresponding to Y, Cb, Cr. The filter can be applied to intra-frame pictures and inter-frame pictures. An additional flag can be signaled for each CTU to indicate the on / off state of the DRNLF.

[0037] Reference now Figure 5A and Figure 5B , which depicts diagrams of intra-frame prediction modes 500A and 500B. Intra-frame prediction modes can be used to generate intra-picture prediction signals on rectangular blocks in future video codecs. These intra-frame prediction modes perform the following two main steps. A set of features are extracted from the decoded samples. These features are used to select an affine linear combination of predefined image patterns as the prediction signal. In addition, a specific signaling scheme can be used for intra-frame prediction modes.

[0038] On a given MxN block 502A with M≤32 and N≤32, a luma prediction signal pred is generated by processing a set of reference samples r by a neural network 506A. The reference samples r consist of K rows of size N+K above the block 502A and K columns of size M to the left of 502A. The number K may depend on M and N, for example, K may be set to 2 for all M and N.

[0039] As described below, the neural network 506A first extracts a vector ftr of features from the reconstructed sample r. If d0=K*(N+M+K) represents the number of samples r, then r is considered as a vector in a real vector space of d0 dimensions. For fixed integer square matrices A1 and A2, which have d0 rows and d0 columns, respectively, and for fixed integer bias vectors b1 and b2 of d0 dimensions, t1=ρ(A1·r+b1) is first calculated.

[0040] Here, · denotes the ordinary matrix-vector product. In addition, the function ρ is an integer approximation of the ELU function ρ0, where the latter function is defined on a p-dimensional vector υ by:

[0041]

[0042] Where ρ0(v) i and i Denotes the i-th component of the vector. Apply similar operations to t1 and calculate:

[0043] t2=ρ(A2·t1+b2).

[0044] For a fixed integer d1, where 0≤d1≥d0, there may be a predefined integer matrix A3 having d1 rows and d0 columns, and one or more bias weights 504A, for example, a predefined integer bias vector b3 of dimension d1, such that the feature vector ftr is calculated as follows:

[0045] ftr=ρ(A3·t2+b3).

[0046] The value of d1 depends on M and N. At this time, it is assumed that d1 = d0.

[0047] In the feature vector ftr, an affine linear mapping (followed by a standard clipping operation Clip depending on the bit depth) is used to generate the final prediction signal. Therefore, there is a predefined matrix A4 with M*N rows and d1 columns and a predefined bias vector b4 of M*N dimensions so that the calculation

[0048] pred=Clip(A4·ftr+b4). Now refer to Figure 5B, n different intra prediction modes 508B can be used, where, for max(M,N) < 32, n is set to 35, and in other cases, n is set to 11. Therefore, the index predmode (prediction mode, where 0 ≤ predmode ≤ n) will be signaled by the encoder and parsed by the decoder, and the following syntax can be used. There is n = 3 + 2 k , where, if max(M,N) = 32, then k = 3; in other cases, k = 5. In the first step, the following code is used to signal the index predIdx (0 ≤ predIdx ≤ n). First, a single binary number (one bin) encodes whether preIdx < 3. If preIdx < 3, then a second binary number (second bin) encodes whether preIdx = 0, and if preIdx ≠ 0, then another binary number (another bin) encodes whether preIdx is equal to 1 or 2. If preIdx ≥ 3, then k binary numbers are used to signal the value of preIdx in a canonical way.

[0049] From the index preIdx, the actual index predmode is derived using a fully connected neural network 506B with one hidden layer, and the hidden layer takes the reconstructed sample r′ as input. The reconstructed sample r′ is located on two rows of size N + 2 above the block 502B and two columns of size M to the left of the block 502B.

[0050] The reconstructed sample r′ is regarded as a vector in a 2*(M + N + 2)-dimensional real vector space. There is a fixed square matrix A1′, which has 2*(M + N + 2) rows or 2*(M + N + 2) columns respectively, and there may be one or more bias weights 504B. For example, there is a fixed bias vector b1′ in a 2*(M + N + 2)-dimensional real vector space, so that

[0051] t1′ = ρ(A1′·r′ + b1′) can be calculated.

[0052] In an n-dimensional real vector space, there may be a matrix A2′ with n rows and 2*(M + N + 2) columns and a fixed bias vector b2′, so that

[0053] lgt = A′2·t′1 + b′2 can be calculated.

[0054] The index predmode is now derived as the position of the preIdx-th largest component of lgt. Here, for k ≠ 1, if two components (lgt) k and (lgt) l are equal, then when k < l, (lgt) k is considered to be greater than (lgt) l, and in other cases, it is considered that (lgt) l Greater than (lgt) k .

[0055] Reference now Figure 6 , an exemplary cost calculation 600 is described. For entropy coding of transform coefficients, a bypass mode is usually used to encode the coefficient sign separately from the amplitude / level (absolute value of the coefficient value). This means that each coefficient symbol costs one bit to encode, which is expensive. In order to improve the efficiency of entropy coding of coefficient symbols, a symbol prediction method can be used. Instead of signaling the symbol value, a flag indicating whether the predicted symbol is the same as the actual symbol is entropy encoded with context. Since larger level values ​​lead to more accurate predicted symbol values, the value of the context depends on the level of the coefficient (absolute value of the coefficient value).

[0056] First, a set of transform coefficients are identified for which associated symbols need to be predicted. Then a set of hypotheses are created for the predicted symbol values ​​of these coefficients. For example, for three coefficients, the number of hypotheses can be as many as 8 (i.e., 2^3). To predict the symbol values, there is a cost value associated with each hypothesis, and the hypothesis with the smallest cost is used to specify the predicted symbol values ​​of the coefficients involved in the hypothesis.

[0057] The cost of each hypothesis is calculated as follows. After the normal reconstruction process (dequantization, inverse transform), a reconstructed block associated with a given hypothesis is generated, and the boundary samples of the reconstructed block 602, i.e., p 0,y and p x,0 For each reconstructed pixel p at the left boundary of the reconstructed block 602 0,y , performs a simple linear prediction using the two pixels on the left to obtain its prediction pred 0,y =(2p -1,y –p -2,y ). Combine the prediction with the reconstructed pixel p 0,y The absolute difference between them is added to the assumed cost equation 604. A similar process is performed for the pixels in the upper row of the reconstructed block. x,0 =(2p x,-1 –p x,-2 ) and reconstructed pixel p x,0 The absolute differences between them are summed.

[0058] Reference now Figure 7 , depicts an operational flow diagram showing the steps of a method 700 for encoding video data. In some implementations, Figure 7 One or more processing blocks of the computer 102 ( Figure 1 ) and server computer 114 ( Figure 1 ) is executed. In some implementations, Figure 7 One or more of the processing blocks of may be performed by another device or group of devices separate from or including the computer 102 and the server computer 114 .

[0059] At 702, method 700 includes identifying reference samples and magnitudes of transform coefficients corresponding to a current block of video data from an input to a neural network.

[0060] At 704, method 700 includes predicting a symbol value associated with the transform coefficient based on the identified reference samples.

[0061] At 706, method 700 includes encoding the video data based on the predicted symbol values.

[0062] Understandably, Figure 7 Only an illustration of one implementation is provided and does not imply any limitations on how different embodiments may be implemented.Many modifications may be made to the described environment depending on design and implementation requirements.

[0063] In one embodiment, an indicator of the auxiliary transform type or kernel is an input to the convolutional neural network.

[0064] In one embodiment, the method further comprises: obtaining estimated reconstructed samples of the current block based on the predicted symbol value candidates, wherein the estimated reconstructed samples of the current block and adjacent reconstructed values ​​are inputs of the convolutional neural network.

[0065] In one embodiment, a score value associated with the predicted symbol value is an output of the convolutional neural network, and wherein a symbol value is selected from the predicted symbol values ​​by the current block based on the score value.

[0066] In one embodiment, the number of transform coefficients used as input to the convolutional neural network varies based on the auxiliary transform applied.

[0067] In one embodiment, based on said secondary transform applied to said current block, a number of transform coefficients in a forward scanning order less than a predefined number of coefficients are used.

[0068] Figure 8 According to the illustrative embodiment Figure 1 102, 114. It should be understood that Figure 8 This only provides an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the described environment may be made depending on design and implementation requirements.

[0069] Computer 102 ( Figure 1 ) and server computer 114 ( Figure 1 ) can include Figure 8 Respective sets of internal components 800A, 800B and external components 900A, 900B are shown. Each set of internal components 800 includes one or more processors 820, one or more computer-readable random access memories (RAM) 822 and one or more computer-readable read-only memories (ROM) 824 on one or more buses 826, one or more operating systems 828, and one or more computer-readable tangible storage devices 830.

[0070] The processor 820 is implemented in hardware, firmware, or a combination of hardware and software. The processor 820 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor 820 includes one or more processors that can be programmed to perform functions. The bus 826 includes components that allow communication between internal components 800A and 800B.

[0071] Server computer 114 ( Figure 1 ) on one or more operating systems 828, software programs 108 ( Figure 1 ) and video encoding program 116 ( Figure 1 ) are stored on one or more of the respective computer-readable tangible storage devices 830 for execution by one or more of the respective processors 820 via one or more of the respective RAMs 822 (which typically include cache memories). Figure 8 In the illustrated embodiment, each computer-readable tangible storage device 830 is a magnetic disk storage device of an internal hard drive. Alternatively, each computer-readable tangible storage device 830 is a semiconductor storage device such as ROM 824, an erasable programmable read-only memory (EPROM), a flash memory, an optical disk, a magneto-optical disk, a solid-state disk, a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or another type of non-transitory computer-readable tangible storage device that can store computer programs and digital information.

[0072] Each set of internal components 800A, 800B also includes an R / W drive or interface 832 to read from or write to one or more portable computer readable tangible storage devices 936, such as CD-ROMs, DVDs, memory sticks, tapes, disks, optical disks, or semiconductor storage devices. Software programs, such as software program 108 ( Figure 1 ) and video encoding program 116 ( Figure 1 ) can be stored on one or more of the corresponding portable computer-readable tangible storage devices 936, read via the corresponding R / W drive or interface 832 and loaded into the corresponding hard drive 830.

[0073] Each set of internal components 800A, 800B also includes a network adapter or interface 836, such as a TCP / IP adapter card; a wireless Wi-Fi interface card; or a 3G, 4G or 5G wireless interface card or other wired or wireless communication link. Server computer 114 ( Figure 1 ) on the software program 108 ( Figure 1 ) and video encoding program 116 ( Figure 1 ) can be downloaded from an external computer to the computer 102 via a network (eg, the Internet, a local area network, or other wide area network) and a corresponding network adapter or interface 836 ( Figure 1 ) and the server computer 114. From the network adapter or interface 836, the software program 108 and the video encoding program 116 on the server computer 114 are loaded into the corresponding hard disk drive 830. The network may include copper wire, optical fiber, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers.

[0074] Each of the external component groups 900A, 900B may include a computer display 920, a keyboard 930, and a computer mouse 934. The external components 900A, 900B may also include a touch screen, a virtual keyboard, a touch pad, a pointing device, and other human-computer interaction devices. Each of the internal component groups 800A, 800B also includes a device driver 840 that interfaces with the computer display 920, the keyboard 930, and the computer mouse 934. The device driver 840, the R / W driver or interface 832, and the network adapter or interface 836 include hardware and software (stored in the storage device 830 and / or the ROM 824).

[0075] It should be understood in advance that although the present disclosure includes detailed descriptions about cloud computing, the implementation of the teachings recorded herein is not limited to cloud computing environments. Instead, some embodiments can be implemented in conjunction with any other type of computing environment now known or later developed.

[0076] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly provisioned and released, minimizing the workload of managing resources and interaction with service providers. Cloud models may include at least five characteristics, at least three service models, and at least four deployment models.

[0077] Features are as follows:

[0078] On-demand self-service: Cloud consumers can automatically and unilaterally provision computing capacity, such as server time and network storage, as needed, without requiring manual interaction with the service provider.

[0079] Broad network access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use on a variety of thin-client or thick-client platforms, such as mobile phones, notebooks, and personal digital assistants (PDAs).

[0080] Resource pooling: The provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, dynamically allocating and reallocating different physical and virtual resources based on demand. There is a sense of location independence. Resource location has confidentiality because consumers generally do not control or understand the exact location of the provided resources, but can specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0081] Rapid Elasticity: Capabilities can be provisioned quickly and elastically, in some cases automatically, to scale up quickly, and released quickly to scale up and down quickly. To consumers, the capabilities available for provisioning often appear to be unlimited and can be purchased at any time and in any quantity.

[0082] Metrics for measurable services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts) at some level of abstraction appropriate to the type of service. Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being used.

[0083] The service model is as follows:

[0084] Software as a Service (SaaS): The functionality provided to the consumer is to use the vendor's applications running on the cloud infrastructure. These applications are accessed from a variety of client devices through a thin client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functions, with the possible exception of limited user-specific application configuration settings.

[0085] Platform as a Service (PaaS): The functionality provided to consumers is to deploy applications created or acquired by consumers onto cloud infrastructure, where these applications are created using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but consumers control deployed applications and may control the application hosting environment configuration.

[0086] Infrastructure as a Service (IaaS): The capabilities provided to consumers are processing, storage, networking, and other basic computing resources on which consumers can deploy and run arbitrary software, which can include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but do control the operating system, storage, and deployed applications, and may have limited control over selected network components (e.g., host firewalls).

[0087] The deployment model is as follows:

[0088] Private cloud: Cloud infrastructure is run only for a certain organization. The cloud infrastructure may be managed by the organization itself or by a third party, and may be deployed on-premises or off-premises.

[0089] Community Cloud: Cloud infrastructure is shared by multiple organizations and supports a specific community with common concerns (e.g., mission, security requirements, policy, and compliance considerations). The cloud infrastructure may be managed by the organization or by a third party and may exist on-premises or off-premises.

[0090] Public cloud: Cloud infrastructure is provided to the general public or large industry groups and is owned by the organization that sells cloud services.

[0091] Hybrid cloud: A cloud infrastructure consisting of two or more clouds (private, community, or public) that remain independent entities but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for inter-cloud load balancing).

[0092] The cloud computing environment is service-oriented and focuses on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure consisting of a network of interconnected nodes.

[0093] refer to Fig. 9 , an illustrative cloud computing environment 1000 is described. As shown, the cloud computing environment 1000 includes one or more cloud computing nodes 10, with which local computing devices used by cloud consumers, such as PDAs or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automobile computer systems 54N, can communicate. The cloud computing nodes 10 can communicate with each other. The cloud computing nodes can be physically grouped (not shown) or virtually grouped (not shown) in one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above. This allows the cloud computing environment 1000 to provide infrastructure, platform, and / or software as a service without the cloud consumer having to maintain resources on a local computing device. It should be understood that Fig. 9 The types of computing devices 54A-54N shown in FIG. 1 are for illustration only, and cloud computing node 10 and cloud computing environment 1000 may communicate with any type of computerized device over any type of network and / or network-addressable connection (eg, using a web browser).

[0094] refer to Fig.10 , showing a cloud computing environment 1000 ( Fig. 9 ) provides a set of functional abstraction layers 1100. It should be understood in advance that Fig.10 The components, layers and functions shown are for illustration only and the embodiments are not limited thereto. As shown, a hardware and software layer 60, a virtualization layer 70, a management layer 80 and a workload layer 90 and corresponding functions are provided.

[0095] The hardware and software layer 60 includes hardware components and software components. Examples of hardware components include: mainframes 61; servers based on reduced instruction set computer (RISC) architecture 62; servers 63; blade servers 64; storage devices 65; and networks and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0096] Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71; virtual storage 72; virtual networks 73, which include virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0097] In one example, the management layer 80 may provide the functionality described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources for performing tasks within a cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized in the cloud computing environment, and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, and provides protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management so that the required service levels are met. Service level agreement (SLA) planning and implementation 85 provides pre-arrangement and procurement of cloud computing resources, for which future demand is anticipated based on the SLA.

[0098] Workload layer 90 provides examples of functionality that can utilize a cloud computing environment. Examples of workloads and functionality that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics processing 94; transaction processing 95; and video encoding 96. Video encoding 96 can encode video data based on predicting the symbol values ​​of transform coefficients of the video data.

[0099] Some embodiments may involve systems, methods and / or computer-readable media at any possible level of technical detail integration. The computer-readable media may include a computer-readable non-transitory storage medium (or multiple storage media) having computer-readable program instructions thereon for causing a processor to perform operations.

[0100] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a RAM, a ROM, an EPROM or flash memory, a static random access memory (SRAM), a portable CD-ROM (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. The computer-readable storage medium used herein should not be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal propagating through a wire.

[0101] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing device or processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing device or processing device.

[0102] The computer readable program code / instruction for performing an operation can be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, a state setting data, the configuration data of an integrated circuit, or a source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++ and process programming languages ​​such as "C" programming language or similar programming languages. The computer readable program instruction can be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a LAN or WAN, or the remote computer can be connected to an external computer (for example, by using the Internet of an Internet service provider). In certain embodiments, the electronic circuit including, for example, a programmable logic circuit, an FPGA or a programmable logic array (PLA) can execute the computer readable program instruction by utilizing the state information of the computer readable program instruction, to personalize the electronic circuit, so as to perform aspects or operations.

[0103] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in the flowchart and / or block diagram.

[0104] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in the flowchart and / or block diagram.

[0105] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the system, method and computer-readable medium according to multiple embodiments. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of an instruction, and the module, a program segment or a part of an instruction contains one or more executable instructions for realizing the specified logical function. The method, computer system and computer-readable medium may include additional blocks, fewer blocks, different blocks or blocks arranged differently from the blocks shown in the figure. In some alternative implementations, the functions marked in the box may also occur in a different order than those marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they may also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of boxes in the block diagram and / or flow chart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0106] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit these implementations. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code—it will be appreciated that software and hardware may be designed to implement the systems and / or methods based on the descriptions herein.

[0107] Unless such description is clearly provided, the elements, actions or instructions used herein should not be interpreted as critical or necessary. In addition, as used herein, the articles "one" and "a" are intended to include one or more items and can be used interchangeably with "one or more". In addition, as used herein, the term "group" is intended to include one or more items (e.g., related items, non-related items, a combination of related and non-related items, etc.), and can be used interchangeably with "one or more". In the case of only one item, the term "one" or similar language is used. In addition, as used herein, the terms "have", "have", "have" etc. are intended to be open terms. In addition, unless otherwise expressly stated, the expression "based on" is intended to mean "based at least in part on".

[0108] Descriptions of various aspects and embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the disclosed embodiments. Even if a combination of features is listed in the claims and / or disclosed in the specification, many of these features may be combined in ways not specifically disclosed in the claims and / or specification. Although each dependent claim listed below may directly reference only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications or technical improvements to technologies found on the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A video encoding method, executed by a processor, characterized in that: The method comprises: Identifying reference samples and magnitudes of transform coefficients corresponding to a current block of video data, the reference samples and magnitudes of transform coefficients of the current block of video data serving as inputs to a neural network; predicting a symbol value associated with a transform coefficient based at least on the identified reference sample and a magnitude of the transform coefficient, wherein the symbol value is predicted using the neural network; and The video data is encoded based on the predicted symbol values.

2. The method according to claim 1, characterized in that The reference samples include reconstructed samples from spatially neighboring blocks corresponding to the current block.

3. The method according to claim 1, characterized in that: The reference samples include reconstructed samples specified by motion vectors associated with a reference picture from the video data.

4. The method according to claim 1, characterized in that The transform coefficients are dequantized transform coefficients.

5. The method according to claim 1, characterized in that The predicted symbol values ​​correspond to a finite set of transform coefficients.

6. The method according to claim 5, characterized in that The limited transform coefficient set includes low frequency coefficients located at a predefined upper left corner area of ​​the current block.

7. The method according to claim 5, characterized in that The limited set of transform coefficients includes a predefined number of coefficients in a forward scanning order.

8. The method according to claim 1, characterized in that A convolutional neural network is used to predict the symbol value.

9. The method according to claim 8, characterized in that Based on reconstructed samples associated with a current image and a plurality of reference images, only phase-correlated differences between the current image and the plurality of reference images are used as input to the neural network, wherein the only phase-correlated differences include POC differences.

10. The method according to claim 8, characterized in that An indicator of the primary transform type or kernel is an input to the convolutional neural network.

11. The method according to claim 1, characterized in that: Based on the assumed predicted symbol values, reconstructed samples of the current block are generated, and a subset of the reconstructed samples of the current block is used as input to the neural network.

12. The method according to claim 11, characterized in that The subset of reconstructed samples includes boundary samples of the current block.

13. The method according to claim 12, characterized in that The boundary samples include: a first predetermined number of upper rows of the current block and a second predetermined number of left columns of the current block.

14. The method according to claim 13, characterized in that The first predetermined number and the second predetermined number are based on a size associated with the current block.

15. A computer system for encoding video data, the computer system comprising: one or more computer-readable non-transitory storage media configured to store computer program code; as well as One or more computer processors are configured to access the computer program code and execute the operations indicated by the computer program code to perform the method according to any one of claims 1 to 14.

16. A non-transitory computer-readable medium storing a computer program for encoding video data, the computer program being configured to cause one or more computer processors to execute the method according to any one of claims 1 to 14.

Images