Video encoding method, computer equipment, device and computer readable storage medium

Abstract

A method, apparatus, and storage medium for performing video encoding are provided. The method includes: obtaining a plurality of image frames in a video sequence; determining a feature map of each of the plurality of image frames, and determining an offset map based on the feature map; determining an alignment feature map by performing a temporal deformable convolution (TDC) on the feature map and the offset map; and generating a plurality of aligned frames based on the alignment feature map.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims priority to U.S. Provisional Application No. 63 / 090,126 filed with the United States Patent and Trademark Office on October 9, 2020, and U.S. Application No. 17 / 394,504 filed with the United States Patent and Trademark Office on August 5, 2021, the disclosures of which are incorporated herein by reference in their entirety. Background Art 1. Technical Field

[0004] The present disclosure relates to the field of video coding, and more particularly to a device and method for using a loop filter to process decoded video based on a deep neural network (DNN) with temporal deformable convolution (TDC), and more particularly to a method, computer device, apparatus, and non-transitory computer-readable storage medium for video coding.

[0005] 2. Description of Related Technology

[0006] Traditional video coding standards such as H.264 / Advanced Video Coding (H.264 / AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC) share a similar (recursive) block-based hybrid prediction / transform framework in which individual coding tools such as intra / inter prediction, integer transform, and context-adaptive entropy coding are carefully handcrafted to optimize overall efficiency. Essentially, spatiotemporal pixel neighborhoods are used to predict signal structure to obtain corresponding residuals for subsequent transform, quantization, and entropy coding. On the other hand, the nature of deep neural networks (DNNs) is to extract spatiotemporal stimuli at different levels by analyzing the spatiotemporal information of the receptive fields of neighboring pixels. The ability to exploit highly nonlinear and non-local spatiotemporal correlations offers a promising opportunity to significantly improve compression quality.

[0007] However, compressed videos inevitably suffer from compression artifacts, which severely degrade the quality of experience (QoE). DNN-based methods have been developed to improve the visual quality of compressed images, such as image denoising, super-resolution, and deblurring. When these methods are applied to videos, image-based methods suffer from instabilities and fluctuations caused by changes in the compressed video quality, video scene, and object motion in the video. Therefore, it is important to utilize information from neighboring frames in the video to stabilize and improve the improved visual quality.

[0008] One caveat to leveraging information from multiple adjacent video frames is the complex motion caused by moving cameras and dynamic scenes. Traditional block-based motion vectors are not suitable for non-translational motion. Furthermore, while learning-based optical flow methods can provide more accurate motion information at the pixel level, these methods are still prone to errors, especially along the boundaries of moving objects.

[0009] Therefore, one or more embodiments of the present disclosure provide a DNN-based model with temporal deformable convolution (TDC) to handle arbitrary and complex motions in a data-driven manner without explicit motion estimation. Summary of the Invention

[0010] According to an embodiment, a method for performing video encoding using one or more neural networks with a loop filter is provided. The method includes: obtaining a plurality of image frames in a video sequence; determining a feature map of each of the plurality of image frames, and determining an offset map based on the feature map; determining an alignment feature map by performing a temporal deformable convolution (TDC) on the feature map and the offset map; and generating a plurality of aligned frames.

[0011] According to an embodiment, a computer device is provided, comprising: one or more computer-readable non-transitory storage media configured to store computer program code; and one or more computer processors configured to access the computer program code and execute the above method for video encoding as instructed by the computer program code.

[0012] According to an embodiment, a device is provided, which includes: an acquisition unit configured to acquire multiple image frames in a video sequence; a determination unit configured to: determine a feature map of each of the multiple image frames, and determine an offset map based on the feature map, and determine an alignment feature map by performing temporal deformable convolution (TDC) on the feature map and the offset map; and a generation unit configured to generate multiple aligned frames.

[0013] According to an embodiment, a non-transitory computer-readable storage medium is provided, which stores computer program code. When the computer program code is executed by at least one processor, the at least one processor performs the above method for video encoding.

[0014] The video encoding method, computer device, apparatus, and non-transitory computer-readable storage medium of the present invention provide a DNN-based model with temporal deformable convolution (TDC) to handle arbitrary and complex motions in a data-driven manner without explicit motion estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The following description briefly introduces the accompanying drawings, which illustrate one or more embodiments of the present disclosure.

[0016] Figure 1 is a diagram showing a video encoding system according to an embodiment;

[0017] Figure 2 is a diagram illustrating an example of a computing device including an encoder and a decoder according to an embodiment;

[0018] Figure 3 is a diagram illustrating a method of training a DNN LF module according to an embodiment;

[0019] Figure 4 is a diagram illustrating the operation of a DNN LF module using 2D TDC according to an embodiment;

[0020] Figure 5 is a diagram illustrating the operation of a DNN LF module using 3D TDC according to an embodiment; and

[0021] Figure 6 is a block diagram of an apparatus for processing video data using a DNN with a TDC according to an embodiment. DETAILED DESCRIPTION

[0022] Example embodiments are described in detail herein with reference to the accompanying drawings. It should be understood that one or more embodiments of the present disclosure described herein are merely example embodiments and should not be construed as limiting the scope of the present disclosure.

[0023] Figure 1 is a diagram illustrating a video encoding system according to an embodiment.

[0024] Reference Figure 1, the video coding system 100 may include an encoder 110 and a decoder 120 connected to each other via a network 130. The encoder 110 may be configured to compress image data or video data. The encoder 110 may be set on the server side of the video coding system 100. The server may be a cloud server or a server cluster including multiple servers. The server may include a processor, a communication interface, and a database. The processor may be configured to encode one or more image frames of a video sequence. The communication interface may be configured to send and receive one or more image frames of a video sequence or other data that may be related to encoding the video sequence. The database may be configured to store one or more images of a video sequence, encoded data, and decoded data. However, one or more embodiments are not limited thereto. For example, the encoder 110 may be implemented in a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a mobile device, a personal digital assistant (PDA), a console, a wearable device, etc.) connected to the network 130. The encoder 110 may be software running on the computing device.

[0025] Figure 2 is a diagram showing an example of a computing device implemented as an encoder and / or decoder according to an embodiment. The computing device 200 may be implemented as Figure 1 The encoder 110 and / or decoder 120 shown in FIG.

[0026] Reference Figure 2 , the computing device 200 may include a processor 210 , a memory 220 , a storage device 230 , an input interface 240 , an output interface 250 , a communication interface 260 , and a bus 270 .

[0027] The processor 210 is implemented in hardware, firmware, or a combination of hardware and software. The processor 210 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor 210 includes one or more processors that can be programmed to perform functions.

[0028] Memory 220 includes random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by processor 210.

[0029] Storage device 230 stores information and / or software related to the operation and use of computing device 200. For example, storage device 230 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or another type of non-transitory computer-readable medium and a corresponding drive.

[0030] Input interface 240 includes components that allow computing device 200 to receive information, for example, via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, input interface 240 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output interface 250 includes components that provide output information from computing device 200 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0031] The communication interface 260 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the computing device 200 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. The communication interface 260 can allow the computing device 200 to receive information from another device and / or provide information to another device. For example, the communication interface 260 can include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0032] Bus 270 includes components that allow communication between components of computing device 200 .

[0033] The computing device 200 can perform one or more operations described herein. The computing device 200 can perform the operations described herein in response to the processor 210 executing software instructions stored in a non-transitory computer-readable medium, such as the memory 220 and / or the storage device 230. A computer-readable medium is defined herein as a non-transitory memory device. A memory device can include memory space within a single physical storage device or memory space distributed across multiple physical storage devices.

[0034] The software instructions may be read into the memory 220 and / or storage device 230 from another computer-readable medium or from another device via the communication interface 260. When executed, the software instructions stored in the memory 220 and / or storage device 230 may cause the processor 210 to perform one or more of the processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of software instructions or in combination with hardwired circuitry to perform one or more of the processes described herein. Thus, the implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0035] supply Figure 2 The number and arrangement of components shown in FIG2 are provided as examples. In practice, computing device 200 may include additional components, fewer components, different components, or components that are different from those in FIG200 . Figure 2 Additionally or alternatively, one set of components (eg, one or more components) of computing device 200 may perform one or more functions described as being performed by another set of components of computing device 200.

[0036] Figure 3 is a diagram illustrating a method of training a DNN LF module according to an embodiment.

[0037] First, a typical video compression framework will be described in this paper. Assume that the input video x consists of multiple original image frames x1, ..., x t In the first motion estimation step, multiple original image frames are divided into spatial blocks, each of which can be iteratively further divided into smaller blocks, and the current original image frame x is calculated for each spatial block. t with the set of previously reconstructed frames The motion vector m between t Here, the subscript t represents the current t-th encoding cycle, which may not match the timestamp of the image frame. In addition, the set of previously reconstructed frames Frames from multiple previous coding cycles may be included. In the second motion compensation step, the motion vector m is t Copy the previously reconstructed frame The corresponding pixels of the predicted frame are obtained And through Get the current original image frame x t With predicted frame The residual r t In the third estimation step, after performing discrete cosine transform (DCT) on the spatial block, the residual r t The DCT coefficients are quantized to obtain the quantized residual Thus, the motion vector mt and the quantized residual Both are encoded into a bitstream by entropy coding, and the bitstream is sent to one or more decoders. On the decoder side, the quantized residual is first dequantized (e.g., by an inverse transform, such as an inverse discrete cosine transform (IDCT)) to obtain a recovered residual Then pass Will recover the residual Add back to the predicted frame to obtain the reconstructed frame.

[0038] Additionally, additional components can be used to improve the reconstructed frame One or more of the enhancement modules such as deblocking filter (DF), sample adaptive offset (SAO), adaptive loop filter (ALF), cross-component adaptive filter (CCALF), etc. can be selected to process the reconstructed frame For example, a deblocking filter (DF) is a video filter that can be applied to decoded video to improve visual quality and prediction performance by smoothing sharp edges formed between macroblocks when block coding techniques are used. Sample adaptive offset (SAO) is an in-loop filtering technique that reduces average sample distortion by adding an offset value to each sample. SAO includes two types of offset techniques, which are edge offset (EO) and band offset (BO). EO is driven by the local directional structure in the image frame to be filtered, and BO modifies the intensity value of the image frame but does not depend on the field. An adaptive loop filter (ALF) can be used to minimize the mean square error between the original sample image and the decoded sample image. Various modifications can be made to the processing order of the enhancement modules and the selection of the enhancement modules according to user settings.

[0039] According to an embodiment, a holistic method for training a DNN LF is provided. Figure 3 , the set of N previously reconstructed frames is provided as input to the DNN LF module 310. The DNN LF module 310 can generate high-quality frames by using an end-to-end (E2E) LF DNN network. Then, the reconstruction quality calculation module 320 calculates the generated high-quality frame To evaluate the original image frame x t Then, the reconstruction quality calculation module 320 calculates the reconstruction quality Output to the back propagation module 330. The back propagation module 330 reconstructs the quality Feedback is given to the DNN LF module 310 to train the DNN LF module 310 according to the reconstruction quality evaluated by the reconstruction quality calculation module 320.

[0040] Additionally, the alignment loss of the set of N previously reconstructed frames output by the DNN LF module 310 is can be fed into the back-propagation module 330, the alignment loss Then it is fed back to the DNN LF module 310 through the back propagation module 330 to train the DNN LF module 310. Figure 4 Let’s describe the alignment loss in more detail.

[0041] In addition, high-quality frames and the original image frame x t Can be input to the discrimination module 340 to identify and detect high-quality frames With the original image frame x t That is, the discrimination module 340 can be based on and x t To calculate the discriminant loss And the discrimination loss is sent to the back propagation module 330. It can be fed back to the DNN LF module 310 and the discriminant module 340 through the back-propagation module 330 to train the DNN LF module 310 and the discriminant module 340.

[0042] The discriminative DNN can be used and x t At least one of them is used as input to calculate the discriminant feature map or d(x t ) classification network. Based on the discriminant feature map or d(x t ), the discriminant DNN input is the original image frame x t Or generated (or synthesized) high-quality frames Classification can be performed. The classification loss can be calculated To measure misclassification loss, such as classification cross entropy loss. In addition, feature discrimination loss can be calculated Based on the generated high-quality image frames The calculated discriminant feature map is based on the original image frame x t Compute the difference between the discriminative feature maps.

[0043] Total discriminative loss Can be and The total discriminant loss is calculated according to the following equation (1):

[0044] Here, γ is the discriminant feature map and d(xt ) associated weights.

[0045] As mentioned above, the reconstruction quality output by the reconstruction quality calculation module 320 is Alignment loss output by DNN LF310 The discriminant loss output by the discriminant module 340 is sent to the back propagation module 330. Based on these calculations received from various modules, the back propagation module 330 can calculate the joint loss L joint The gradient of the joint loss L joint The gradient of can be fed back to the DNN LF module 310 and the discriminant module 340. The joint loss L can be calculated according to the following equation (2): joint Gradient:

[0046]

[0047] Here, λ is the weight associated with the alignment loss and β is the weight associated with the discrimination loss.

[0048] The joint loss L can be back-propagated through the back-propagation module 330 joint The gradient of is used to update the DNN weight coefficients in the LF DNN (e.g., feature extraction DNN, offset generation DNN, TDC DNN, frame reconstruction DNN, frame synthesis DNN, discriminative DNN, and TDC and feature fusion DNN).

[0049] Based on the joint loss L joint Feedback to one or more DNNs above to predict frames is added to update the set of N previously reconstructed frames For example, the oldest frame at the farthest distance from the current frame may be removed from the set of N previously reconstructed frames, and the predicted frame may be added To replace the oldest frame removed. After that, the encoder can enter the next encoding cycle from t to t+1.

[0050] According to an embodiment, the DNN LF module 310 can be used in combination with one or more of the above-mentioned additional components (e.g., DF, SAO, ALF, CCALF, etc.) to improve the reconstructed frame. For example, the reconstructed frame can be processed sequentially through DF, DNN LF module, SAO and ALF. However, one or more embodiments are not limited thereto, and the order in which the additional components are processed may be configured differently. In an embodiment, the DNN LF module 310 may be used alone as a replacement for all other additional components to improve the reconstructed frame. visual quality.

[0051] Figure 4 is a diagram illustrating the operation of a DNN LF module using 2D TDC according to an embodiment.

[0052] Reference Figure 4 , the DNN LF module 400 may include a feature extraction module 410 , an offset generation module 420 , a TDC module 430 , a frame reconstruction module 440 , a frame synthesis module 450 and an alignment error calculation module 460 .

[0053] The feature extraction module 410 may receive a set of N previously reconstructed frames As input, the feature extraction module 410 can be configured to calculate the feature map by using the feature extraction DNN through forward reasoning For example, suppose the frame is used as a reference frame to which all other frames must be aligned, the offset generation module 420 can generate the offset DNN by connecting the feature maps and And pass the connected feature map base and To calculate the offset map ΔP j→i，t Here, the frame Can be a set of N previously reconstructed frames Without loss of generality, the set of N previously reconstructed frames Sort them in ascending order according to their timestamps. Therefore, N reconstructed frames can be For example, when the goal is to improve the current reconstructed frame When That is, all other previously reconstructed neighboring frames can be In another embodiment, a portion of a previously reconstructed adjacent frame may be Before, and the remaining frames can be after.

[0054] The offset map ΔP generated by the offset generation module 420 j→i，t can be input to the TDC module 430. Figure 4 In the example, a TDC DNN can be formed by stacking several TDC layers, each followed by a nonlinear activation layer such as a rectified linear unit (ReLU) and other DNN layers (e.g., a bottleneck layer). The TDC module 430 uses the TDC DNN to extract the feature map output by the feature extraction module 410 based on the feature map. and the offset map ΔP output by the offset generation module 420 j→i，t Compute alignment feature maps

[0055] According to an embodiment, the TDC DNN may include a two-dimensional (2D) TDC layer. For example, assume that Wk represents a weight coefficient of a 2D TDC kernel, where k is a natural number greater than or equal to 1 (e.g., k=1, ..., K), and p k represents the predetermined offset of the k-th position in the kernel (e.g., a 3x3 kernel is defined with K=9 and pk∈{(-1,-1),(-1,0),...,(1,1)}). The 2D TDC layer can be based on the input feature f in and learnable offset ΔP to calculate the output feature f out , where the feature at the sampling position p0 is determined based on the following equation:

[0056]

[0057] Here, the sum of the offsets (p0+p k +Δp k ) may be irregular and may not be an integer, the TDC operation may perform interpolation (eg, bilinear interpolation) to correct (p0+p k +ΔP k )'s irregular position.

[0058] Furthermore, the alignment error calculation module 460 may be configured to calculate the alignment loss To measure and For example, the L1 norm (or the average of the absolute error) or the L2 norm (the average of the square root error) can be used to calculate the misalignment. The frame reconstruction module 440 can use the frame reconstruction DNN calculated by feedforward inference to calculate the feature map output by the feature extraction module 410. and the alignment feature map output by the TDC module 430 Generate alignment frame Align Frame can be used as input to the frame synthesis module 450, and the frame synthesis module 450 can use the frame synthesis DNN to generate a synthesized high-quality frame

[0059] Although some specific embodiments of the DNN LF module have been described above, it should be understood that one or more embodiments of the present disclosure are not limited thereto. For example, the type of layer, the number of layers, the kernel size, etc. can be configured differently for each of the feature extraction DNN, the offset generation DNN, the TDCDNN, the frame reconstruction DNN, and the frame synthesis DNN. For example, any backbone network such as ResNET can be used as a feature synthesis DNN. For example, a set of regular convolutions and a bottleneck layer can be stacked as an offset generation DNN. For example, a set of TDC layers can be stacked as a TDCDNN, and some convolution layers using skip connections can be stacked together as a frame reconstruction DNN. For example, some residual block layers can be stacked together as a frame synthesis DNN.

[0060] Figure 5 is a diagram illustrating the operation of a DNN LF module using 3D TDC according to an embodiment.

[0061] Reference Figure 5 , the DNN LF module 500 may include a feature extraction module 510, a TDC and feature fusion module 520, and a feature reconstruction module 530. It should be understood that the above reference Figure 4 Some of the features or functions of the components of the DNN LF module 500 have been described. Therefore, repeated description thereof may be omitted.

[0062] According to an embodiment, the input frame can be stacked together to obtain a 4D input tensor of size (n, c, h, w), where c is the number of channels (e.g., three channels for color frames) and (h, w) provides the resolution of the video frame. The feature extraction module 510 can be configured to compute feature maps by forward inference using the feature extraction DNN. In an embodiment, the feature extraction DNN uses a 3D convolutional layer (e.g., C3D) to compute the feature map And capture the spatiotemporal characteristics of the video. In another embodiment, the Figure 4 The 2D convolutional layer described here computes each individual feature map. Afterwards, the feature map can be Therefore, the feature extraction module 510 can output a 4D feature tensor.

[0063] For example, suppose w k represents the weight coefficient of the 3D TDC kernel and p krepresents a predetermined offset of the k-th position in the kernel, where k is a natural number greater than or equal to 1 (eg, k=1...., K). The 3D TDC kernel may be defined as K=27 and p k ∈{(-1, -1, -1), (-1, -1, 0), ..., (1, 1, 1)}. The 3D TDC layer can be based on the input feature f in and learnable offset ΔP to calculate the output feature f out , where the characteristics at sampling position p0 are given using the same equation (3) provided above.

[0064] exist Figure 5 In

[15] , TDC and feature fusion DNNs can be formed by stacking one or more 3D TDC layers, where each layer is followed by a nonlinear activation layer, such as ReLU. Additionally or alternatively, each 3D TDC layer can be followed by other DNN layers, such as a bottleneck layer. For example, assuming the frame is the reference frame to which all other frames need to be aligned, then the TDC and feature fusion module 520 uses TDC and feature fusion DNN to Compute fused aligned feature maps The frame reconstruction module 530 uses the frame reconstruction DNN to align the feature maps based on the fusion Compute high-quality reconstructed frames Here, no explicit spatial alignment is performed when training the DNN LF module 500. Instead, the 3D deformable convolution directly learns spatiotemporal offsets to generate aligned fusion features.

[0065] Although some specific embodiments of the DNN LF module have been described above, it should be understood that one or more embodiments of the present disclosure are not limited thereto. For example, the type of layer, the number of layers, the kernel size, etc. may be configured differently for each of the feature extraction DNN, the TDC and feature fusion DNN, and the frame reconstruction DNN.

[0066] Figure 6 FIG is a block diagram of an apparatus for processing video data using a DNN LF with a TDC according to an embodiment. Figure 4 and Figure 5 Some of the features or functions of the components of the apparatus 600 have been described, and therefore, repeated description thereof may be omitted.

[0067] The apparatus 600 may include: at least one memory storing computer program code; and at least one processor configured to access the at least one memory and operate according to instructions of the computer program code. The computer program code 600 may include acquisition code 610, determination code 620, and generation code 630.

[0068] The acquisition code 610 may be configured to acquire a set of reconstructed image frames in a video sequence. Figure 4 and Figure 5 The operations of the feature extraction modules 410 and 510 are described.

[0069] The determination code 620 may be configured to: determine a feature map of each of the plurality of image frames, determine an offset map based on the feature map, and determine an alignment feature map by performing a temporal deformable convolution (TDC) on the feature map and the offset map. Figure 4 The offset generation module 420, TDC 430 and alignment error calculation module 460 described above and the Figure 5 The operation of the TDC and feature fusion module 520 is described.

[0070] The generation code 630 may be configured to generate a plurality of aligned frames and synthesize the plurality of aligned frames to output a plurality of high-quality frames corresponding to the plurality of image frames. Figure 4 The frame reconstruction module 430 and the frame synthesis module 450 and Figure 5 The operation of the frame reconstruction module 530.

[0071] Although the apparatus 600 is described as including only the acquisition code 610, the determination code 620, and the generation code 630, one or more embodiments of the present disclosure are not limited thereto. One or more embodiments may include more than one embodiment. Figure 6 More or fewer components or parts than those shown.

[0072] In one embodiment, the method further comprises determining an alignment loss indicative of a misalignment error between the feature map and the aligned feature map, wherein the one or more neural networks are trained by the alignment loss.

[0073] In one embodiment, acquiring the plurality of image frames includes stacking the plurality of image frames to acquire a 4-dimensional (4D) input tensor.

[0074] In one embodiment, determining the offset map includes generating a deep neural network (DNN) to connect multiple feature maps based on the offsets.

[0075] In one embodiment, the plurality of image frames are further processed using at least one of a deblocking filter (DF), a sample adaptive offset (SAO), an adaptive loop filter (ALF), or a cross-component adaptive filter (CCALF).

[0076] In one embodiment, the plurality of high-quality image frames are evaluated to determine a reconstruction quality of the plurality of image frames, wherein the reconstruction quality of the plurality of image frames is back-propagated through the one or more neural networks, and wherein the one or more neural networks are trained by the reconstruction quality of the plurality of image frames.

[0077] In one embodiment, the method further includes determining a discriminative loss that indicates an error in classifying whether each of the plurality of image frames is an original image frame or a high-quality frame, and wherein one or more neural networks implemented in the device are trained by the discriminative loss.

[0078] In one embodiment, determining the alignment feature map comprises using a temporally deformable convolutional deep neural network (TDC DNN), wherein the TDC DNN comprises a plurality of TDC layers in a stack, and wherein each of the plurality of TDC layers is followed by a nonlinear activation layer comprising a rectified linear unit (ReLU).

[0079] Terms such as “unit” or “module” used in one or more embodiments of the present disclosure indicate a unit for processing at least one function or operation and can be implemented in hardware, software, or a combination of hardware and software.

[0080] The term "unit," "code," or "module" can be implemented by a program stored in an addressable storage medium and executed by a processor.

[0081] For example, the terms "unit," "code," or "module" may include software components, object-oriented software components, class components and task components, processes, functions, properties, procedures, subroutines, program code segments, drivers, firmware, microcode, circuit systems, data, databases, data structures, tables, arrays, and / or variables.

[0082] Some of the embodiments of the present disclosure have been shown and described above. However, one or more embodiments of the present disclosure are not limited to the aforementioned specific embodiments. It should be understood that various modifications, replacements, improvements and their equivalents can be made without departing from the spirit and scope of the present disclosure. It should be understood that such modifications, replacements, improvements and their equivalents should fall within the scope of protection of the present disclosure and should not be interpreted as independent of the inventive concept or prospects of the present disclosure.

Claims

1. A method for performing video encoding using one or more neural networks with loop filters, characterized in that The method comprises: Acquire multiple image frames in a video sequence; determining a feature map for each image frame of the plurality of image frames; Selecting a reference frame from the plurality of image frames, wherein the reference frame is a frame to which other frames in the plurality of image frames are to be aligned; concatenating a feature map of the reference frame with feature maps of other frames in the plurality of image frames, and passing the concatenated feature map through an offset generation deep neural network (DNN) to generate an offset map; determining an alignment feature map by performing a temporal deformable convolution (TDC) on the feature map and the offset map; and A plurality of aligned frames are generated based on the aligned feature map.

2. The method according to claim 1, characterized in that The method further comprises: The plurality of aligned frames are synthesized to output a plurality of high-quality frames corresponding to the plurality of image frames.

3. The method according to any one of claims 1 to 2, characterized in that The method further comprises: determining an alignment loss indicative of a misalignment error between the feature map and the aligned feature map, Wherein, the one or more neural networks are trained by the alignment loss.

4. The method according to any one of claims 1 to 2, characterized in that Acquiring the plurality of image frames includes stacking the plurality of image frames to acquire a 4-dimensional (4D) input tensor.

5. The method according to any one of claims 1 to 2, characterized in that Determining the offset map includes generating a deep neural network (DNN) to connect multiple feature maps based on the offsets.

6. The method according to any one of claims 1 to 2, characterized in that The plurality of image frames are also processed using at least one of a deblocking filter (DF), a sample adaptive offset (SAO), an adaptive loop filter (ALF), or a cross-component adaptive filter (CCALF).

7. The method according to claim 2, characterized in that evaluating the plurality of high-quality frames to determine a reconstruction quality of the plurality of image frames, wherein the reconstruction quality of the plurality of image frames is back-propagated through the one or more neural networks, and The one or more neural networks are trained by the reconstruction quality of the multiple image frames.

8. The method according to any one of claims 1 to 2, characterized in that The method also includes determining a discriminative loss that indicates an error in classifying whether each of the plurality of image frames is an original image frame or a high-quality frame, and Wherein, one or more neural networks implemented in the device are trained by the discriminative loss.

9. The method according to any one of claims 1 to 2, characterized in that Determining the alignment feature map includes: using a temporal deformable convolutional deep neural network (TDC DNN), The time-deformable convolutional deep neural network includes multiple TDC layers in a stack, and Each of the plurality of TDC layers is followed by a nonlinear activation layer including a rectified linear unit (ReLU).

10. A computer device, characterized in that: The computer device comprises: one or more computer-readable non-transitory storage media configured to store computer program code; and One or more computer processors configured to access the computer program code and execute the method according to any one of claims 1 to 9 as instructed by the computer program code.

11. A device, characterized in that: The device comprises: an acquisition unit, configured to acquire a plurality of image frames in a video sequence; A determining unit, wherein the determining unit is configured to: determining a feature map for each of the plurality of image frames, selecting a reference frame among the plurality of image frames, wherein the reference frame is a frame to which other frames in the plurality of image frames are to be aligned; concatenating the feature map of the reference frame with feature maps of other frames in the plurality of image frames, and passing the concatenated feature map through an offset generation deep neural network (DNN) to generate an offset map; determining an alignment feature map by performing a temporal deformable convolution (TDC) on the feature map and the offset map; and A generating unit is configured to generate a plurality of aligned frames.

12. The device according to claim 11, characterized in that The generating unit is further configured to synthesize the plurality of aligned frames to output a plurality of high-quality frames corresponding to the plurality of image frames.

13. The device according to any one of claims 11 to 12, characterized in that The determining unit is further configured to: determine an alignment loss, the alignment loss indicating an error in misalignment between the feature map and the aligned feature map, and wherein one or more neural networks implemented in the apparatus are trained by the alignment loss.

14. The device according to any one of claims 11 to 12, characterized in that The acquisition unit is further configured to arrange the plurality of image frames in a stack to acquire a 4-dimensional (4D) input tensor.

15. The device according to any one of claims 11 to 12, characterized in that The determining unit is further configured to determine the offset map by connecting a plurality of feature maps according to an offset generating deep neural network (DNN).

16. The device according to any one of claims 11 to 12, characterized in that The device further comprises: A processing unit is configured to process the plurality of image frames using at least one of a deblocking filter (DF), a sample adaptive offset (SAO), an adaptive loop filter (ALF), or a cross-component adaptive filter (CCALF).

17. The device according to claim 12, characterized in that The plurality of high quality frames are evaluated to determine a reconstruction quality of the plurality of image frames, wherein the reconstruction quality of the plurality of image frames is back-propagated to one or more neural networks, and The one or more neural networks are trained by the reconstruction quality of the multiple image frames.

18. The device according to any one of claims 11 to 12, characterized in that The determining unit is further configured to: determine a discrimination loss indicating an error in classification of whether each of the plurality of image frames is an original image frame or a high-quality frame, and wherein one or more neural networks implemented in the apparatus are trained by the discriminative loss.

19. The device according to any one of claims 11 to 12, characterized in that The determining unit is further configured to: determine the alignment feature map using a temporally deformable convolutional deep neural network (TDC DNN), The time-deformable convolutional deep neural network includes multiple TDC layers in a stack, and Each of the plurality of TDC layers is followed by a nonlinear activation layer including a rectified linear unit (ReLU).

20. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer program code, which, when executed by at least one processor, causes the at least one processor to perform the method according to any one of claims 1 to 9.

21. A method for storing a bit stream, characterized in that: Execute the method according to any one of claims 1 to 9 to generate a bit stream; and store the bit stream.

22. A method for transmitting a bit stream, characterized in that: Execute the method according to any one of claims 1 to 9 to generate a bit stream; and transmit the bit stream.

23. A computer-readable storage medium having stored thereon a computer program / instruction and a bit stream, characterized in that: When the computer program / instructions are executed by a processor, the computer program / instructions implement the steps of the method according to any one of claims 1 to 9 to generate the bit stream.

Images