Lightweight spatial upsampling method for machine vision

Abstract

The present disclosure provides a spatial upsampling model for processing video data suitable for machine vision tasks. An exemplary decoding method includes receiving a bitstream; and decoding one or more images using the encoded information of the bitstream, where the decoding includes: generating one or more decompressed images by decompressing one or more compressed images included in the bitstream; and performing spatial upsampling on the one or more decompressed images by a spatial upsampling model to obtain one or more reconstructed images, respectively, in which the total number of coding bits of parameters of the spatial upsampling model is less than a threshold predetermined based on the desired quality of the reconstructed images.

Description

[0001] Cross Reference to Related Applications This disclosure claims priority to U.S. Provisional Application No. 63 / 511,660, filed July 2, 2023, and claims priority to U.S. Patent Application 18 / 748,659, entitled “Lightweight Spatial Upsampling METHODS FOR MACHINE VISION,” filed June 20, 2024. Both applications are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0002] The present disclosure relates generally to video processing, and more specifically, to lightweight spatial upsampling models for processing video data suitable for machine vision tasks. BACKGROUND

[0003] With the rise of machine learning techniques and machine vision applications, the amount of video and image (collectively referred to as “image data”) processed by machines is rapidly growing. Typical use cases include autonomous driving, intelligent transportation, smart city, intelligent content management, etc., which involve machine vision tasks such as object detection, instance segmentation, and object tracking.

[0004] Since machine vision tasks require a large amount of image data, it is essential to compress the image data for efficient transmission and storage. However, traditional image / video compression techniques have always focused on ensuring the quality of images / videos perceived by humans, but machines process and understand visual data differently from human vision. Therefore, image / video compression techniques suitable for machine vision can be different from traditional compression techniques. Thus, new compression techniques are needed to achieve performance optimization for machine use. SUMMARY

[0005] The present disclosure provides lightweight spatial upsampling for machine vision tasks. Specifically, embodiments of the present disclosure provide a method for decoding a bitstream to output one or more images of a video stream, a method for encoding a video sequence into a bitstream, and a non-transitory computer-readable storage medium storing a video bitstream.

[0006] In a first aspect of the disclosure, a method for decoding a bitstream to output one or more images of a video stream is provided, the method comprising: receiving a bitstream; and decoding one or more images using coded information of the bitstream, wherein decoding the one or more images using the coded information of the bitstream comprises: generating one or more decompressed images by decompressing one or more compressed images included in the bitstream; and performing spatial upsampling on the one or more decompressed images by a spatial upsampling model to obtain one or more reconstructed images, respectively, wherein a total number of coded bits of parameters of the spatial upsampling model is less than a threshold predetermined based on a desired quality of the reconstructed images.

[0007] In a second aspect of the disclosure, a method for encoding a video sequence into a bitstream is provided, the method comprising: receiving a video sequence; encoding one or more images of the video sequence; and generating a bitstream associated with the one or more images, wherein encoding the one or more images of the video sequence comprises: performing spatial downsampling on the one or more images by a spatial downsampling model to obtain one or more downsampled images, respectively; generating one or more compressed images by compressing the one or more downsampled images; and generating parameters of a spatial upsampling model used for decoding the one or more compressed images, wherein a total number of coded bits of the parameters of the spatial upsampling model is less than a threshold predetermined based on a desired quality of reconstructed images of the one or more images.

[0008] In a third aspect of the disclosure, an apparatus for decoding a bitstream to output one or more images of a video stream is provided, the apparatus comprising: a receiving module configured to receive a bitstream; and a decoding module configured to decode one or more images using coded information of the bitstream, wherein the decoding module is configured to: generate one or more decompressed images by decompressing one or more compressed images included in the bitstream; and perform spatial upsampling on the one or more decompressed images by a spatial upsampling model to obtain one or more reconstructed images, respectively, wherein a total number of coded bits of parameters of the spatial upsampling model is less than a threshold predetermined based on a desired quality of the reconstructed images.

[0009] In a fourth aspect of the disclosure, an apparatus for encoding a video sequence into a bitstream is provided, the apparatus comprising: a receiving module configured to receive a video sequence; an encoding module configured to encode one or more pictures of the video sequence; and a generating module configured to generate a bitstream associated with the one or more pictures, wherein the encoding module is configured to: spatially downsample the one or more pictures by a spatial downsampling model to obtain one or more downsampled pictures, respectively; generate one or more compressed pictures by compressing the one or more downsampled pictures; and generate parameters of a spatial upsampling model for decoding the one or more compressed pictures, wherein a total number of coded bits of the parameters of the spatial upsampling model is less than a threshold predetermined based on a desired quality of reconstructed pictures of the one or more pictures.

[0010] In a fifth aspect of the disclosure, an electronic device is provided, the electronic device comprising: one or more processors; and a computer-readable storage medium communicatively coupled to the one or more processors, wherein the computer-readable storage medium stores computer-readable instructions executable by the one or more processors, the computer-readable instructions, when executed by the one or more processors, perform the method according to the first aspect or the second aspect.

[0011] In a sixth aspect of the disclosure, a non-transitory computer-readable storage medium having a video bitstream stored thereon is provided, wherein the bitstream, when decoded by a decoder, causes the decoder to perform the method according to the first aspect.

[0012] In a seventh aspect of the disclosure, a non-transitory computer-readable storage medium having a video bitstream stored thereon is provided, wherein the bitstream, when encoded by an encoder, causes the encoder to perform the method according to the second aspect.

[0013] In an eighth aspect of the disclosure, a computer program product is provided, the computer program product comprising: computer program instructions that enable a computer to perform the method according to the first aspect or the second aspect.

[0014] In a ninth aspect of the disclosure, a computer program is provided, the computer program enabling a computer to perform the method according to the first aspect or the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0015] Embodiments and aspects of the disclosure are illustrated by way of example in the following detailed description and in conjunction with the figures. Various features shown in the figures were not necessarily drawn to scale.

[0016] Figure 1is a schematic diagram illustrating an exemplary system for pre-processing and encoding image data, in accordance with some embodiments of the present disclosure.

[0017] Figure 2A is a schematic diagram illustrating an exemplary encoding process of a hybrid video encoding system, consistent with embodiments of the present disclosure.

[0018] Figure 2B is a schematic diagram illustrating another exemplary encoding process of a hybrid video encoding system, consistent with embodiments of the present disclosure.

[0019] Figure 3A is a schematic diagram illustrating an exemplary decoding process of a hybrid video encoding system, consistent with embodiments of the present disclosure.

[0020] Figure 3B is a schematic diagram illustrating another exemplary decoding process of a hybrid video encoding system, consistent with embodiments of the present disclosure.

[0021] Figure 4 is a block diagram of an exemplary apparatus for pre-processing or encoding image data, in accordance with some embodiments of the present disclosure.

[0022] Figure 5 is a schematic diagram illustrating an exemplary spatial resampling-based compression framework.

[0023] Figure 6A is a flowchart illustrating an exemplary method for decoding a bitstream to output one or more images of a video stream, in accordance with some embodiments of the present disclosure.

[0024] Figure 6B is a flowchart illustrating sub-steps of the method shown in Figure 6A is a flowchart illustrating sub-steps of the method shown in

[0025] Figure 7A is a schematic diagram illustrating an exemplary spatial upsampling model.

[0026] Figure 7B is a schematic diagram illustrating another exemplary spatial upsampling model, in accordance with some embodiments of the present disclosure, which achieves light-weighting by reducing the number of network layers.

[0027] Figure 8A is a schematic diagram illustrating a light-weight spatial upsampling model, in accordance with some embodiments of the present disclosure.

[0028] Figure 8B is a schematic diagram illustrating another light-weight spatial upsampling model, in accordance with some embodiments of the present disclosure, which is achieved by replacing shallow convolutional layers with bottleneck residual blocks (BRes).

[0029] Figure 8Cis a schematic diagram illustrating BRes including three convolutional layers according to some embodiments of the present disclosure.

[0030] Figure 9A is a flowchart illustrating an exemplary method for encoding a video sequence into a bitstream according to some embodiments of the present disclosure.

[0031] Figure 9B is a flowchart illustrating a sub-step of the method shown in Figure 9A DETAILED DESCRIPTION

[0032] Reference will now be made in detail to the example embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers represent the same or similar elements between the several figures. The implementations set forth in the following description of example embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of apparatuses and methods consistent with aspects related to the present disclosure as described in the appended claims. Specific aspects of the present disclosure are described in further detail below. To the extent not inconsistent with the text of this specification, the terminology used herein merges with that used in the art and / or by the U.S. Patent and Trademark Office.

[0033] The present disclosure relates to "Video Compression for Machines" (VCM), which aims to compress input videos and images or feature maps for machine vision tasks. The disclosed technology is applicable to compress image data used by any machine vision task, such as object recognition and tracking, face recognition, image / video search, mobile augmented reality (MAR), autonomous driving vehicles, Internet of Things (IoT), image matching, three-dimensional structure construction, stereo correspondence, motion tracking, etc.

[0034] Figure 1 is a block diagram illustrating a system 100 for pre-processing and encoding image data according to some disclosed embodiments. The image data can include an image (also referred to as a "picture" or a "frame"), a plurality of images, or a video. An image is a static image. The plurality of images can be spatially or temporally related or unrelated. A video is a set of images arranged in a temporal order.

[0035] As Figure 1 ​As shown, system 100 includes a source device 120 that provides encoded video data for subsequent decoding by a destination device 140. In conformance with the disclosed embodiments, each of source device 120 and destination device 140 can comprise any of a variety of devices, including a desktop computer, a notebook (e.g., laptop) computer, a server, a tablet computer, a set-top box, a phone, a vehicle, a camera, an image sensor, a robot, a television, a wearable device (e.g., a smartwatch or a wearable camera), a display device, a digital media player, a video gaming console, a video streaming device, or the like. Source device 120 and destination device 140 can support wireless or wired communication.

[0036] Referring to Figure 1 Source device 120 can include an image / video preprocessor 122, an image / video encoder 124, and an output interface 126. Destination device 140 can include an input interface 142, an image / video decoder 144, and one or more machine vision applications 146. Image / video preprocessor 122 pre-processes image data, i.e., one or more images or one or more videos, and generates an input bitstream for image / video encoder 124. Image / video encoder 124 encodes the input bitstream and outputs an encoded bitstream 162 via output interface 126. Encoded bitstream 162 is transmitted over communication medium 160 and received by input interface 142. Image / video decoder 144 then decodes encoded bitstream 162 to generate decoded data, which can be used by machine vision applications 146.

[0037] More specifically, source device 120 can also include various devices (not shown) for providing source image data to be pre-processed by image / video preprocessor 122. The devices for providing source image data can include an image / video capture device, such as a camera, an image / video archive or storage device containing previously captured images / videos, or an image / video feed interface for receiving images / videos from an image / video content provider.

[0038] The image / video encoder 124 and the image / video decoder 144 each can be implemented as any of a variety of suitable encoder or decoder circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware or any combinations thereof. When the encoding or decoding is implemented partially in software, the image / video encoder 124 or the image / video decoder 144 can store instructions for the software in a suitable, non- transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of the present disclosure. Each of the image / video encoder 124 or the image / video decoder 144 can be included in one or more encoders or decoders, any of which alone can be integrated

[0039] The image / video encoder 124 and the image / video decoder 144 can operate according to any video coding standards, such as advanced video coding (AVC), high efficiency video coding (HEVC), versatile video coding (VVC), AOMedia Video 1 (AV1), joint photographic experts group (JPEG), moving pictures experts group (MPEG), etc. Alternatively, the image / video encoder 124 and the image / video decoder 144 can be custom devices that do not conform to an existing standard. Although Figure 1 The image / video encoder 124 and the image / video decoder 144 can each be integrated with an audio encoder and decoder, and can include appropriate MUX-DEMUX units, or other hardware and software, to handle encoding of both audio and video in a common data stream or separate data streams, if needed.

[0040] The output interface 126 can include any type of medium or device capable of transmitting the encoded bitstream 162 from the source device 120 to the destination device 140. For example, the output interface 126 can include a transmitter or a transceiver configured to transmit the encoded bitstream 162 from the source device 120 directly to the destination device 140 in real-time. The encoded bitstream 162 can be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the destination device 140.

[0041] Communication medium 160 can include transient media, such as wireless broadcasts or wired networks transmitted over physical transmission lines. For example, communication medium 160 can include radio frequency (RF) spectrum or one or more physical transmission lines, such as coaxial cable, copper wires, or fiber optics. Communication medium 160 can form part of a packet-based network, such as a local area network, a wide-area network, or a global network such as the Internet. In some embodiments, communication medium 160 can include routers, switches, base stations, or any other equipment that can be utilized to facilitate communication from source device 120 to destination device 140. For example, a network server (not shown) can receive encoded bitstream 162 from source device 120 and provide encoded bitstream 162 to destination device 140, such as via network transmission.

[0042] Communication medium 160 can also be in the form of storage media such as non- transitory storage media, for example, a hard disk, flash drive, compact disk, digital video disk, Blu-ray disk, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded picture data. In some embodiments, a computing device of a medium production facility, such as a compact disk pressing facility, can receive encoded picture data from source device 120 and produce a compact disk containing the encoded video data.

[0043] Input interface 142 can include any type of medium or device capable of receiving information from communication medium 160. Received information includes encoded bitstream 162. For example, input interface 142 can include a receiver or transceiver configured to receive encoded bitstream 162 in real-time.

[0044] Machine vision application 146 includes various hardware and / or software for using decoded picture data generated by image / video decoder 144. For example, machine vision application 146 can include a display device configured to display decoded picture data to a user, and can include any of various display devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device. As another example, machine vision application 146 can include one or more processors configured to use the decoded picture data to perform various machine vision applications, such as object recognition and tracking, facial recognition, image matching, image / video search, augmented reality, robot vision and navigation, autonomous driving, three-dimensional structure construction, stereo correspondence, motion tracking, or the like.

[0045] Next, example image data encoding and decoding techniques are described in conjunction with Figures 2A-2B and Figures 3A-3B FIGS. 1-3.

[0046] Figure 2AA schematic diagram of an example encoding process 200A consistent with embodiments of this disclosure is shown. For example, the encoding process 200A may be performed by an encoder, such as... Figure 1 Image / video encoder 124 in the example. Figure 2A As shown, the encoder can encode video sequence 202 into video bitstream 228 according to process 200A. Video sequence 202 may include a set of images (referred to as "raw images") arranged in chronological order. The encoder can divide each raw image of video sequence 202 into multiple basic processing units, multiple basic processing sub-units, or multiple regions for processing. In some embodiments, the encoder can perform process 200A at the level of basic processing units for each raw image of video sequence 202. For example, the encoder can perform process 200A iteratively, wherein the encoder can encode basic processing units in one iteration of process 200A. In some embodiments, the encoder can perform process 200A in parallel for multiple regions of each raw image of video sequence 202.

[0047] exist Figure 2A In this process, the encoder feeds the basic processing unit (referred to as the "raw BPU") of the original image of video sequence 202 to prediction stage 204 to generate prediction data 206 and prediction BPU 208. The encoder subtracts prediction BPU 208 from the raw BPU to generate residual BPU 210. The encoder feeds residual BPU 210 to transform stage 212 and quantization stage 214 to generate quantization transform coefficients 216. The encoder feeds prediction data 206 and quantization transform coefficients 216 to binary encoding stage 226 to generate video bitstream 228. Components 202, 204, 206, 208, 210, 212, 214, 216, 226, and 228 may be referred to as the "forward path". During process 200A, after quantization stage 214, the encoder feeds quantization transform coefficients 216 to inverse quantization stage 218 and inverse transform stage 220 to generate reconstructed residual BPU 222. The encoder can add the reconstructed residual BPU 222 to the prediction BPU 208 to generate a prediction reference 224, which is used in the next iteration of process 200A during prediction phase 204. Components 218, 220, 222, and 224 of process 200A can be referred to as a "reconstruction path." The reconstruction path can be used to ensure that both the encoder and the decoder use the same reference data for prediction.

[0048] The encoder can iteratively perform process 200A to encode each original BPU of the original picture (in the forward path) and generate a prediction reference 224 for encoding a next original BPU of the original picture (in the reconstruction path). After encoding all original BPUs of the original picture, the encoder can proceed to encode a next picture in video sequence 202.

[0049] Referring to process 200A, the encoder can receive a video sequence 202 generated by a video capture device (e.g., a camera). As used herein, the term “receive” can refer to any action that gets, obtains, retrieves, acquires, accesses, or inputs data in any manner.

[0050] At prediction stage 204, at the current iteration, the encoder can receive an original BPU and a prediction reference 224 and perform a prediction operation to generate prediction data 206 and a predicted BPU 208. Prediction reference 224 can be generated from the reconstruction path of a previous iteration of process 200A. The purpose of prediction stage 204 is to reduce information redundancy by extracting, from prediction data 206 and prediction reference 224, prediction data 206 that can be used to reconstruct the original BPU into predicted BPU 208.

[0051] Ideally, predicted BPU 208 can be identical to the original BPU. However, due to non-ideal prediction and reconstruction operations, predicted BPU 208 is typically slightly different from the original BPU. To account for such differences, after generating predicted BPU 208, the encoder can subtract predicted BPU 208 from the original BPU to generate a residual BPU 210. For example, the encoder can subtract values (e.g., grayscale or RGB values) of pixels of predicted BPU 208 from values of corresponding pixels of the original BPU. Each pixel of residual BPU 210 can have a residual value that is generated by such subtraction between corresponding pixel values of the original BPU and predicted BPU 208. Prediction data 206 and residual BPU 210 can have fewer bits than the original BPU, but they can be used to reconstruct the original BPU without apparent quality degradation. Thus, the original BPU is compressed.

[0052] To further compress the residual BPU 210, at a transform stage 212, the encoder can reduce the spatial redundancy of the residual BPU 210 by decomposing the residual BPU 210 into a set of two-dimensional "basis patterns," each basis pattern associated with a "transform coefficient." The basis patterns can have the same size (e.g., the size of the residual BPU 210). Each basis pattern can represent a varying frequency (e.g., of luminance variation) component of the residual BPU 210. No basis pattern can be reproduced by any combination (e.g., linear combination) of any other basis pattern. In other words, the decomposition can decompose the variations of the residual BPU 210 into the frequency domain. This decomposition is similar to a discrete Fourier transform of a function, where the basis patterns are similar to the basis functions (e.g., trigonometric functions) of the discrete Fourier transform, and the transform coefficients are similar to the coefficients associated with the basis functions.

[0053] Different transform algorithms can use different basis patterns. Various transform algorithms can be used at the transform stage 212, such as a discrete cosine transform, a discrete sine transform, and the like. The transform of the transform stage 212 is invertible. That is, the encoder can recover the residual BPU 210 by the inverse operation of the transform (called "inverse transform"). For example, to recover a certain pixel of the residual BPU 210, the inverse transform can be multiplying the values of the corresponding pixels of the basis patterns by the respective associated coefficients and adding the products to produce a weighted sum. For video coding standards, the same transform algorithm (and thus the same basis patterns) can be used by the encoder and the decoder. Thus, the encoder can record only the transform coefficients, and the decoder can reconstruct the residual BPU 210 from the transform coefficients without receiving the basis patterns from the encoder. The transform coefficients can have fewer bits than the residual BPU 210, but they can be used to reconstruct the residual BPU 210 without noticeable quality degradation. Thus, the residual BPU 210 is further compressed.

[0054] The encoder can further compress the transform coefficients at a quantization stage 214. During the transform process, different basis modes can represent different frequencies of variation (e.g., luminance frequency of variation). Because the human eye is generally better at recognizing low frequency variations, the encoder can ignore information of high frequency variations without causing noticeable quality degradation in decoding. For example, at the quantization stage 214, the encoder can generate quantized transform coefficients 216 by dividing each transform coefficient by an integer value (referred to as a "quantization parameter") and performing a round-off operation on the quotient. After this operation, some transform coefficients of high frequency basis modes can be converted to zero, while the transform coefficients of low frequency basis modes can be converted to smaller integers. The encoder can ignore the quantized transform coefficients 216 of zero value, whereby transform coefficients are further compressed. The above-mentioned quantization process is also reversible, in which the quantized transform coefficients 216 can be reconstructed to the transform coefficients in an inverse operation of quantization (referred to as "dequantization").

[0055] Because the encoder ignores the remainder of this division in the round-off operation, the quantization stage 214 can be lossy. Generally, the quantization stage 214 can constitute the most dominant source of information loss in the process 200A. The greater the information loss, the fewer bits the quantized transform coefficients 216 can require. To obtain different degrees of information loss, the encoder can use different values of the quantization parameter or any other parameter of the quantization process.

[0056] At a binary encoding stage 226, the encoder can encode the prediction data 206 and the quantized transform coefficients 216 using binary encoding techniques, such as entropy encoding, variable length encoding, arithmetic encoding, Huffman encoding, context adaptive binary arithmetic encoding, or any other lossless or lossy compression algorithm, for example. In some embodiments, in addition to the prediction data 206 and the quantized transform coefficients 216, the encoder can also encode other information at the binary encoding stage 226, such as, for example, the prediction modes used at the prediction stage 204, parameters of the prediction operations, the transform types of the transform stage 212, parameters of the quantization process (e.g., quantization parameter), encoder control parameters (e.g., bit rate control parameters), and the like. The encoder can use the output data of the binary encoding stage 226 to generate a video bitstream 228. In some embodiments, the video bitstream 228 can be further packetized for network transmission.

[0057] Referring to the reconstruction path of process 200A, at inverse quantization stage 218, the encoder can perform inverse quantization on quantized transform coefficients 216 to generate reconstructed transform coefficients. At inverse transform stage 220, the encoder can generate reconstructed residual BPU 222 based on the reconstructed transform coefficients. The encoder can add reconstructed residual BPU 222 to predicted BPU 208 to generate prediction reference 224 to be used in the next iteration of process 200A.

[0058] It should be noted that other variants of process 200A can be used to encode video sequence 202. In some embodiments, the stages of process 200A can be performed in a different order by the encoder. In some embodiments, one or more stages of process 200A can be combined into a single stage. In some embodiments, a single stage of process 200A can be divided into multiple stages. For example, transform stage 212 and quantization stage 214 can be combined into a single stage. In some embodiments, process 200A can include additional stages. In some embodiments, process 200A can omit one or more stages of Figure 2A

[0059] Figure 2B A schematic diagram illustrating another example encoding process 200B consistent with embodiments of the present disclosure is shown. Process 200B can be modified from process 200A. For example, process 200B can be used by an encoder that conforms to a hybrid video coding standard (e.g., the H.26x family). In comparison to process 200A, the forward path of process 200B additionally includes mode decision stage 230, and divides prediction stage 204 into spatial prediction stage 2042 and temporal prediction stage 2044. The reconstruction path of process 200B additionally includes loop filtering stage 232 and buffer 234.

[0060] Generally, prediction techniques can be divided into two types: spatial prediction and temporal prediction. Spatial prediction (e.g., intra-picture prediction or "intra-frame prediction") can use pixels from one or more already encoded neighboring BPUs in the same picture to predict the current BPU. That is, prediction reference 224 in spatial prediction can include the neighboring BPUs. The spatial prediction can reduce the inherent spatial redundancy of pictures. Temporal prediction (e.g., inter-picture prediction or "inter-frame prediction") can use regions from one or more already encoded pictures to predict the current BPU. That is, prediction reference 224 in temporal prediction can include the already encoded pictures. The temporal prediction can reduce the inherent temporal redundancy of pictures.

[0061] ​Referring to process 200B, in the forward path, the encoder performs prediction operations at spatial prediction stage 2042 and temporal prediction stage 2044. For example, at spatial prediction stage 2042, the encoder can perform intra prediction. For a certain original BPU of a certain picture being encoded, prediction reference 224 can include one or more neighboring BPUs that have been encoded (in the forward path) and reconstructed (in the reconstruction path) in the same picture. The encoder can generate predicted BPU 208 by extrapolating the neighboring BPUs. The extrapolation technique can include, for example, linear extrapolation or interpolation, polynomial extrapolation or interpolation, etc. In some embodiments, the encoder can perform the extrapolation at the pixel level, e.g., extrapolate a corresponding pixel value for each pixel of predicted BPU 208. The neighboring BPUs used for extrapolation can be located from various directions relative to the original BPU, e.g., in a vertical direction (e.g., above the original BPU), a horizontal direction (e.g., to the left of the original BPU), a diagonal direction (e.g., to the lower left, lower right, upper left, or upper right of the original BPU), or any direction defined in the video coding standard used. For intra prediction, prediction data 206 can include, for example, the location (e.g., coordinates) of the neighboring BPU used, the size of the neighboring BPU used, the parameters of the extrapolation, the direction of the neighboring BPU used relative to the original BPU, etc.

[0062] For another example, at the temporal prediction stage 2044, the encoder can perform the inter prediction. For a certain original BPU of the current image, the prediction reference 224 can include one or more images (referred to as "reference images") that have been encoded (in the forward path) and reconstructed (in the reconstruction path). In some embodiments, the reference images can be encoded and reconstructed on a BPU-by-BPU basis. For example, the encoder can add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a reconstructed BPU. When all reconstructed BPUs of the same image are generated, the encoder can generate a reconstructed image as a reference image. The encoder can perform a "motion estimation" operation to search for a matching region within a certain range (referred to as a "search window") of the reference image. The location of the search window in the reference image can be determined based on the location of the original BPU in the current image. For example, the search window can be centered at a location in the reference image that has the same coordinates as the original BPU in the current image, and can extend outward by a predetermined distance. When the encoder identifies (e.g., by using a pixel recursive algorithm, a block matching algorithm, etc.) a region in the search window that is similar to the original BPU, the encoder can determine such a region as a matching region. The matching region can have a different size (e.g., smaller, equal, larger, or have a different shape) than the original BPU. Because the reference image and the current image are temporally separated in the time axis, the matching region can be considered to "move" to the location of the original BPU over time. The encoder can record the direction and distance of such motion as a "motion vector." When multiple reference images are used, the encoder can search for a matching region for each reference image and determine an associated motion vector for the matching region. In some embodiments, the encoder can assign weights to the pixel values of the matching region for individual reference images.

[0063] The motion estimation can be used to identify various types of motion, such as translation, rotation, scaling, etc. For inter prediction, the prediction data 206 can include, for example, the location (e.g., coordinates) of the matching region, the motion vector associated with the matching region, the number of reference images, the weights associated with the reference images, etc.

[0064] To generate the predicted BPU 208, the encoder can perform a "motion compensation" operation. The motion compensation can be used to reconstruct the predicted BPU 208 based on the prediction data 206 (e.g., motion vectors) and the prediction reference 224. For example, the encoder can move the matching region of the reference image according to the motion vectors, whereby the encoder can predict the original BPU of the current image. When multiple reference images are used, the encoder can move the matching region of a reference image according to the respective motion vectors and average pixel values of the matching region. In some embodiments, if the encoder has assigned weights to the pixel values of the matching region of the respective matching reference images, the encoder can perform a weighted sum of the pixel values of the moved matching region.

[0065] In some embodiments, the inter-frame prediction can be uni-directional or bi-directional. Uni-directional inter-frame prediction can use one or more reference images in the same temporal direction relative to the current image. Uni-directional inter-frame prediction uses a reference image that precedes the current image. Bi-directional inter-frame prediction can use one or more reference images in both temporal directions relative to the current image.

[0066] Still referring to the forward path of the process 200B, after the spatial prediction stage 2042 and the temporal prediction stage 2044, at a mode decision stage 230, the encoder can select a prediction mode (e.g., one of intra-prediction or inter-prediction) for the current iteration of the process 200B. For example, the encoder can perform a rate-distortion optimization technique, in which the encoder can select a prediction mode according to the bit rate of the candidate prediction mode and the distortion of the reconstructed reference image under the candidate prediction mode to minimize the value of a cost function. According to the selected prediction mode, the encoder can generate a corresponding predicted BPU 208 and prediction data 206.

[0067] In the reconstruction path of process 200B, if an intra-frame prediction mode has already been selected in the forward path, the encoder can directly feed the prediction reference 224 (e.g., the current BPU that has been encoded and reconstructed in the current image) to the spatial prediction stage 2042 for subsequent use (e.g., for extrapolation of the next BPU in the current image) after generating the prediction reference 224. If an inter-frame prediction mode has already been selected in the forward path, the encoder can feed the prediction reference 224 (e.g., the current image where all BPUs have been encoded and reconstructed) to the loop filtering stage 232 after generating the prediction reference 224, in which the encoder can apply loop filtering to the prediction reference 224 to reduce or eliminate distortions (e.g., blockiness) introduced by the inter-frame prediction. The encoder can apply various loop filtering techniques in the loop filtering stage 232, such as deblocking, sample adaptive shifting, adaptive loop filtering, etc. The loop-filtered reference image can be stored in buffer 234 (or “decoded image buffer”) for subsequent use (e.g., as an inter-frame prediction reference image for future images of video sequence 202). The encoder can store one or more reference images in buffer 234 for use in the temporal prediction stage 2044. In some embodiments, the encoder can encode the loop-filter parameters (e.g., loop-filter strength), as well as the quantization transform coefficients 216, prediction data 206, and other information in the binary encoding stage 226.

[0068] Figure 3A A schematic diagram of an example decoding process 300A consistent with embodiments of this disclosure is shown. Process 300A may be corresponding to... Figure 2A The decompression process of the compression process 200A described in the previous step. In some embodiments, process 300A may be similar to the reconstruction path described in process 200A. Decoder (e.g., Figure 1 The image / video decoder 144 in the image / video decoder can decode the video bitstream 228 into a video stream 304 according to process 300A. The video stream 304 can be very similar to the video sequence 202. However, due to information loss during compression and decompression (e.g., Figures 2A-2B In the quantization stage 214, video stream 304 typically differs from video sequence 202. Similar to... Figures 2A-2BIn some embodiments, the decoder can perform process 300A for each picture encoded in video bitstream 228 at the level of a basic processing unit (BPU). For example, the decoder can perform process 300A in an iterative manner, where the decoder can decode one basic processing unit in one iteration of process 300A. In some embodiments, the decoder can perform process 300A in parallel for multiple regions of each picture encoded in video bitstream 228.

[0069] In Figure 3A In some embodiments, the decoder can perform process 300A for each picture encoded in video bitstream 228 at the level of a basic processing unit (BPU). For example, the decoder can perform process 300A in an iterative manner, where the decoder can decode one basic processing unit in one iteration of process 300A. In some embodiments, the decoder can perform process 300A in parallel for multiple regions of each picture encoded in video bitstream 228.

[0070] The decoder can iteratively perform process 300A to decode each coded BPU of a coded picture and generate a prediction reference 224 for encoding a next coded BPU of the coded picture. After decoding all coded BPUs of the coded picture, the decoder can output the picture to video stream 304 for display and continue decoding a next coded picture in video bitstream 228.

[0071] At the binarization decoding stage 302, the decoder can perform the inverse operation of the binarization technique used by the encoder (e.g., entropy coding, variable length coding, arithmetic coding, Huffman coding, context adaptive binary arithmetic coding, or any other lossless compression algorithm). In some embodiments, in addition to the prediction data 206 and the quantized transform coefficients 216, the decoder can also decode other information at the binarization decoding stage 302, such as, for example, prediction modes, parameters of the prediction operations, transform types, parameters of the quantization process (e.g., quantization parameter), encoder control parameters (e.g., bit rate control parameters), and the like. In some embodiments, if the video bitstream 228 is transmitted over a network in the form of packets, the decoder can depacketize the video bitstream 228 before feeding the video bitstream to the binarization decoding stage 302.

[0072] Figure 3B A diagram illustrating another example decoding process 300B consistent with embodiments of the present disclosure is shown. The process 300B can be modified from the process 300A. For example, the process 300B can be used by a decoder conforming to a hybrid video coding standard (e.g., the H.26x family). Compared to the process 300A, the process 300B additionally divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044, and additionally includes a loop filtering stage 232 and a buffer 234.

[0073] In the process 300B, for an encoded base processing unit (referred to as a "current BPU") of an encoded image being decoded (referred to as a "current image"), the prediction data 206 decoded by the decoder from the binarization decoding stage 302 can include various types of data depending on what prediction mode was used by the encoder to encode the current BPU. For example, if the encoder used intra prediction to encode the current BPU, the prediction data 206 can include a prediction mode indicator (e.g., a flag value) indicating the intra prediction, parameters of the intra prediction operation, and the like. The parameters of the intra prediction operation can include, for example, locations (e.g., coordinates) of one or more neighboring BPUs used as references, sizes of the neighboring BPUs, parameters of extrapolation, directions of the neighboring BPUs relative to the original BPU, and the like. For another example, if the encoder used inter prediction to encode the current BPU, the prediction data 206 can include a prediction mode indicator (e.g., a flag value) indicating the inter prediction, parameters of the inter prediction operation, and the like. The parameters of the inter prediction operation can include, for example, a number of reference images associated with the current BPU, weights respectively associated with the reference images, locations (e.g., coordinates) of one or more matching regions in the respective reference images, one or more motion vectors respectively associated with the matching regions, and the like.

[0074] Based on the prediction mode indicator, the decoder can determine whether to perform spatial prediction (e.g., intra prediction) at the spatial prediction stage 2042 or to perform temporal prediction (e.g., inter prediction) at the temporal prediction stage 2044. Figure 2B Details of performing such spatial prediction or temporal prediction are described in Figure 3A , which will not be repeated here. After performing such spatial prediction or temporal prediction, the decoder can generate a predicted BPU 208. The decoder can add the predicted BPU 208 and the reconstructed residual BPU 222 to generate a prediction reference 224, as

[0075] At the process 300B, the decoder can feed the prediction reference 224 to the spatial prediction stage 2042 or the temporal prediction stage 2044 for performing a prediction operation in the next iteration of the process 300B. For example, if the current BPU is decoded using the intra prediction at the spatial prediction stage 2042, after generating the prediction reference 224 (e.g., the decoded current BPU), the decoder can feed the prediction reference 224 directly to the spatial prediction stage 2042 for subsequent use (e.g., for extrapolation of the next BPU of the current picture). If the current BPU is decoded using the inter prediction at the temporal prediction stage 2044, after generating the prediction reference 224 (e.g., a reference picture in which all BPU have been decoded), the encoder can feed the prediction reference 224 to the in-loop filtering stage 232 to reduce or remove distortion (e.g., blocking artifacts). The decoder can apply in-loop filtering to the prediction reference 224 in a manner as described in Figure 2B The in-loop filtered reference picture can be stored in the buffer 234 (e.g., a decoded picture buffer in computer memory) for subsequent use (e.g., as an inter prediction reference picture for some future encoded picture of the video bitstream 228). The decoder can store one or more reference pictures in the buffer 234 for use at the temporal prediction stage 2044. In some embodiments, when the prediction mode indicator of the prediction data 206 indicates that the current BPU is encoded using inter prediction, the prediction data can further include parameters of in-loop filtering (e.g., in-loop filtering strength).

[0076] Referring back to Figure 1 , each of the image / video pre-processor 122, the image / video encoder 124, and the image / video decoder 144 can be implemented as any suitable hardware, software, or combination thereof. Figure 4 is a block diagram of an example apparatus 400 for processing image data, consistent with embodiments of the present disclosure. For example, the apparatus 400 can be a pre-processor, an encoder, or a decoder. AsFigure 4 As shown, device 400 may include processor 402. When processor 402 executes the instructions described herein, device 400 may become a dedicated machine for preprocessing, encoding, and / or decoding image data. Processor 402 may be any type of circuit system capable of manipulating or processing information. For example, processor 402 may include any number of central processing units (or “CPU”), graphics processing units (or “GPU”), neural processing units (“NPU”), microcontroller units (“MCU”), optical processors, programmable logic controllers, microcontrollers, microprocessors, digital signal processors, intellectual property (IP) cores, programmable logic arrays (PLAs), programmable array logic (PALs), general-purpose array logic (GALs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), system-on-a-chip (SoCs), application-specific integrated circuits (ASICs), and any combination thereof. In some embodiments, processor 402 may also be a set of processors grouped into individual logic components. For example, as Figure 4 As shown, processor 402 may include multiple processors, including processor 402a, processor 402b and processor 402n.

[0077] The device 400 may also include a memory 404 configured to store data (e.g., instruction sets, computer code, intermediate data, etc.). For example, such as Figure 4 As shown, the stored data may include program instructions (e.g., program instructions for implementing the stages in processes 200A, 200B, 300A, or 300B) and data for processing (e.g., video sequence 202, video bitstream 228, or video stream 304). Processor 402 can access the program instructions and data for processing (e.g., via bus 410) and execute the program instructions to perform operations or manipulations on the data for processing. Memory 404 may include a high-speed random access memory device or a non-volatile memory device. In some embodiments, memory 404 may include any combination of any number of random access memories (RAM), read-only memories (ROM), optical discs, magnetic disks, hard disks, solid-state drives, flash drives, secure digital cards (SD cards), memory sticks, compact flash (CF) cards, etc. Memory 404 may also be a group of memories grouped into a single logical component. Figure 4 (Not shown in the image).

[0078] Bus 410 may be a communication device for transmitting data between components within device 400, such as an internal bus (e.g., CPU-memory bus), an external bus (e.g., a Universal Serial Bus port, a Peripheral Component Interconnect Fast Port), etc.

[0079] For ease of explanation and without causing ambiguity, the processor 402 and other data processing circuitry are collectively referred to as“data processing circuitry” in this disclosure. The data processing circuitry can be implemented entirely as hardware, or as a combination of software, hardware, or firmware. Moreover, the data processing circuitry can be a single standalone module, or can be combined in whole or in part into any other component of the apparatus 400.

[0080] The apparatus 400 can also include a network interface 406 to provide wired or wireless communication with a network (e.g., the Internet, an intranet, a local area network, a mobile communications network, etc.). In some embodiments, the network interface 406 can include any combination of network interface controllers (NICs), radio frequency (RF) modules, transponders, transceivers, modems, routers, gateways, wired network adapters, wireless network adapters, Bluetooth adapters, infrared adapters, near field communication (“NFC”) adapters, cellular network chips, etc.

[0081] In some embodiments, the apparatus 400 can further include a peripheral interface 408 to provide connectivity to one or more peripheral devices, as shown. The peripheral devices can include, but are not limited to, a cursor control device (e.g., a mouse, touchpad, or touchscreen), a keyboard, a display (e.g., a cathode ray tube display, liquid crystal display, or light emitting diode display), a video input device (e.g., a camera or input interface coupled to a video archive), etc. Figure 4

[0082] It should be noted that a video codec (e.g., a codec that performs the processes 200A, 200B, 300A, or 300B) can be implemented as any combination of any number or software or hardware modules in the apparatus 400. For example, some or all stages of the processes 200A, 200B, 300A, or 300B can be implemented as one or more software modules of the apparatus 400, e.g., program instructions that can be loaded into the memory 404. For another example, some or all stages of the processes 200A, 200B, 300A, or 300B can be implemented as one or more hardware modules of the apparatus 400, e.g., a special purpose data processing circuit (e.g., an FPGA, ASIC, NPU, etc.).

[0083] ​It is observed that the amount of visual data has significantly increased in recent decades. This trend can be attributed to the progress in multimedia acquisition, representation, and application technologies. In various visual data-centric applications, efficient representation of visual data is of paramount importance. Visual data exhibits a large amount of spatial correlation, and several spatial resampling techniques have been explored to improve compression efficiency while ensuring optimal human perceptual quality. The advent of machine vision has further accelerated this trend, as machines have surpassed human perception as the primary consumer of visual data in a variety of applications such as smart security and autonomous driving.

[0084] Various algorithms have been proposed to improve compression efficiency for human perception. Specifically, a down-sampling based paradigm has been proposed to improve image compression performance at low bitrates. Local random convolution kernels have been proposed to preserve high-frequency information at low bitrate compression. In the era of deep learning, an end-to-end image compression framework based on convolutional neural networks has been established, which has achieved significant performance improvement compared to several existing coding standards.

[0085] Spatial resampling can also be applied to machine vision-oriented visual data compression. Specifically, a joint loss function consisting of a signal-level distortion and a machine vision loss function has been proposed to improve a spatial resampling model for machine vision-oriented tasks. Furthermore, to boost spatial resampling performance, a codec simulation network has been proposed for joint optimization with the spatial resampling model. Although performance improvement has been achieved through spatial resampling, the representation overhead of the spatial up-sampling model has not been fully considered, which can introduce additional bandwidth cost if the up-sampling model needs to be transmitted.

[0086] Many spatial resampling algorithms have been proposed for machine vision, achieving significant performance improvement. Figure 5 A typical spatial resampling-based compression framework 500 is shown. Input visual data is rescaled by a spatial down-sampling model implemented in the spatial down-sampling module of the encoding end, and the output down-sampled data is compressed by the encoding module. The compressed image can then be packaged into a bitstream and transmitted to the decoding end. At the decoding end, the decompressed down-sampled data is obtained by the decoding module. Based on the resampling ratio transmitted with the bitstream, the reconstructed visual data is the output of the spatial up-sampling module. However, the spatial up-sampling model can be complex in structure and represented by a large number of encoding bits, and therefore should be transmitted with the bitstream, as the decoding end can not be configured with the up-sampling model. This results in additional transmission cost and reduces encoding efficiency. Therefore, there is a great need for a lightweight spatial up-sampling model.

[0087] The present disclosure provides embodiments of using a lightweight spatial up-sampling model to decode video data suitable for machine vision tasks, which will be described in detail below.

[0088] Figure 6A is a flowchart illustrating an exemplary method 600 for decoding a bitstream to output one or more images of a video stream according to some embodiments of the present disclosure. As Figure 6A illustrated, the method 600 can include the following steps 602 and 604, which can be implemented by one or more processors associated with a decoder (e.g., the image / video decoder 144 or Figure 1 the apparatus 400 in the electronic device 100) in the electronic device 100. Here, the decoder can include a decoding module (e.g., the decoding module in the electronic device 100) for decompressing the encoded images, and an up-sampling module (e.g., the spatial up-sampling module in the electronic device 100) for up-sampling the decompressed images. Figure 4 Figure 5 Figure 5

[0089] In step 602, the decoder can receive a bitstream.

[0090] In step 604, the decoder can decode one or more images using the encoded information of the bitstream. Specifically, the decoder can perform spatial up-sampling on the images to obtain corresponding reconstructed images, e.g., for machine vision applications.

[0091] Figure 6B is a flowchart illustrating sub-steps of the method 600 shown in Figure 6A FIG. 6B according to some embodiments of the present disclosure. As Figure 6B illustrated, decoding one or more images using the encoded information of the bitstream in step 604 can include the following sub-steps 612 and 614, which can be implemented by the decoder, specifically, the decoding module and the up-sampling module. Figure 6B

[0092] In sub-step 612, the decoding module of the decoder can generate one or more decompressed images by decompressing one or more compressed images contained in the bitstream. The decoding module can decode the images according to the standard followed for encoding the images.

[0093] ​​​​In sub-step 614, the up-sampling module of the decoder can perform spatial up-sampling on the one or more decompressed images by a light-weight spatial up-sampling model to obtain one or more reconstructed images, respectively. That is, the light-weight spatial up-sampling model can be implemented in the up-sampling module. Here, the total number of encoded bits of the parameters of the light-weight spatial up-sampling model can be less than a threshold value predetermined based on the expected quality of the reconstructed images. That is, compared with a conventional up-sampling scheme, the structure of the decoder, in particular, the up-sampling module, can be represented with fewer bits.

[0094] In some embodiments, the decoder can be equipped with the necessary hardware required by the up-sampling module. However, the structure of the up-sampling module can need to be implemented according to additional configuration information. For example, in some embodiments, step 604 can further include sub-step 616, which is also implemented by the decoder. In sub-step 616, the decoder, in particular, the up-sampling module, can extract the parameters contained in the bitstream. For example, the parameters can include the weights and biases of each layer of the light-weight spatial up-sampling model. As mentioned above, the encoded bits of the parameters of the light-weight spatial up-sampling module can be less than the threshold value. Compared with a conventional up-sampling scheme, the single parameters in the bitstream can be significantly reduced. Therefore, the coding efficiency of the video stream can be improved.

[0095] In some embodiments, the light-weight spatial up-sampling model can be implemented by reducing the convolutional layers of the network. Figure 7A is a schematic diagram illustrating an exemplary spatial up-sampling model 700A, while Figure 7B is a schematic diagram illustrating another exemplary spatial up-sampling model 700B implemented by reducing the number of network layers according to some embodiments of the present disclosure. Specifically, the light-weight spatial up-sampling model 700B is implemented by reducing the number of convolutional layers from 20 Figure 7A to 11 Figure 7B That is, for the light-weight spatial up-sampling model, there can be 9 intermediate convolutional layers between the input convolutional layer and the output convolutional layer. If the parameters of each layer are all encoded with equal length, the number of bits used to encode the parameters of the light-weight spatial up-sampling model will be reduced by 9 / 20.

[0096] In some embodiments, as Figure 7BAs shown, the input convolutional layer can be configured as a convolutional layer with a kernel size of 3, a stride of 1, 3 input channels, and 64 output channels. Each of the nine intermediate convolutional layers can be configured as a convolutional layer with a kernel size of 3, a stride of 1, 64 input channels, and 64 output channels. The output convolutional layer can be configured as a convolutional layer with a kernel size of 3, a stride of 1, 64 input channels, and 3 output channels.

[0097] Figure 8A is a schematic diagram illustrating a lightweight spatial up-sampling model 800A, which shows the detailed structure of the lightweight spatial up-sampling model 700A shown in Figure 7B Figure 8B is a schematic diagram illustrating another lightweight spatial up-sampling model 800B implemented by replacing shallow convolutional layers with bottleneck residual blocks (Bres) according to some embodiments of the present disclosure. In some embodiments, the lightweight spatial up-sampling model can include an input convolutional layer, an output convolutional layer, and three bottleneck residual blocks, as shown in Figure 8B Specifically, the input convolutional layer can be configured as a convolutional layer with a kernel size of 3, a stride of 1, 3 input channels, and 64 output channels. The output convolutional layer can be configured as a convolutional layer with a kernel size of 3, a stride of 1, 64 input channels, and 3 output channels.

[0098] In some embodiments, each bottleneck residual block can include three convolutional layers, as shown in Figure 8C Specifically, the first (top) and third (bottom) convolutional layers are responsible for dimension reduction and recovery of data, respectively, using 1x1 convolutional kernels, thereby improving computational efficiency. The second (middle) convolutional layer is responsible for feature extraction using a 3x3 convolutional kernel. The outputs of the first and second layers are activated with a linear rectifier function (Relu). As shown in Figure 8C Specifically, the first convolutional layer can be configured as a convolutional layer with a kernel size of 1, a stride of 1, 64 input channels, and 32 output channels. The second convolutional layer can be configured as a convolutional layer with a kernel size of 3, a stride of 1, 32 input channels, and 32 output channels. The third convolutional layer can be configured as a convolutional layer with a kernel size of 1, a stride of 1, 32 input channels, and 64 output channels.

[0099] In some embodiments, the spatial up-sampling model can be lightened by reducing numerical precision. For example, the parameters of the lightweight spatial up-sampling model are quantized into a predetermined format. Specifically, the model weights can be quantized to float16, int8, or even binary bit depth for inference, achieving a significant reduction in representation overhead.

[0100] ​By combining convolutional layer reduction, bottleneck residual blocks, and reduced numerical precision, the lightweight nature of spatial upsampling models can be further improved.

[0101] Figure 9A This is a flowchart illustrating an exemplary method 900 for encoding a video sequence into a bitstream according to some embodiments of the present disclosure. Figure 9A As shown, method 900 includes steps 902 to 906, which can be performed by an encoder (e.g., Figure 1 Image / video encoder 124 or Figure 4 The encoder is implemented by one or more processors associated with the device 400 in the document. Here, the encoder may include a downsampling module (e.g., Figure 5 The spatial downsampling module in the image is used to downsample the image, and the encoding module (e.g., Figure 5 The encoding module in the image is used to compress the downsampled image.

[0102] In step 902, the encoder can receive a video sequence.

[0103] In step 904, the encoder may encode one or more images of the video sequence.

[0104] In step 906, the encoder generates a bitstream associated with the encoded image. The bitstream may include the encoded result generated in step 904.

[0105] Figure 9B This illustrates some embodiments according to the present disclosure. Figure 9A The flowchart illustrates the sub-steps of method 900. In some embodiments, encoding one or more images of the video sequence in step 904 may include, for example... Figure 9B The following sub-steps 912 to 916 are shown. These sub-steps can be implemented by the encoder, specifically the downsampling module and the encoding module.

[0106] In sub-step 912, the downsampling module of the encoder can spatially downsample the one or more images using a spatial downsampling model to obtain one or more downsampled images respectively. The downsampling model can be configured to include three convolutional layers, a structure adopted by video coding standards.

[0107] In sub-step 914, the encoding module of the encoder can generate one or more compressed images by compressing one or more downsampled images.

[0108] In sub-step 916, the down-sampling module can generate parameters of a light-weight spatial up-sampling model for decoding one or more compressed images at a decoding end. That is, the encoder can determine the parameters of the up-sampling model in the decoder and send the parameters for use by the decoder. In some embodiments, the structure of the encoder can be fixed, e.g., including three convolutional layers, while the structure of the decoder can be determined by the designer of the encoder according to his desired quality of the reconstructed image. In some embodiments, the total number of coded bits for the parameters of the light-weight spatial up-sampling model can be less than a threshold value predetermined based on the desired quality of the reconstructed image based on the one or more images.

[0109] Generally speaking, machine vision can not require ultra-high resolution images. Thus, the light-weight spatial up-sampling model can be relatively small in scale to save the number of bits required to encode its parameters. As a result, the coding efficiency and transmission capacity can be improved.

[0110] In some embodiments, the decoder can configure the necessary hardware required for the up-sampling module. However, the structure of the up-sampling module can need to be implemented according to additional configuration information. In some embodiments, for example, in step 906, the encoder can also send the parameters of the light-weight spatial up-sampling model into the bitstream. For example, the parameters can include the weights and biases of the layers of the light-weight spatial up-sampling model. As mentioned above, the number of coded bits for the parameters of the light-weight spatial up-sampling model can be less than the threshold value. The single parameter in the bitstream can be significantly reduced compared to conventional up-sampling schemes.

[0111] As mentioned above, the light-weight spatial up-sampling model can be implemented by reducing the convolutional layers of the network to reduce the parameters to be sent in the bitstream. Figure 7A is a schematic diagram illustrating an exemplary spatial up-sampling model 700A, while Figure 7B is a schematic diagram illustrating another exemplary spatial up-sampling model 700B implemented by reducing the number of network layers according to some embodiments of the present disclosure. The light-weight spatial up-sampling model 700B is implemented by reducing the number of convolutional layers from 20 Figure 7A to 11 Figure 7B That is, for the light-weight spatial up-sampling model, there can be 9 intermediate convolutional layers between the input convolutional layer and the output convolutional layer. If the parameters of each layer are all encoded in equal length, the number of bits used to encode the parameters of the light-weight spatial up-sampling model will be reduced by 9 / 20.

[0112] As Figure 7BAs shown, the input convolutional layer can be configured as a convolutional layer with a kernel size of 3, a stride of 1, 3 input channels, and 64 output channels. Each of the nine intermediate convolutional layers can be configured as a convolutional layer with a kernel size of 3, a stride of 1, 64 input channels, and 64 output channels. The output convolutional layer can be configured as a convolutional layer with a kernel size of 3, a stride of 1, 64 input channels, and 3 output channels.

[0113] As shown in FIG. 2, the spatial up-sampling model can be implemented by replacing the shallow convolutional layers with bottleneck residual blocks (BRes). For example, the spatial up-sampling model can include an input convolutional layer, an output convolutional layer, and three bottleneck residual blocks, as shown in FIG. 2. Specifically, the input convolutional layer can be configured as a convolutional layer with a kernel size of 3, a stride of 1, 3 input channels, and 64 output channels. The output convolutional layer can be configured as a convolutional layer with a kernel size of 3, a stride of 1, 64 input channels, and 3 output channels. Figure 8A and Figure 8B As shown, the lightweight spatial up-sampling model can be implemented by replacing the shallow convolutional layers with bottleneck residual blocks (BRes). For example, the lightweight spatial up-sampling model can include an input convolutional layer, an output convolutional layer, and three bottleneck residual blocks, as shown in FIG. 2. Specifically, the input convolutional layer can be configured as a convolutional layer with a kernel size of 3, a stride of 1, 3 input channels, and 64 output channels. The output convolutional layer can be configured as a convolutional layer with a kernel size of 3, a stride of 1, 64 input channels, and 3 output channels. Figure 8B As shown in FIG. 2, the spatial up-sampling model can be implemented by replacing the shallow convolutional layers with bottleneck residual blocks (BRes). For example, the spatial up-sampling model can include an input convolutional layer, an output convolutional layer, and three bottleneck residual blocks, as shown in FIG. 2. Specifically, the input convolutional layer can be configured as a convolutional layer with a kernel size of 3, a stride of 1, 3 input channels, and 64 output channels. The output convolutional layer can be configured as a convolutional layer with a kernel size of 3, a stride of 1, 64 input channels, and 3 output channels.

[0114] Figure 8C In some embodiments, each of the bottleneck residual blocks can include three convolutional layers, as shown in FIG. 3. Specifically, the first and third convolutional layers are responsible for data dimension reduction and recovery, respectively, using 1x1 convolutional kernels, thereby improving computational efficiency. The second convolutional layer is responsible for feature extraction using a 3x3 convolutional kernel. The outputs of the first and second layers are activated using Relu, as shown in FIG. 3. Figure 8C As shown, the first convolutional layer can be configured as a convolutional layer with a kernel size of 1, a stride of 1, 64 input channels, and 32 output channels. The second convolutional layer can be configured as a convolutional layer with a kernel size of 3, a stride of 1, 32 input channels, and 32 output channels. The third convolutional layer can be configured as a convolutional layer with a kernel size of 1, a stride of 1, 32 input channels, and 64 output channels.

[0115] In some embodiments, the spatial up-sampling model can be lightened by reducing numerical precision. For example, the parameters of the lightweight spatial up-sampling model can be quantized into a predetermined format. Specifically, the model weights can be quantized into float16, int8, or even binary bit depth for inference, achieving a significant reduction in representation overhead.

[0116] In some embodiments, to further reduce the parameters to be sent in the bitstream, the degree of lightening of the spatial up-sampling model can be further improved by jointly using convolutional layer reduction, bottleneck residual blocks, and numerical precision reduction.

[0117] ​It should be understood that embodiments of the present disclosure can be combined with another embodiment or some other embodiments.

[0118] In some embodiments, a non-transitory computer-readable storage medium storing one or more bitstreams processed according to the above method is also provided. For example, the one or more bitstreams can be encoded and decoded using the lightweight spatial upsampling model shown in FIGS. 7B, 8A, 8B, 8C, 9A, and 9B. Figure 6A 、 6B

[0119] In some embodiments, a non-transitory computer-readable storage medium including instructions is also provided, and the instructions can be executed by a device (e.g., the disclosed encoder and decoder) for performing the above method. Common forms of non-transitory media include, for example, a floppy disk, a flexible disk, a hard disk, a solid-state drive, a magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM or any other flash memory, NVRAM, a cache, a register, any other memory chip or cartridge, and a networked version of any of the foregoing. The device can include one or more processors (CPUs), input / output interfaces, network interfaces, and / or memory.

[0120] Embodiments can be further described using the following clauses: 1. A method for decoding a bitstream to output one or more images of a video stream, the method comprising: receiving a bitstream; and decoding one or more images using encoded information of the bitstream, wherein decoding the one or more images using the encoded information of the bitstream comprises: decompressing one or more compressed images included in the bitstream; and performing spatial upsampling on the one or more decompressed images by a spatial upsampling model to obtain one or more reconstructed images, respectively, wherein a total number of encoded bits of parameters of the spatial upsampling model is less than a threshold predetermined based on a desired quality of the reconstructed images.

[0121] 2. The method of clause 1, wherein the decoding further comprises: extracting parameters from the bitstream, the parameters comprising weights and biases of layers of the spatial upsampling model.

[0122] ​3. The method of clause 1 or 2, wherein the spatial upsampling model comprises: 11 convolutional layers.

[0123] 4. The method of clause 3, wherein the spatial upsampling model comprises: an input convolutional layer with a kernel size of 3, a stride of 1, an input number of channels of 3, and an output number of channels of 64; nine intermediate convolutional layers with a kernel size of 3, a stride of 1, an input number of channels of 64, and an output number of channels of 64; and an output convolutional layer with a kernel size of 3, a stride of 1, an input number of channels of 64, and an output number of channels of 3.

[0124] 5. The method of clause 1 or 2, wherein the spatial upsampling model comprises one or more bottleneck residual blocks, BRes.

[0125] 6. The method of clause 5, wherein the spatial upsampling model comprises: an input convolutional layer with a kernel size of 3, a stride of 1, an input number of channels of 3, and an output number of channels of 64; an output convolutional layer with a kernel size of 3, a stride of 1, an input number of channels of 64, and an output number of channels of 3; and three BRes disposed between the input convolutional layer and the output convolutional layer.

[0126] 7. The method of clause 6, wherein the BRes comprises: a first convolutional layer with a kernel size of 1, a stride of 1, an input number of channels of 64, and an output number of channels of 32; a second convolutional layer with a kernel size of 3, a stride of 1, an input number of channels of 32, and an output number of channels of 32; and a third convolutional layer with a kernel size of 1, a stride of 1, an input number of channels of 32, and an output number of channels of 64.

[0127] 8. The method of any one of clauses 1 to 7, wherein the parameters of the spatial upsampling model are quantized to a predetermined format.

[0128] 9. The method of clause 8, wherein the predetermined format is one of float 16, int 8, or binary.

[0129] 10. A method for encoding a video sequence into a bitstream, the method comprising: receiving a video sequence; encoding one or more images of the video sequence; and generating a bitstream associated with the one or more images, wherein the encoding the one or more images of the video sequence comprises: performing spatial down-sampling on the one or more images by a spatial down-sampling model to obtain one or more down-sampled images, respectively; compressing the one or more down-sampled images; and generating parameters of a spatial up-sampling model for decoding the one or more compressed images, wherein a total number of coded bits of the parameters of the spatial up-sampling model is less than a threshold predetermined based on a desired quality of reconstructed images of the one or more images.

[0130] 11. The method of clause 10, wherein the generating the bitstream associated with the encoded one or more images comprises: sending the parameters of the spatial up-sampling model into the bitstream, the parameters comprising weights and biases of layers of the spatial up-sampling model.

[0131] 12. The method of clause 10 or 11, wherein the spatial up-sampling model comprises: 11 convolutional layers.

[0132] 13. The method of clause 12, wherein the spatial up-sampling model comprises: an input convolutional layer with a kernel size of 3, a stride of 1, an input number of channels of 3, and an output number of channels of 64; nine intermediate convolutional layers with a kernel size of 3, a stride of 1, an input number of channels of 64, and an output number of channels of 64; and an output convolutional layer with a kernel size of 3, a stride of 1, an input number of channels of 64, and an output number of channels of 3.

[0133] 14. The method of clause 10 or 11, wherein the spatial up-sampling model comprises: one or more bottleneck residual blocks (BRes).

[0134] 15. The method of clause 14, wherein the spatial up-sampling model comprises: an input convolutional layer with a kernel size of 3, a stride of 1, an input number of channels of 3, and an output number of channels of 64; an output convolutional layer with a kernel size of 3, a stride of 1, an input number of channels of 64, and an output number of channels of 3; and three BRes arranged between the input convolutional layer and the output convolutional layer.

[0135] 16. The method of clause 15, wherein the BRes comprises: a first convolutional layer with a kernel size of 1, a stride of 1, an input channel number of 64, and an output channel number of 32; a second convolutional layer with a kernel size of 3, a stride of 1, an input channel number of 32, and an output channel number of 32; and a third convolutional layer with a kernel size of 1, a stride of 1, an input channel number of 32, and an output channel number of 64.

[0136] 17. The method of any one of clauses 10 to 16, wherein the parameters of the spatial upsampling model are quantized into a predetermined format.

[0137] 18. The method of clause 17, wherein the predetermined format is one of float16, int8, or binary.

[0138] 19. An apparatus for decoding a bitstream to output one or more images of a video stream, the apparatus comprising: a receiving module configured to receive a bitstream; and a decoding module configured to decode one or more images using encoded information of the bitstream, wherein the decoding module is configured to: decompress one or more compressed images included in the bitstream; and perform spatial upsampling on the one or more decompressed images by a spatial upsampling model to obtain one or more reconstructed images, respectively, wherein a total number of encoded bits of parameters of the spatial upsampling model is less than a threshold value predetermined based on a desired quality of the reconstructed images.

[0139] 20. The apparatus of clause 19, wherein the decoding module is configured to: extract parameters from the bitstream, the parameters comprising weights and biases of layers of the spatial upsampling model.

[0140] 21. The apparatus of clause 19 or 20, wherein the spatial upsampling model comprises 11 convolutional layers.

[0141] 22. The apparatus of clause 21, wherein the spatial upsampling model comprises: an input convolutional layer with a kernel size of 3, a stride of 1, an input channel number of 3, and an output channel number of 64; nine intermediate convolutional layers with a kernel size of 3, a stride of 1, an input number of channels of 64, and an output number of channels of 64; and an output convolutional layer with a kernel size of 3, a stride of 1, an input number of channels of 64, and an output number of channels of 3.

[0142] 23. The apparatus of clause 19 or 20, wherein the spatial upsampling model comprises one or more bottleneck residual blocks (BRes).

[0143] 24. The apparatus of clause 23, wherein the spatial upsampling model comprises: an input convolutional layer with a kernel size of 3, a stride of 1, an input number of channels of 3, and an output number of channels of 64; an output convolutional layer with a kernel size of 3, a stride of 1, an input number of channels of 64, and an output number of channels of 3; and three BRes disposed between the input convolutional layer and the output convolutional layer.

[0144] 25. The apparatus of clause 24, wherein the BRes comprises: a first convolutional layer with a kernel size of 1, a stride of 1, an input number of channels of 64, and an output number of channels of 32; a second convolutional layer with a kernel size of 3, a stride of 1, an input number of channels of 32, and an output number of channels of 32; and a third convolutional layer with a kernel size of 1, a stride of 1, an input number of channels of 32, and an output number of channels of 64.

[0145] 26. The apparatus of any of clauses 19-25, wherein the parameters of the spatial upsampling model are quantized into a predetermined format.

[0146] 27. The apparatus of clause 26, wherein the predetermined format is one of float 16, int 8, or binary.

[0147] 28. An apparatus for encoding a video sequence into a bitstream, the apparatus comprising: a receiving module configured to receive a video sequence; an encoding module configured to encode one or more pictures of the video sequence; and a generating module configured to generate a bitstream associated with the one or more pictures, wherein the encoding module is configured to: performing spatial down-sampling on the one or more images by a spatial down-sampling model to obtain one or more down-sampled images, respectively; compressing the one or more down-sampled images; and generating parameters of a spatial up-sampling model for decoding the one or more compressed images, wherein a total number of encoded bits of the parameters of the spatial up-sampling model is less than a threshold predetermined based on a desired quality of a reconstructed image of the one or more images.

[0148] 29. The apparatus of clause 28, wherein the generating module is configured to: send the parameters of the spatial up-sampling model into the bitstream, the parameters comprising weights and biases of layers of the spatial up-sampling model.

[0149] 30. The apparatus of clause 28 or 29, wherein the spatial up-sampling model comprises 11 convolutional layers.

[0150] 31. The apparatus of clause 30, wherein the spatial up-sampling model comprises: an input convolutional layer having a kernel size of 3, a stride of 1, an input number of channels of 3, and an output number of channels of 64; nine intermediate convolutional layers having a kernel size of 3, a stride of 1, an input number of channels of 64, and an output number of channels of 64; and an output convolutional layer having a kernel size of 3, a stride of 1, an input number of channels of 64, and an output number of channels of 3.

[0151] 32. The apparatus of clause 28 or 29, wherein the spatial up-sampling model comprises one or more bottleneck residual blocks (BRes).

[0152] 33. The apparatus of clause 32, wherein the spatial up-sampling model comprises: an input convolutional layer having a kernel size of 3, a stride of 1, an input number of channels of 3, and an output number of channels of 64; an output convolutional layer having a kernel size of 3, a stride of 1, an input number of channels of 64, and an output number of channels of 3; and three BRes disposed between the input convolutional layer and the output convolutional layer.

[0153] 34. The apparatus of clause 33, wherein the BRes comprises: a first convolutional layer having a kernel size of 1, a stride of 1, an input number of channels of 64, and an output number of channels of 32; a second convolutional layer with a kernel size of 3, a stride of 1, an input channel number of 32, and an output channel number of 32; and a third convolutional layer with a kernel size of 1, a stride of 1, an input channel number of 32, and an output channel number of 64.

[0154] 35. The apparatus of any of clauses 28-34, wherein the parameters of the spatial upsampling model are quantized to a predetermined format.

[0155] 36. The apparatus of clause 35, wherein the predetermined format is one of float 16, int 8, or binary.

[0156] 37. An electronic device, the electronic device comprising: one or more processors, and a computer-readable storage medium communicatively coupled to the one or more processors, wherein the computer-readable storage medium stores computer-readable instructions executable by the one or more processors, the computer-readable instructions, when executed by the one or more processors, perform the method of any of clauses 1-18.

[0157] 38. A non-transitory computer-readable storage medium having stored thereon a video bitstream, wherein the bitstream, when decoded by a decoder, causes the decoder to perform the method of any of clauses 1-9.

[0158] 39. A non-transitory computer-readable storage medium having stored thereon a video bitstream, wherein the bitstream, when encoded by an encoder, causes the encoder to perform the method of any of clauses 10-18.

[0159] 40. A computer program product, the computer program product comprising: computer program instructions, wherein the computer program instructions enable a computer to perform the method of any of clauses 1-18.

[0160] 41. A computer program, wherein the computer program enables a computer to perform the method of any of clauses 1-18.

[0161] It should be noted that the relational terms herein, such as first, second, and the like, are used solely to distinguish one from another entity or action without necessarily implying any actual relationship or order between such entities or actions. Moreover, the words "comprising," "having," "containing," and "including," and other similar forms, are intended to be equivalent in meaning and be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items, or meant to be limited to only the listed item or items.

[0162] As used herein, unless specifically stated otherwise, the term "or" encompasses all possible combinations. For example, if a database is stated as including A or B, then unless specifically stated otherwise, the database can include A; or B; or both A and B. As a second example, if a database is stated as including A, B, or C, then unless specifically stated otherwise, the database can include A; or B; or C; or A and B; or A and C; or B and C; or A and B and C.

[0163] It should be understood that the above-described embodiments can be implemented by hardware, or software (program code), or a combination of hardware and software. If implemented by software, it can be stored in the above-described computer-readable medium. The software, when executed by a processor, can perform the disclosed method. The computing units and other functional units described in the present disclosure can be implemented by hardware, or software, or a combination of hardware and software. Those of ordinary skill in the art should also understand that a plurality of the above-described modules / units can be combined into one module / unit, and each of the above-described modules / units can be further divided into a plurality of sub-modules / sub-units.

[0164] In the above specification, the embodiments have been described with reference to numerous specific details that can be varied in accordance with implementation. Certain adjustments and modifications can be made to the described embodiments. Other embodiments will be apparent to those of ordinary skill in the art from consideration of the specification and practice of the present disclosure disclosed herein. The specification and examples are to be considered exemplary only, with the true scope and spirit of the disclosure indicated by the claims that follow. The order of steps shown in the figures is also considered to be for illustrative purposes only, and is not intended to be limited to any particular order of steps. Thus, those of ordinary skill in the art will appreciate that the steps can be performed in different orders while implementing the same method.

[0165] In the drawings and specification, there have been disclosed exemplary embodiments. However, many variations and modifications can be made to these embodiments. Consequently, it is intended that the scope of the application be limited only by the broadest interpretation of the appended claims to be accorded under 35 U.S.C. § 112.

Claims

1. A method for decoding a bitstream to output one or more images of a video stream, the method comprising: Receive bit stream; as well as Decode one or more images using the encoded information of the bitstream. Decoding the one or more images using the encoded information of the bitstream includes: Decompress one or more compressed images included in the bitstream; and Spatial upsampling is performed on the one or more decompressed images using a spatial upsampling model to obtain one or more reconstructed images respectively. Wherein, the total number of encoded bits of the parameters of the spatial upsampling model is less than a threshold predetermined based on the expected quality of the reconstructed image.

2. The method according to claim 1, wherein, Decoding the one or more images using the encoded information of the bitstream further includes: Parameters are extracted from the bitstream, including the weights and biases of each layer of the spatial upsampling model.

3. The method according to claim 1 or 2, wherein, The spatial upsampling model consists of 11 convolutional layers.

4. The method according to claim 3, wherein, The spatial upsampling model includes: The input convolutional layer has a kernel size of 3, a stride of 1, 3 input channels, and 64 output channels. Nine intermediate convolutional layers, each with a kernel size of 3, a stride of 1, 64 input channels, and 64 output channels; and The output convolutional layer has a kernel size of 3, a stride of 1, 64 input channels, and 3 output channels.

5. The method according to claim 1 or 2, wherein, The spatial upsampling model includes one or more bottleneck residual blocks (BRes).

6. The method according to claim 5, wherein, The spatial upsampling model includes: The input convolutional layer has a kernel size of 3, a stride of 1, 3 input channels, and 64 output channels. The output convolutional layer has a kernel size of 3, a stride of 1, 64 input channels, and 3 output channels; and Three BRes are positioned between the input convolutional layer and the output convolutional layer.

7. The method according to claim 6, wherein, The BRes include: The first convolutional layer has a kernel size of 1, a stride of 1, 64 input channels, and 32 output channels. The second convolutional layer has a kernel size of 3, a stride of 1, 32 input channels, and 32 output channels; and The third convolutional layer has a kernel size of 1, a stride of 1, 32 input channels, and 64 output channels.

8. The method according to any one of claims 1 to 7, wherein, The parameters of the spatial upsampling model are quantized into a predetermined format.

9. The method according to claim 8, wherein, The predetermined format is one of float16, int8, or binary.

10. A method for encoding a video sequence into a bitstream, the method comprising: Receive video sequences; Encode one or more images from the video sequence; as well as Generate a bitstream associated with the one or more images. Encoding the one or more images in the video sequence includes: Spatial downsampling is performed on the one or more images using a spatial downsampling model to obtain one or more downsampled images respectively; Compress the one or more downsampled images; and Generate parameters for a spatial upsampling model used to decode the one or more compressed images. Wherein, the total number of encoded bits of the parameters of the spatial upsampling model is less than a predetermined threshold for the expected quality of the reconstructed image based on the one or more images.

11. The method according to claim 10, wherein, Generating the bitstream associated with the one or more encoded images includes: The parameters of the spatial upsampling model are sent to the bitstream, and the parameters include the weights and biases of each layer of the spatial upsampling model.

12. The method according to claim 10 or 11, wherein, The spatial upsampling model consists of 11 convolutional layers.

13. The method according to claim 12, wherein, The spatial upsampling model includes: The input convolutional layer has a kernel size of 3, a stride of 1, 3 input channels, and 64 output channels. Nine intermediate convolutional layers, each with a kernel size of 3, a stride of 1, 64 input channels, and 64 output channels; and The output convolutional layer has a kernel size of 3, a stride of 1, 64 input channels, and 3 output channels.

14. The method according to claim 10 or 11, wherein, The spatial upsampling model includes one or more bottleneck residual blocks (BRes).

15. The method according to claim 14, wherein, The spatial upsampling model includes: The input convolutional layer has a kernel size of 3, a stride of 1, 3 input channels, and 64 output channels. The output convolutional layer has a kernel size of 3, a stride of 1, 64 input channels, and 3 output channels; and Three BRes are positioned between the input convolutional layer and the output convolutional layer.

16. The method according to claim 15, wherein, The BRes include: The first convolutional layer has a kernel size of 1, a stride of 1, 64 input channels, and 32 output channels. The second convolutional layer has a kernel size of 3, a stride of 1, 32 input channels, and 32 output channels; and The third convolutional layer has a kernel size of 1, a stride of 1, 32 input channels, and 64 output channels.

17. The method according to any one of claims 10 to 16, wherein, The parameters of the spatial upsampling model are quantized into a predetermined format.

18. The method according to claim 17, wherein, The predetermined format is one of float16, int8, or binary.

19. An apparatus for decoding a bitstream to output one or more images of a video stream, the apparatus comprising: A receiving module configured to receive bit streams; as well as A decoding module configured to decode one or more images using the encoded information of the bitstream. The decoding module is configured as follows: Decompress one or more compressed images included in the bitstream; and Spatial upsampling is performed on the one or more decompressed images using a spatial upsampling model to obtain one or more reconstructed images respectively. Wherein, the total number of encoded bits of the parameters of the spatial upsampling model is less than a threshold predetermined based on the expected quality of the reconstructed image.

20. The apparatus according to claim 19, wherein, The decoding module is configured to: Extract parameters from the bitstream that include the weights and biases of the corresponding layers of the spatial upsampling model.

21. The apparatus according to claim 19 or 20, wherein, The spatial upsampling model consists of 11 convolutional layers.

22. The apparatus according to claim 21, wherein, The spatial upsampling model includes: The input convolutional layer has a kernel size of 3, a stride of 1, 3 input channels, and 64 output channels. Nine intermediate convolutional layers with a kernel size of 3, a stride of 1, and 64 input and 64 output channels; and The output convolutional layer has a kernel size of 3, a stride of 1, 64 input channels, and 3 output channels.

23. The apparatus according to claim 19 or 20, wherein, The spatial upsampling model includes one or more bottleneck residual blocks (BRes).

24. The apparatus according to claim 23, wherein, The spatial upsampling model includes: The input convolutional layer has a kernel size of 3, a stride of 1, 3 input channels, and 64 output channels. The output convolutional layer has a kernel size of 3, a stride of 1, 64 input channels, and 3 output channels; and Three BRes are arranged between the input convolutional layer and the output convolutional layer.

25. The apparatus according to claim 24, wherein, The BRes include: The first convolutional layer has a kernel size of 1, a stride of 1, 64 input channels, and 32 output channels. The second convolutional layer has a kernel size of 3, a stride of 1, 32 input channels, and 32 output channels; and The third convolutional layer has a kernel size of 1, a stride of 1, 32 input channels, and 64 output channels.

26. The apparatus according to any one of claims 19 to 25, wherein, The parameters of the spatial upsampling model are quantized into a predetermined format.

27. The apparatus according to claim 26, wherein, The predetermined format is one of float16, int8, or binary.

28. An apparatus for encoding a video sequence into a bitstream, the apparatus comprising: A receiving module configured to receive video sequences; An encoding module configured to encode one or more images of the video sequence; as well as A generation module, configured to generate a bitstream associated with the one or more images. The encoding module is configured as follows: Spatial downsampling is performed on the one or more images using a spatial downsampling model to obtain one or more downsampled images respectively; Compress the one or more downsampled images; and Parameters are generated for a spatial upsampling model used to decode the one or more compressed images. Wherein, the total length of the encoded bits of the parameters of the spatial upsampling model is less than a predetermined threshold for the expected quality of the reconstructed image based on the one or more images.

29. The apparatus according to claim 28, wherein, The generation module is configured to: The parameters of the spatial upsampling model are sent to the bitstream, the parameters including the weights and biases of the corresponding layers of the spatial upsampling model.

30. The apparatus according to claim 28 or 29, wherein, The spatial upsampling model consists of 11 convolutional layers.

31. The apparatus according to claim 30, wherein, The spatial upsampling model includes: The input convolutional layer has a kernel size of 3, a stride of 1, 3 input channels, and 64 output channels. Nine intermediate convolutional layers with a kernel size of 3, a stride of 1, and 64 input and 64 output channels; and The output convolutional layer has a kernel size of 3, a stride of 1, 64 input channels, and 3 output channels.

32. The apparatus according to claim 28 or 29, wherein, The spatial upsampling model includes one or more bottleneck residual blocks (BRes).

33. The apparatus according to claim 32, wherein, The spatial upsampling model includes: The input convolutional layer has a kernel size of 3, a stride of 1, 3 input channels, and 64 output channels. The output convolutional layer has a kernel size of 3, a stride of 1, 64 input channels, and 3 output channels; and Three BRes are arranged between the input convolutional layer and the output convolutional layer.

34. The apparatus according to claim 33, wherein, The BRes include: The first convolutional layer has a kernel size of 1, a stride of 1, 64 input channels, and 32 output channels. The second convolutional layer has a kernel size of 3, a stride of 1, 32 input channels, and 32 output channels; and The third convolutional layer has a kernel size of 1, a stride of 1, 32 input channels, and 64 output channels.

35. The apparatus according to any one of claims 28 to 34, wherein, The parameters of the spatial upsampling model are quantized into a predetermined format.

36. The apparatus according to claim 35, wherein, The predetermined format is one of float16, int8, or binary.

37. An electronic device, the electronic device comprising: One or more processors, and A computer-readable storage medium communicatively coupled to the one or more processors. The computer-readable storage medium stores computer-readable instructions executable by the one or more processors, which, when executed by the one or more processors, perform the method according to any one of claims 1 to 18.

38. A non-transitory computer-readable storage medium for storing a bitstream of video, wherein, The bitstream causes the decoder to perform the method according to any one of claims 1 to 9 when it is decoded by the decoder.

39. A non-transitory computer-readable storage medium for storing a bitstream of video, wherein, The bitstream, when encoded by the encoder, causes the encoder to perform the method according to any one of claims 10 to 18.

40. A computer program product, the computer program product comprising: Computer program instructions, wherein the computer program instructions enable a computer to perform the method according to any one of claims 1 to 18.

41. A computer program, wherein, The computer program enables the computer to perform the method according to any one of claims 1 to 18.

Images