Analysis-based adaptive time resampling decision algorithm for machine vision

Abstract

A method of decoding a bitstream associated with a video sequence is provided. The method comprises: decompressing a bitstream associated with a video sequence; reconstructing one or more frames of the video sequence based on the decompressed bitstream; determining a sequence level average intersection-to-parallel ratio (MIOU) based on the one or more reconstructed frames; obtaining a decision index according to the sequence-level MIOU to determine a resampling ratio and whether to perform time resampling; and resampling the one or more reconstructed frames based on the decision indicator.

Description

[0001] Cross-references to related applications This application claims priority to U.S. Provisional Application 63 / 709,583, filed October 21, 2024, entitled "Analysis-Based Adaptive Temporal Resampling Decision Algorithm for Machine Vision," which is incorporated herein by reference in its entirety. This application also claims priority to U.S. Patent Application No. 19 / 328,707, filed September 15, 2025. Technical Field

[0002] This disclosure generally relates to video processing, and more specifically, to an analytics-based adaptive temporal resampling decision algorithm for machine vision. Background Technology

[0003] Video consists of a set of still images (or "frames") that capture visual information. To reduce storage memory and transmission bandwidth, video can be compressed before storage or transmission and decompressed before display. The compression process is usually called encoding, while the decompression process is usually called decoding. There are many video coding formats that use standardized video coding techniques, the most common being based on prediction, transform, quantization, entropy coding, and in-loop filtering. Standardization organizations have developed video coding standards, such as the High Efficiency Video Coding (HEVC / H.265) standard, the Universal Video Coding (VVC / H.266) standard, and the AVS standard, which specify particular video coding formats. As more and more advanced video coding techniques are incorporated into video standards, the coding efficiency of new video coding standards is also increasing. Summary of the Invention

[0004] Embodiments of this disclosure provide an analytics-based adaptive temporal resampling decision algorithm for machine vision.

[0005] In some embodiments, a method is provided for decoding a bitstream associated with a video sequence. The method includes: decompressing the bitstream associated with the video sequence; reconstructing one or more frames of the video sequence based on the decompressed bitstream; determining a sequence-level average intersection-over-union (MIOU) based on the one or more reconstructed frames; obtaining a decision metric based on the sequence-level MIOU to determine a resampling ratio and whether to perform temporal resampling; and resampling the one or more reconstructed frames based on the decision metric.

[0006] In some embodiments, a method for encoding a video sequence into a bitstream is provided. The method includes: determining a sequence-level average intersection-to-union ratio (MIOU) based on one or more frames of the video sequence; obtaining a decision metric based on the sequence-level MIOU to determine a resampling ratio and whether to perform temporal resampling; resampling the one or more frames based on the decision metric; and encoding the one or more resampled frames to generate a bitstream.

[0007] In some embodiments, a method for transmitting a bitstream is provided. The method includes: receiving a video sequence comprising one or more frames; encoding the video sequence by: determining a sequence-level average intersection-to-union ratio (MIOU) based on the one or more frames of the video sequence; obtaining a decision metric based on the sequence-level MIOU to determine a resampling ratio and whether to perform temporal resampling; resampling the one or more frames based on the decision metric; encoding the one or more resampled frames; and transmitting a bitstream generated based on the encoding. Attached Figure Description

[0008] Embodiments and aspects of this disclosure are illustrated in the following detailed description and accompanying drawings. Various features shown in the figures are not drawn to scale.

[0009] Figure 1 The structure of an exemplary video sequence according to some embodiments of this disclosure is shown.

[0010] Figure 2 A schematic diagram illustrates an example framework for video compression in a video encoding system according to some embodiments of the present disclosure.

[0011] Figure 3 This is a schematic diagram illustrating an example analysis-based time resampling compression framework according to some embodiments of the present disclosure.

[0012] Figure 4 This is a block diagram of an example apparatus for encoding or decoding image data according to some embodiments of the present disclosure.

[0013] Figure 5 This is a flowchart of an example method for decoding a bitstream associated with a video sequence according to some embodiments of the present disclosure.

[0014] Figure 6 This is a flowchart of an example method for encoding a video sequence into a bitstream according to some embodiments of the present disclosure. Detailed Implementation

[0015] Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, in which, unless otherwise stated, the same numerals in different figures represent the same or similar elements. The embodiments set forth in the following description of the exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with aspects of the invention as described in the appended claims. Specific aspects of this disclosure are described in more detail below. In the event of any conflict with terms and / or definitions incorporated by reference, the terms and definitions provided herein shall prevail.

[0016] Video compression is a key technology in the digital media field, indispensable for the practical and efficient application of video in today's data-driven, interconnected world. The importance of video coding can be summarized in several aspects, including bandwidth efficiency, reduced storage requirements, content accessibility, enabling technologies, and quality maintenance. Specifically, regarding bandwidth efficiency, uncompressed video requires significant bandwidth for transmission due to its high data rate. Video coding significantly reduces video file size, enabling high-quality streaming even on networks with limited bandwidth, such as mobile data connections. Regarding reduced storage requirements, the storage requirements for video content are significantly reduced as file sizes decrease through video coding. This makes it possible to store large amounts of video on servers, personal devices, and in the cloud without exhausting storage capacity. Regarding content accessibility, through efficient compression, video coding enables the widespread distribution and accessibility of video content. Users can quickly and reliably download or stream video regardless of geographical location or network quality. Regarding enabling technologies, advancements in video coding form the basis of modern technologies such as video conferencing, streaming services, digital television, and social media platforms, where video is the primary form of content. To maintain quality, efficient video coding techniques strive to preserve the highest possible video quality at the lowest possible bitrate. This minimizes data usage while preserving the viewing experience, which is especially important for users with data limits or slow network connections. Driven by this, video compression has continuously evolved in recent decades, attracting widespread attention from academia and industry.

[0017] The development of video coding standards is crucial to the advancement of digital video, enabling efficient storage, transmission, and compatibility across various devices and services. Over the years, several major standards have emerged, each building upon the success of its predecessors (e.g., Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), and Universal Video Coding (VVC)) while learning from their limitations. AVC / H.264 has been one of the most widely adopted video coding standards since 2003. Developed jointly by the ITU-T Video Coding Experts Group and the ISO / IEC Moving Picture Experts Group, AVC improves upon previous standards by using more sophisticated prediction, transform, and entropy coding tools. It offers significantly better compression efficiency, meaning it can deliver good video quality at approximately half the bitrate of its predecessors. AVC's flexibility makes it suitable for a wide range of applications, from streaming video and Blu-ray discs to satellite television and video conferencing. With the continued growth in demand for higher video resolution and quality, a more efficient coding standard is needed. The HEVC / H.265 standard, finalized in 2013, met this requirement by achieving a significant improvement in compression efficiency compared to AVC. HEVC provides the same visual quality as AVC but at approximately 50% lower bitrate, making it ideal for 4K and 8K video resolutions. This improvement is achieved through enhancements to various coding tools, including more advanced prediction algorithms, larger transform blocks, improved entropy coding, and better motion vector prediction. VVC, the latest evolution in the video coding standards series, was finalized in 2020. Its development focused on the future of video technology, aiming to maintain the same level of video quality while further reducing the bitrate by 50% compared to HEVC. VVC introduces new tools and technologies such as improved intra-frame prediction, enhanced motion modeling, and stronger parallel processing capabilities, all relevant to next-generation high-resolution video content, including VR and 360-degree video. VVC is designed for high versatility, suitable not only for traditional broadcasting and streaming but also for emerging applications that rely on high-fidelity video content.

[0018] Driven by the development of deep learning, the compression performance of deep learning-based video compression has been greatly improved in recent years. Deep learning-based video compression represents a paradigm shift from traditional codec design—which typically employs manually designed algorithms and heuristic rules—to new methods that leverage the capabilities of neural networks to learn data-driven and compression-oriented optimized representations directly from massive image or video datasets.

[0019] Furthermore, according to Sandvine's 2022 Global Internet Phenomena Report, video accounts for 65% of all internet traffic, thanks to the rich information contained in video data. Meanwhile, due to the large volume of video data, video compression is essential for real-time transmission in various applications such as live streaming and video conferencing. With the development of computer vision, machine vision is replacing human vision in several new application scenarios, such as intelligent transportation and smart cities, for analyzing and understanding video data. Driven by this, efficient video compression (VCM) technology for machine vision has become an urgent need.

[0020] Based on the development of video compression, the background content in this section is arranged as follows: an introduction to traditional video codecs, the development of deep learning-based video compression, and video compression for machine vision.

[0021] Traditional hybrid video codecs represent the infrastructure of most video compression standards used today. These codecs combine spatial and temporal compression techniques to significantly reduce the size of video data while maintaining a balance between compression efficiency and visual quality. The term "hybrid" refers to the simultaneous use of inter-frame (temporal) and intra-frame (spatial) coding methods within the codec. Hybrid video codecs have become the core technology of advanced video compression standards such as H.264 / AVC (for Blu-ray discs, streaming media, and broadcasting), HEVC / H.265 (for high-resolution video), and VVC / H.266 (for 2K and higher resolution video). Each generation of codecs improves the efficiency of these core components, allowing for higher quality video at lower bitrates. However, the fundamental principles of hybrid coding remain at the heart of how these codecs operate.

[0022] Figure 1 The structure of an example video sequence according to some embodiments of this disclosure is shown. Video sequence 100 may be live video or video that has already been captured and archived. Video sequence 100 may be real-life video, computer-generated video (e.g., computer game video), or a combination thereof (e.g., real-life video with augmented reality effects). Video sequence 100 may originate from a video capture device (e.g., a camera), a video archive containing previously captured video (e.g., a video file stored on a storage device), or a video feed interface (e.g., a video broadcast transceiver) that receives video from a video content provider. Figure 1 As shown, video sequence 100 may include a series of images arranged temporally along a time axis, including images 102, 104, 106, and 108. Images 102-106 are consecutive, and there are more images between images 106 and 108.

[0023] When video is being compressed or decompressed, useful information about the image being encoded (referred to as the "current image") includes changes relative to a reference image (e.g., a previously encoded and reconstructed image). These changes can include variations in pixel position, brightness, or color. For example, changes in the position of a set of pixels can reflect the motion of a target represented by those pixels between two images (e.g., the reference image and the current image).

[0024] For example, such as Figure 1 As shown, image 102 is an I-image, using itself as a reference image. Image 104 is a P-image, using image 102 as its reference image, as indicated by the arrow. Image 106 is a B-image, using images 104 and 108 as its reference images, as indicated by the arrow. In some embodiments, the reference image of an image may not immediately precede or follow that image. For example, the reference image of image 104 may be an image preceding image 102, that is, an image not immediately preceding image 104. Figure 1 The reference images 102-106 shown are merely examples and are not intended to limit this disclosure.

[0025] Due to computational complexity, in some embodiments, a video codec may divide an image into multiple basic segments and encode or decode the image segment by segment. That is, the video codec does not encode or decode the entire image at once. Such basic segments are referred to herein as basic processing units (“BPUs”). For example, Figure 1 An example structure 110 of images from video sequence 100 (e.g., any one of images 102-108) is also shown. For example, structure 110 can be used to segment image 108. Figure 1 As shown, image 108 is divided into 4×4 basic processing units. In some embodiments, the basic processing unit may be called a "coding tree unit" ("CTU") in some video coding standards (e.g., AVS3, H.265 / HEVC, or H.266 / VVC), or a "macroblock" in some video coding standards (e.g., MPEG family, H.261, H.263, or H.264 / AVC). In AVS3 or VVC, the coding tree unit (CTU) can be the largest block unit and can be as large as 128×128 luminance samples (plus the corresponding chroma samples according to the chroma format).

[0026] Figure 1The basic processing units in the image are for illustrative purposes only. These basic processing units can have variable sizes in the image, such as 128×128, 64×64, 32×32, 16×16, 4×8, 16×32, or any arbitrary shape and size of pixels. The size and shape of the basic processing units can be selected for the image based on a balance between coding efficiency and the level of detail to be maintained within the basic processing units.

[0027] The basic processing unit can be a logical unit that may include a set of different types of video data stored in computer memory (e.g., in a video frame buffer). For example, a basic processing unit for a color image may include a luminance component (Y) representing non-color luminance information, one or more chrominance components (e.g., Cb and Cr) representing color information, and associated syntax elements, wherein the size of the luminance and chrominance components may be the same as that of the basic processing unit. In some video coding standards, the luminance and chrominance components may be referred to as "code tree blocks" ("CTBs"). Operations performed on a basic processing unit may be repeated on its luminance and chrominance components.

[0028] Video coding involves multiple operational stages, and the size of the basic processing unit may still be too large to process. Therefore, it can be further divided into multiple segments referred to herein as "basic processing subunits." For example, in the mode decision stage, the encoder can divide the basic processing unit into multiple basic processing subunits and determine the prediction type for each individual basic processing subunit. Figure 1 As shown, the basic processing unit 112 in structure 110 is further divided into 4×4 basic processing subunits. For example, the coding tree unit CTU can be further divided into coding units (CUs) using a quadtree, binary tree, or extended binary tree. Figure 1The basic processing subunits described herein are for illustrative purposes only. Different basic processing units of the same image may be divided into basic processing subunits using different schemes. These basic processing subunits may be referred to as “coding units” (“CUs”) in some video coding standards (e.g., AVS3, H.265 / HEVC, or H.266 / VVC), or as “blocks” in some video coding standards (e.g., MPEG series, H.261, H.263, or H.264 / AVC). The size of a basic processing subunit may be equal to or smaller than the size of a basic processing unit. Similar to the basic processing unit, a basic processing subunit is also a logical unit, which may include a set of different types of video data (e.g., Y, Cb, Cr, and associated syntax elements) stored in computer memory (e.g., in a video frame buffer). Operations performed on a basic processing subunit may be repeated on its luminance and chrominance components. Depending on processing needs, this partitioning is performed at deeper levels, and different schemes may be used to segment the basic processing units at different stages. At the leaf nodes of the segmentation structure, coding information such as the coding mode (intra-frame prediction mode or inter-frame prediction mode), motion information (reference index, motion vector (MV), etc.) required for the coding mode, and quantization residual coefficients are transmitted.

[0029] In some cases, the basic processing subunit may still be too large to be processed in certain operational stages of video coding, such as the prediction or transform stage. Therefore, the encoder may further segment the basic processing subunit into smaller segments (e.g., called "prediction blocks" or "PBs"), at which prediction operations can be performed. Similarly, the encoder may further segment the basic processing subunit into smaller segments (e.g., called "transform blocks" or "TBs"), at which transform operations can be performed. The segmentation scheme for the same basic processing subunit can differ between the prediction and transform stages. For example, the prediction blocks (PBs) and transform blocks (TBs) of the same CU can have different sizes and numbers.

[0030] Figure 2 A schematic diagram of an example framework 200 for video compression in a video encoding system according to some embodiments of the present disclosure is shown. Typically, a video compression encoder generates a bitstream based on the input current frame. Furthermore, the decoder reconstructs the video frame based on the received bitstream. Figure 2 The framework 200 follows a prediction-transformation architecture.

[0031] In some embodiments, the hybrid video codec includes the following key components that work together in a sequential pipeline manner, such as Figure 2 As shown.

[0032] The input video is processed block by block. Specifically, the input frames (For example, the current frame 202) is divided into sets of blocks of the same size (e.g., 8 × 8), such as square regions. The encoding process of the video compression algorithm at the encoding end will be discussed below.

[0033] The motion estimation and compensation operations are performed during the encoding process. The input frame Processed by block-based motion estimation module 210, which is configured to estimate the current frame. Compared with the previously reconstructed frame Movement between them. Based on the input frame. and the previously reconstructed frame The block-based motion estimation module 210 outputs motion vectors for each block. Motion estimation is a crucial part of inter-frame compression. The codec estimates the motion occurring between frames and encodes this motion using motion vectors. During decoding, these vectors are used to shift pixel blocks in a reference frame (I or P) to predict the current frame, thus reducing the need to encode the entire frame from scratch.

[0034] Then, inter-frame (temporal) compression can be performed during the encoding process. The corresponding motion vectors are processed by the inter-frame prediction module 2044. In order to achieve this by using motion vectors defined in the motion estimation module 210 The previously reconstructed frame The corresponding pixels in the image are copied to the current frame to obtain the prediction frame. Therefore, the original frame is obtained. With the predicted frame residuals between ,for In some embodiments, the motion compensation prediction performed above is also referred to as "inter-frame prediction," "inter-image prediction," or "temporal prediction." The inter-frame coding utilizes redundancy between consecutive frames. In some embodiments, intra-frame coding is used to encode only the first frame in the sequence, referred to as an I-frame or keyframe. Subsequent frames, referred to as P-frames (predictive) or B-frames (bidirectional prediction frames), are encoded only by using differences from previous or future frames. As described above, the motion estimation is used to find matching blocks in a reference frame and generate motion vectors to describe how these blocks move from one frame to the next. The differences, or residuals, and the motion vectors can then be encoded.

[0035] Intra-frame (spatial) compression is also performed during the encoding process. The first step in the encoding process typically involves intra-frame coding, which compresses each frame into an independent image. For example, in the intra-frame prediction stage 2042, the encoder may perform intra-frame prediction. It uses the spatial redundancy within the frame to reduce the amount of data. Techniques such as transform coding (e.g., Discrete Cosine Transform, DCT) and quantization are used to transform the spatial pixel values ​​to the frequency domain, where the spatial pixel values ​​can be encoded more efficiently. The result is a set of coefficients, which can be further compressed using entropy coding techniques such as Huffman or arithmetic coding. In addition to the above, some intra-frame compression techniques also employ predictive coding to reduce redundancy. Predictive coding works by estimating pixel values ​​based on neighboring pixels and encoding only the difference between the actual and predicted values. This method is particularly effective for reducing spatial redundancy in smooth or slowly changing image regions.

[0036] like Figure 2 As shown, in generating the residual Then, the encoder can convert the residual The feed is sent to the transformation stage 212 and the quantization stage 214 to generate the quantization result. In some embodiments, a linear transformation (e.g., DCT) may be used prior to the quantization to achieve better compression performance. Different transformation algorithms may use different basic modes. Various transformation algorithms may be used in the transformation stage 212, for example, discrete cosine transform, discrete sine transform, etc. The transformation in the transformation stage 212 is reversible. That is, the encoder can recover the residual by performing the inverse operation (referred to as the "inverse transform") of the transformation in the inverse transformation stage 220. For video coding standards, the encoder and corresponding decoder can use the same transform algorithm (and thus the same base mode). Therefore, the encoder can simply record the transform coefficients, and the decoder can reconstruct the residual from these coefficients. Without needing to receive the basic mode from the encoder.

[0037] The encoder can further compress the transform coefficients in quantization stage 214. During the transform process, different basic modes can represent different change frequencies (e.g., brightness change frequencies). Because the human eye is generally better at identifying low-frequency changes, the encoder can ignore information about high-frequency changes without causing a significant degradation in decoding quality. For example, in quantization stage 214, the encoder can generate quantization residual coefficients by dividing each transform coefficient by an integer value (called a "quantization parameter") and rounding the quotient to the nearest integer. Following this operation, some transform coefficients of the high-frequency fundamental mode can be converted to zero, while the transform coefficients of the low-frequency fundamental mode can be converted to smaller integers. The encoder can ignore the zero-valued quantization residual coefficients. Thus, the transformation coefficients are further compressed. The quantization process is also reversible, wherein the quantization residual coefficients... The transformation coefficients can be reconstructed in the inverse operation of the quantization (referred to as "inverse quantization") performed in the inverse quantization stage 218.

[0038] Because the encoder ignores the remainder of division in the rounding operation, the quantization stage 214 may be lossy. Typically, the quantization stage 214 may constitute the primary source of information loss during the encoding process. The greater the information loss, the lower the quantization residual coefficient. The fewer bits may be needed. To achieve compression with varying degrees of information loss, the encoder can use different values ​​of the quantization parameter or any other parameter of the quantization process.

[0039] Entropy coding is performed during the encoding process. In some embodiments, the symbols are encoded based on a statistical distribution of the symbols (quantization coefficients, motion vectors, and other auxiliary information), and the codec uses entropy coding to further compress the video data. The higher the frequency of a symbol, the shorter the code assigned to it, thus ensuring that the overall bit rate is minimized. Figure 2 As shown, the encoder can transmit the motion vector. and the quantized residual coefficient The data is fed to encoding module 226 to generate a bitstream to complete the forward path. Through encoding module 226, the encoder can use binary encoding techniques to process the motion vector. Quantization residual coefficient Encoding can be performed, for example, using entropy coding, variable-length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding (CABAC), or any other lossless or lossy compression algorithm. Therefore, the motion vector can be encoded using entropy coding methods. and the quantized residual coefficient The bits are encoded into bits and then sent to the decoder.

[0040] As described above, the quantization result is obtained through the inverse transformation module 240. It can be used to obtain the reconstructed residual through the inverse transformation. In this process, after quantization stage 214, the encoder can quantize the residual coefficients. The data is fed into the inverse quantization stage 218 and the inverse transform stage 220 to generate the reconstructed residual. In the inverse quantization stage 218, the encoder can quantize the residual coefficients. Inverse quantization is performed to generate reconstructed transform coefficients. During the inverse transform stage, the encoder can generate the reconstructed residual based on the reconstructed transform coefficients. Then, the encoder can reconstruct the residual. Added to the prediction frame In order to obtain the reconstructed frame to be used in the next iteration of the process. ,Right now The reconstructed frame The motion estimation module 210 will be used for the first Motion estimation of frames.

[0041] After prediction and reconstruction, artifacts may still remain in the video, such as blocking effects due to quantization. Loop filtering (such as deblocking filtering) is applied to the codec to process the reconstructed frames within the loop to eliminate these artifacts and improve visual quality. For example, in some embodiments, during the generation of the reconstructed frames... The encoder can then apply loop filtering stage 232 to the reconstructed frame. This is to reduce or eliminate distortion (e.g., blocking) introduced by the inter-frame prediction. In some embodiments, the encoder may apply various loop filtering techniques in the loop filtering stage 232, such as deblocking, sample adaptive offset (SAO), adaptive loop filtering (ALF), etc. In SAO, after the deblocking filtering, a nonlinear amplitude mapping is introduced within the inter-frame prediction loop to reconstruct the original signal amplitude using a lookup table described by a small number of additional parameters determined by histogram analysis at the encoder end.

[0042] The loop-filtered reference image can be stored in the decoded frame buffer 234 for subsequent use (e.g., as an inter-frame prediction reference frame for a future frame of the video sequence). The encoder can store one or more reference frames in the buffer 234 for use in the inter-frame prediction module 2044. In some embodiments, the encoder can adjust the parameters of the loop filter (e.g., loop filter strength) and the motion vector during the encoding stage. Quantization residual coefficient It encodes other information. The encoder can iteratively execute the process described above to encode each frame of the video sequence.

[0043] For the decoder, based on the bits provided by the encoding module 226 in the encoder, corresponding motion compensation, inverse transform, and frame reconstruction operations can be performed to obtain the reconstructed frame. .

[0044] To further improve coding performance, numerous algorithms have been developed for future video compression standards, including matrix-weighted intra-frame prediction, quadtree plus binary tree, extended coding unit segmentation, affine motion prediction, decoder-side motion vector refinement, and mode-dependent inseparable quadratic transform. Regarding encoder optimization, various optimization algorithms have been developed for different objectives, including rate-distortion optimization for signal and feature quality. For surveillance video data, considered the largest-scale dataset, the concept of golden frames has been introduced to provide better reference quality, and based on this concept, background modeling-based surveillance video compression technology has been developed. Furthermore, with the development of cloud computing, cloud databases have been introduced into image compression as external references to further eliminate redundancy. Moreover, due to the vast amount of information implicit in the bitstream, efforts have been made to analyze the bitstream without complete decoding. Scalable compression is becoming increasingly important in various applications such as video streaming. Scalable extensions have been proposed for various compression standards, including the scalable extensions to H.264 / MPEG-4 Part 10 AVC (H.264 / AVC) and H.265 / HEVC, denoted as Scalable Video Coding (SVC) and Scalable High-Efficiency Video Coding (SHVC), respectively. These scalable extensions support video scalability in terms of temporal resolution, spatial resolution, and quality. Furthermore, SHVC supports scalability in bit depth and color gamut to accommodate the deployment of Ultra High Definition (UHD) video.

[0045] Traditional hybrid codecs have played a key role in the surge of digital video, solving the dual challenges of limited bandwidth and storage with excellent efficiency.

[0046] Deep learning-based video compression technology is rapidly developing, and this research field aims to revolutionize video compression and decompression by leveraging the powerful capabilities of neural networks. Traditional video compression techniques based on manual algorithms and standards have been very successful, but have reached their efficiency limits, especially with the continuous increase in video resolution and frame rate. Deep learning-based methods offer a new approach to improving compression performance by directly learning the optimal representation and compression strategy from the data.

[0047] The initial attempts at deep learning-based video compression began with exploring how machine learning, particularly neural networks, could be applied to improve the limitations of traditional codec architectures. Researchers began experimenting with various components of the video compression pipeline, aiming to explore the adaptable links of neural networks and how they could enhance the overall process. The first experiments in this area replaced specific components of existing codecs with neural network-based schemes. For example, one of the earliest applications was predictive coding, where neural networks were trained to predict pixel values ​​more accurately than traditional linear models. Another area of ​​exploration was using neural networks to improve motion estimation and compensation, which are central to inter-frame compression in traditional codecs. Developers of deep learning-based codecs also focused on intra-frame compression. Convolutional neural networks (CNNs) are particularly well-suited for this task because they are able to capture spatial hierarchies in image data. CNNs are trained to transform images into compact feature maps, which are then quantized and entropy-encoded, similar to the process in traditional codecs. However, unlike manually designed transformations such as DCT, these neural networks can learn transformations tailored specifically to the characteristics of the video content. Entropy coding, a lossless compression step designed to minimize the number of bits required to represent the data, has also attracted significant attention from academia and industry. Researchers have begun exploring how neural networks can optimize entropy coding by learning the probability distribution of the data more effectively. Techniques such as autoencoders and recurrent neural networks (RNNs) have been used to model and encode data in a more compact form than traditional entropy coding methods such as Huffman coding or arithmetic coding. With the advent of deep learning, the concept of in-loop filtering has been reimagined, leading to the development of deep learning-based in-loop filters that can learn to perform the task more effectively. Neural networks are trained to reduce artifacts caused by quantization and other lossy compression processes, such as blockiness, striping, and blurring. Unlike static filtering, deep learning-based filtering can adapt to the content and characteristics of the video, potentially offering better reconstruction quality and compression efficiency.

[0048] Beyond integrating deep learning techniques into traditional hybrid video codecs, this paper explores ways to improve video compression efficiency through deep learning-based end-to-end architectures and their optimization methods. End-to-end learning for video compression represents an innovative approach where deep learning techniques are used to conceptualize, design, and optimize the entire compression pipeline as a complete system. This holistic perspective contrasts with traditional methods that consist of disparate modules, each designed to perform a specific task such as motion estimation, transform, or entropy coding. Through end-to-end learning, neural networks are trained to perform all these functions within a unified framework, potentially leading to the discovery of more efficient and higher-quality video compression strategies by directly optimizing a loss function that reflects the desired trade-off between compression ratio and video quality.

[0049] A pioneering work applied recurrent neural networks (RNNs) to end-to-end learning of image representations, achieving performance comparable to JPEG. Researchers proposed a block-transform-based image compression model, based on a combination of discrete cosine transform (DCT) and convolutional neural network (CNN) models, which outperforms JPEG at low bit rates. A generalized divisive normalization (GDN)-based image coding method was proposed, employing a density estimation model and showing a significant improvement in compression performance compared to JPEG 2000. Based on this method, redundancy can be further eliminated using a variational super-prior model, which outperforms BPG in rate-distortion performance. A deep video compression (DVC) method based on an end-to-end model also achieved better performance than H.264 / AVC. In general, deep learning image compression (DLIC) optimizes the parameters of the encoder and decoder through specific rate-distortion (RD) tradeoffs. Various methods for variable bit-rate DLIC have been proposed to explore the generalization ability of DLIC under various RD tradeoffs and reduce the storage requirements of storing model parameters. Specifically, a conditional autoencoder based on Lagrange multipliers and quantization interval size is designed. A gain unit is proposed to achieve continuously variable bitrate in a single model. A channel attention module is introduced to further utilize bitrate variability. Furthermore, a learnable variable bitrate multiband image compression algorithm is proposed based on the proposed Modulation Generalized Octave Convolution (GoConv) and Octave Transposed Convolution (GoTConv). The proposed scheme achieves performance comparable to HEVC in YUV PSNR over a wide bitrate range using only three models, and shows a significant improvement over VVC in YMS-SSIM. A variable bitrate video compression scheme with a deep modulation autoencoder is also proposed, which achieves continuously variable bitrate with compression efficiency almost comparable to multiple fixed bitrate models.

[0050] Video coding for machine vision refers to the specialized optimization of video compression techniques for the needs of machine vision systems rather than human viewers. Machine vision systems are used in various applications such as autonomous vehicles, robotics, industrial automation, and surveillance, where algorithms need to perform detection, classification, and decision-making based on video input. The requirements of machine vision systems can differ significantly from those of human viewers, which has spurred the development of specialized video coding techniques to meet these unique needs. Video coding for machine vision attempts to modify and optimize traditional video compression techniques to suit the requirements of automated analysis systems. Its core principles are preserving features important for machine understanding, maintaining low latency, ensuring robustness to environmental conditions, and achieving efficient compression to reduce storage and bandwidth overhead.

[0051] In recent years, various algorithms have been proposed to improve the compression efficiency of VCM. These algorithms mainly focus on the core component of the codec (called the core codec), namely, the conversion of the visual signal into the bitstream. Generally, the core codec can be classified into codecs with hybrid coding frameworks and deep learning-based codecs, both of which have been studied for machine vision in recent years. In the hybrid coding framework used for image codecs, the quantization model can be optimized in terms of rate-accuracy performance. Similarly, various algorithms have been proposed to improve the compression performance of video codecs for machine vision from the perspective of the QP decision models of HEVC and VVC. Furthermore, due to the low sensitivity of machine vision tasks to bit depth, a bit-level efficiency optimization method has been proposed to improve the coding performance of VSCM by truncating the bit depth of luminance sample values.

[0052] For a deep learning-based codec, an end-to-end compression model is optimized by combining machine vision task loss, image distortion loss, and bitrate loss, which significantly outperforms VVC on object detection and instance segmentation tasks. Furthermore, compression performance is improved by optimizing the encoder of the pre-trained codec through online fine-tuning. A Latent Spatial Masking Network (LSMnet) is proposed to mask spatial elements that may be unimportant for instance segmentation tasks. A saliency-driven hierarchical neural network image compression model is used to reduce redundancy, and an object detection network is used to extract saliency information. In addition, an end-to-end image compression method for multiple machine tasks is designed to achieve the conversion of compressed content from the primary task (object detection) to the secondary task (classification). Inspired by the representational power of deep learning features in machine analysis, a learning-based codec is proposed that can compress extracted machine analysis features and reconstruct the visual signal based on the decoded analysis features. The distortion in the joint loss function of the learning-based compression model mainly consists of two parts: signal-level fidelity and machine vision loss, where the weights between the two parts are set empirically, limiting the compression efficiency. To address this issue, a general rate-precision optimization algorithm is proposed, achieving performance superior to various empirical settings. Joint optimization of the end-to-end compression model and the task model is investigated, resulting in object detection performance superior to the human vision system-guided (HVS)-based end-to-end compression model. Furthermore, a unified optimization framework is proposed, integrating a variable bitrate module into the codec and machine analysis model. Learning-based video compression for machine vision is also investigated. An end-to-end video compression framework combining variable bitrate intra-frame coding and scale-space stream-based inter-frame coding is developed. A learnable image codec for intra-frame coding is proposed and combined with VVC for inter-frame coding, termed NNVVC. Compared to VVC, NNVVC achieves 51.76% and 38.07% BD bitrate savings, respectively, on instance segmentation tasks on the Open Image Dataset and TVD Dataset.

[0053] Driven by the high temporal redundancy of video data, temporal resampling is an important solution for improving video compression efficiency. For temporal resampling geared towards human vision, a frame rate-related quality metric (FRQM) can be selected as the basis for temporal resampling decisions by comparing with a predefined threshold. Generally, temporal downsampling is performed as a preprocessing method at the encoding end, saving coding overhead by discarding frames. At the decoding end, various temporal upsampling algorithms have been proposed to improve reconstruction quality. Motion-compensated temporal upsampling has been studied. With the development of optical flow, various optical flow-based temporal interpolation methods have been proposed. However, all these methods are designed for human perception. To improve the coding efficiency of temporal resampling for machine vision, the frame interpolation model can be optimized to adapt to object detection. However, the time complexity of various video contents may be different, indicating that for better robustness and coding efficiency, temporal resampling should be performed adaptively according to the video content. Currently, adaptive video temporal resampling algorithms for machine vision have not been fully studied.

[0054] The embodiments proposed in this disclosure implement an analytics-based adaptive temporal resampling decision algorithm for machine vision to further improve robustness and coding efficiency.

[0055] Figure 3 This is an example schematic diagram of an analysis-based temporal resampling compression framework 300, illustrated according to some embodiments of the present disclosure.

[0056] like Figure 3 As shown in the temporal resampling-based general compression framework 300, the input video data of the input video 310... It can first be processed by the pre-analysis model 322, and then by the time downsampling model 324 based on the resampling ratio at the encoding end. The data is then rescaled. Therefore, the output downsampled data... The bitstream 330 can be generated by the encoder 326. At the decoding end, the decoder 342 is configured to receive the bitstream 330 and decode it to obtain decompressed data. Based on the resampling ratio transmitted along with the bitstream 330. The decompressed data The video data can be rescaled using the temporal upsampling model 344 at the decoding end to obtain the reconstructed video data. As the output of temporal upsampling. In some embodiments, the reconstructed video data It can be sent to machine analysis model 346 for further machine vision processing.

[0057] For temporal upsampling, various video frame interpolation methods can be selected. For example, in some embodiments, a simple duplication method can be chosen, which performs temporal upsampling by repeating the most recent decoded frame. In some embodiments, the resampling ratio... It can be determined by the pre-analysis model 322.

[0058] Due to the diversity of video data content, determining an appropriate temporal resampling ratio is a key challenge. An adaptive strategy is needed to proactively decide whether temporal resampling should be applied, and if so, at what ratio. However, the development of such adaptive temporal resampling decision algorithms has not been fully researched and explored.

[0059] This disclosure presents embodiments to describe an analytics-based adaptive temporal resampling decision algorithm for machine vision.

[0060] Figure 4 This is a block diagram of an example apparatus 400 for encoding or decoding image data according to some embodiments of the present disclosure. Figure 4 As shown, device 400 may include processor 402. When processor 402 executes the instructions described herein, device 400 may become a dedicated machine for video encoding or decoding. Processor 402 may be any type of circuit system capable of manipulating or processing information. For example, processor 402 may include any number of central processing units (or “CPU”), graphics processing units (or “GPU”), neural processing units (“NPU”), microcontroller units (“MCU”), optical processors, programmable logic controllers, microcontrollers, microprocessors, digital signal processors, intellectual property (IP) cores, programmable logic arrays (PLAs), programmable array logic (PALs), general-purpose array logic (GALs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), system-on-a-chip (SoCs), application-specific integrated circuits (ASICs), and any combination thereof. In some embodiments, processor 402 may also be a set of processors grouped into individual logic components. For example, as Figure 4 As shown, processor 402 may include multiple processors, including processor 402a, processor 402b and processor 402n.

[0061] The device 400 may also include a memory 404 configured to store data (e.g., instruction sets, computer code, intermediate data, etc.). For example, such as Figure 4As shown, the stored data may include program instructions (e.g., program instructions for implementing the stages of the process in framework 200 or framework 300) and data for processing (e.g., video sequences, video bitstreams, or video streams). Processor 402 can access the program instructions and the data for processing (e.g., via bus 410) and execute the program instructions to perform operations or manipulations on the data for processing. Memory 404 may include a high-speed random access memory device or a non-volatile memory device. In some embodiments, memory 404 may include any combination of any number of random access memories (RAM), read-only memories (ROM), optical discs, magnetic disks, hard disks, solid-state drives, flash drives, secure digital cards (SD cards), memory sticks, compact flash (CF) cards, etc. Memory 404 may also be a group of memories grouped into a single logical component. Figure 4 (Not shown in the image).

[0062] Bus 410 may be a communication device for transmitting data between components within device 400, such as an internal bus (e.g., CPU-memory bus), an external bus (e.g., a Universal Serial Bus port, a Peripheral Component Interconnect Fast Port), etc.

[0063] For ease of explanation and to avoid ambiguity, processors 402a-402n and other data processing circuitry are collectively referred to as "data processing circuitry" in this disclosure. The data processing circuitry may be implemented entirely as hardware, or as a combination of software, hardware, or firmware. Furthermore, the data processing circuitry may be a single, independent module, or may be wholly or partially integrated into any other component of the device 400.

[0064] Device 400 may also include a network interface 406 to provide wired or wireless communication with a network (e.g., the Internet, intranet, local area network, mobile communication network, etc.). In some embodiments, network interface 406 may include any combination of any number of network interface controllers (NICs), radio frequency (RF) modules, transceivers, transceivers, modems, routers, gateways, wired network adapters, wireless network adapters, Bluetooth adapters, infrared adapters, near field communication (“NFC”) adapters, cellular network chips, etc.

[0065] In some embodiments, the device 400 may optionally include a peripheral interface 408 to provide connectivity to one or more peripheral devices. Figure 4 As shown, the peripheral devices may include, but are not limited to, cursor control devices (e.g., mouse, touchpad, or touchscreen), keyboards, displays (e.g., cathode ray tube displays, liquid crystal displays, or light-emitting diode displays), video input devices (e.g., cameras or input interfaces coupled to video files), etc.

[0066] It should be noted that the video codec (e.g., the codec that executes the processes in frame 200 or frame 300) can be implemented as any combination of any software or hardware modules in device 400. For example, some or all stages of the processes in frame 200 or frame 300 can be implemented as one or more software modules of device 400, such as program instructions that can be loaded into memory 404. As another example, some or all stages of the processes in frame 200 or frame 300 can be implemented as one or more hardware modules of device 400, such as dedicated data processing circuitry (e.g., FPGA, ASIC, NPU, etc.).

[0067] Figure 5 This is a flowchart of an example method 500 for decoding a bitstream associated with a video sequence according to some embodiments of this disclosure. Method 500 can be performed by a decoder to decode the video bitstream. For example, the decoder can be implemented for decoding the bitstream (e.g., Figure 3 An apparatus for decoding the bitstream 330 in order to reconstruct video frames or video sequences of the bitstream (e.g., ...). Figure 4 One or more software or hardware components of the device 400 in the middle. For example, a processor (e.g., Figure 4 The processor 402 can execute method 500. For example... Figure 5 As shown, method 500 includes the following steps 510-550.

[0068] At step 510, the decoder receives a bitstream associated with the video sequence (e.g., Figure 3 The decoder decompresses the decompressed bitstream (330) and then reconstructs one or more frames of the video sequence based on the decompressed bitstream. Figure 3 Decompressed data ).

[0069] At step 530, the decoder determines the sequence-level average intersection-over-union ratio (MIOU) based on one or more reconstructed frames. Generally, temporal resampling can effectively improve video coding efficiency when the video has low time complexity. From a machine vision perspective, this time complexity can be evaluated by the degree of target overlap between different frames. In some embodiments, the average intersection-over-union (MIOU) ratio is used. It is defined as a sequence-level metric and used to determine the time resampling ratio. The sequence-level MIOU Based on the MIOU between two frames, it is marked as a frame-level MIOU. In other words, MIOU can be calculated at both the frame and sequence levels, and the methods vary depending on the coding configuration. For example, for various coding configurations such as Random Access (RA), Low Latency (LD), and Intra-Frame (AI), the sequence-level MIOU... The computation methods can be different to accommodate reference dependencies and to better assess time complexity.

[0070] In some embodiments, at step 530, the decoder calculates multiple frame-level MIOUs for the frame pair. Each frame pair comprises two frames from the video sequence and is based on the frame-level MIOU. Calculate the sequence-level MIOU The frame-level MIOU will first be defined in the following paragraphs. Then, the sequence-level MIOUs under various configurations are further defined. Finally, an adaptive algorithm for determining the time resampling ratio is shown.

[0071] To calculate frame-level MIOU, given two frames of a video sequence and an object detection model, object detection can be performed on the two frames. The detection result for each frame can be labeled as follows: and The bounding boxes are ranked by confidence score. Sort in descending order. Here, and These are the coordinates of the top-left point of the detected bounding box. and These are the width and height of the detected bounding box. It is the category index of the bounding box, and It is a confidence score ranging from 0 to 1.

[0072] In some embodiments, bounding box matching pairs between two frames are first obtained. For example, various methods can be used to pair detected bounding boxes between two frames, such as greedy algorithms, the Kuhn-Munkres algorithm, and template matching algorithms. Taking the greedy algorithm as an example, specifically, for... For each instance in the dataset, its relationship with... The intersection-union ratio (IOU, denoted as) among all instances of the same class in the dataset. (You can choose) The instance with the largest IOU (where IOU must be greater than 0) is used as... The matching results of the instances. After finding a pair of bounding boxes, and The paired bounding boxes mentioned above are excluded from subsequent matching processes to achieve bijective mapping. Therefore, and The matching results can be characterized as Here, to make the comparison meaningful, confidence scores below a predefined threshold are not considered. The bounding box. For example, It can be 0.5, but this disclosure is not limited to this.

[0073] To differentiate the time complexity differences across various target scales, frame-level MIOU can be measured at three spatial scales. In some embodiments, the paired bounding boxes can be classified into multiple groups. For example, the paired bounding boxes can be classified as targets with regions smaller than [a certain size]. group The target area is smaller than and greater than The middle group and the target area is larger than large group In other words, the paired bounding boxes are classified into three groups, labeled as follows: These correspond to the small group, medium group, and large group, respectively. For each spatial group... The bounding boxes are grouped based on the target category index and labeled as ,in yes The number of target category indexes in the group. The frame-level MIOU can be defined as ,in Instruction Group k and category index j The IOU value, and It is a category index The number of detection results. Based on this, the frame-level MIOU can be defined as... .also, The number of detected bounding boxes is , This can be defined based on confidence scores. Specifically, the number of bounding boxes for the small, medium, and large groups is characterized as... .

[0074] In some embodiments, the detected target categories used for the MIOU calculation can be different for various final machine vision tasks. For example, for a target detection task, all detected target categories may be considered in the MIOU calculation, while for a target tracking task, only relevant moving target categories, such as people, cars, and buses, may be considered.

[0075] Regarding the sequence level MIOU For each sequence, the sequence-level MIOU can be calculated periodically based on a batch of frames. In some embodiments, for simplicity, only the first part of the video sequence is calculated. Frames. The sequence-level MIOU It can rely on encoding configuration to adapt to the encoding structure.

[0076] In some embodiments, at step 530, the decoder may calculate the sequence-level MIOU based on the encoding configuration for encoding the video sequence. For example, in response to a random access (RA) coding configuration, the decoder can identify selected frames to calculate the frame-level MIOU, and calculate the sequence-level MIOU based on a weighted sum of the frame-level MIOUs, and the frame pairs correspond to multiple interval levels.

[0077] For example, for an RA-coded configuration, the minimum interval for calculating the frame-level MIOU is: ,in Should be The power of 2. The frame-level MIOU between the following frame pairs is calculated based on the following formula:

[0078] At different intervals (for Select the frame pair. The interval is defined based on the following formula. Frame-level MIOU at the following levels: , in, for .based on The weighted sum is used to define the sequence level. In some embodiments, for larger intervals The weights are also relatively large. That is to say, for the first interval level of the first frame pair, which is greater than the second interval level of the second frame pair, the first weight of the first frame level MIOU of the first frame pair is greater than the second weight of the second frame level MIOU of the second frame pair.

[0079] For example, for a resampling ratio of 2, It is 32, and The value is 8. Therefore, yes ,and For the resampling ratio of 4, It is 32, and It is 16. Therefore, yes ,and 1.25 is the normalization factor.

[0080] Additionally, in response to a low-latency (LD) coding configuration, the decoder can identify selected frames at fixed intervals to calculate the frame-level MIOU, and calculate the sequence-level MIOU by calculating the average of the frame-level MIOU, wherein each frame pair includes two adjacent selected frames.

[0081] For example, for an LD encoding configuration, the interval used to calculate the frame-level MIOU is: ,in It could also be Powers of 2. Calculate frame pairs. The frame-level MIOU between. Similarly, the sequence-level It can be defined as the average MIOU per frame. For example, for a resampling ratio of 2, It is 32. 8 and For a resampling ratio of 4, It is 32. 16 and .

[0082] In some embodiments, for AI coding configurations, a smaller interval can be selected. To calculate the frame-level MIOU. Calculate frame pairs. The frame-level MIOU between. Similarly, the sequence-level It can be defined as the average MIOU per frame. For example, for a resampling ratio of 2, It is 32. 2 and For a resampling ratio of 4, It is 32. 4 and .

[0083] In some embodiments, adaptive decision-making may be employed. At step 540, the decoder obtains a decision metric based on the sequence-level MIOU to determine the resampling ratio and whether to perform temporal resampling. In some embodiments, the decoder may obtain the number of bounding boxes corresponding to multiple groups of different target region sizes, obtain multiple sequence-level MIOUs for the groups, and obtain a decision metric based on the number of bounding boxes in the groups and the sequence-level MIOUs.

[0084] As described above, the paired bounding boxes can be classified into small, medium, and large groups corresponding to different target region sizes. For each group, the bounding boxes are grouped based on the target category index, and the number of detected bounding boxes in the small, medium, and large groups is [number missing]. The sequence-level MIOUs of the subgroup, intermediate group, and large group can be obtained accordingly. .

[0085] For example, for sequence-level MIOU The decision indicators It can be defined as For the time resampling 2, if the decision index If the value is greater than or equal to 0.5, then time resampling is performed. Otherwise, time resampling is not performed. If the decision metric... If the value is greater than or equal to 0.5, then the decision index of time resampling 4 is also calculated. If the decision metric of time resampling 4 is... If the value is greater than or equal to 0.6, then time resampling 4 is performed to replace time resampling 2.

[0086] Then, at step 550, the decoder may resample the one or more reconstructed frames based on the decision metric.

[0087] Figure 6 This is a flowchart of an example method 600 for encoding a video sequence into a bitstream according to some embodiments of the present disclosure. Method 600 may be performed by an encoder to encode the video bitstream. For example, the encoder may be implemented for encoding the bitstream (e.g., Figure 3 A device for encoding the bitstream 330 in the video stream to reconstruct video frames or video sequences (e.g., ...). Figure 4 One or more software or hardware components of the device 400 in the middle. For example, a processor (e.g., Figure 4 The processor 402 can execute method 600. For example... Figure 6 As shown, method 600 includes steps 610-650. At step 610, the encoder receives a video sequence (e.g., Figure 3 (Input video 310).

[0088] At step 620, the encoder determines the sequence-level average intersection-and-union ratio (MIOU) based on one or more frames of the video sequence. As described above, in some embodiments, the encoder may calculate the frame-level MIOU for frame pairs and calculate the sequence-level MIOU based on the frame-level MIOU. Each frame pair comprises two frames of the video sequence. In some embodiments, the encoder calculates the sequence-level MIOU according to the encoding configuration used to encode the video sequence.

[0089] For example, in response to a random access (RA) coding configuration, the encoder can identify selected frames to compute the frame-level MIOU, wherein the frame pairs correspond to multiple interval levels. The encoder can then compute the sequence-level MIOU based on a weighted sum of the frame-level MIOUs. In some embodiments, for a first interval level of a first frame pair that is greater than a second interval level of a second frame pair, a first weight of the first frame-level MIOU of the first frame pair is greater than a second weight of the second frame-level MIOU of the second frame pair.

[0090] Additionally, in response to a low-latency (LD) coding configuration, the encoder can identify selected frames at fixed intervals to calculate the frame-level MIOU, where each frame pair includes two adjacent selected frames. The encoder can then calculate the sequence-level MIOU by averaging multiple frame-level MIOUs.

[0091] At step 630, the encoder obtains a decision metric based on the sequence-level MIOU to determine the resampling ratio and whether to perform temporal resampling. In some embodiments, the encoder may obtain the number of bounding boxes corresponding to multiple groups of different target region sizes, obtain multiple sequence-level MIOUs for the groups, and obtain the decision metric based on the number of bounding boxes and the sequence-level MIOUs for the groups. The operational details in steps 620 and 630 are similar to those in other embodiments. Figure 5 The operational details in steps 530 and 540 of method 500 are similar or identical, so for the sake of brevity, they will not be repeated here.

[0092] At step 640, the encoder resamples the one or more frames based on the decision metric. For example, the one or more frames can be derived from... Figure 3 The time-downsampling model 324 shown is rescaled.

[0093] At step 650, the encoder encodes the one or more resampled frames to generate a bitstream. For example, such as Figure 3 As shown, the downsampled data The bitstream 330 can be compressed by the encoder 326 to generate the bitstream 330.

[0094] The embodiments described in this disclosure can be freely combined.

[0095] In some embodiments, a non-transitory computer-readable storage medium for storing bitstreams is also provided. The bitstreams can be encoded and decoded according to the disclosed method for implementing an analysis-based adaptive temporal resampling decision algorithm for machine vision.

[0096] In some embodiments, a method for storing a bitstream includes the following operations: receiving a video sequence comprising one or more frames; encoding the video sequence; and transmitting a bitstream generated based on the encoding. Encoding the video sequence involves the following operations: determining a sequence-level average intersection-to-union ratio (MIOU) based on the one or more frames of the video sequence; obtaining a decision metric based on the sequence-level MIOU to determine whether to perform temporal resampling and a resampling ratio; resampling the one or more frames based on the decision metric; and encoding the one or more resampled frames. The operational details of encoding the video sequence are as follows... Figure 5 Method 500 and Figure 6 The operational details in method 600 are similar or identical, so for the sake of brevity, they will not be repeated here.

[0097] In some embodiments, a non-transitory computer-readable storage medium including instructions is also provided, and the instructions can be executed by a device (e.g., the disclosed encoder and decoder) to perform the methods described above. Common forms of non-transitory media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs or any other flash memory, NVRAMs, caches, registers, any other memory chips or cassette tapes, and their networking versions. The device may include one or more processors (CPUs), input / output interfaces, network interfaces, and / or memory.

[0098] It should be noted that the relational terms such as "first," "second," etc., used in this document are only used to distinguish one entity or operation from another, and do not require or imply any actual relationship or sequence between these entities or operations. Furthermore, the words "comprising," "having," "containing," and "including," as well as other similar forms, are intended to be identical in meaning and open-ended, because one or more items following any of these words do not imply an exhaustive list of such items, nor do they imply limitation to only the listed items.

[0099] As used herein, unless otherwise specified, the term "or" covers all possible combinations unless impractical. For example, if it is specified that a database may include A or B, then unless otherwise specified or impractical, the database may include A, or B, or A and B. As a second example, if it is specified that a database may include A, B, or C, then unless otherwise specified or impractical, the database may include A, or B, or C, or A and B, or A and C, or B and C, or A and B and C.

[0100] The embodiments may be further described using the following terms: 1. A method for decoding a bitstream associated with a video sequence, the method comprising: Decompress the bitstream associated with the video sequence; Based on the decompressed bitstream, reconstruct one or more frames of the video sequence; Based on one or more reconstructed frames, determine the sequence-level average intersection-over-union ratio (MIOU). Decision metrics are obtained based on the sequence-level MIOU to determine the resampling ratio and whether to perform temporal resampling; and Based on the decision metrics, the one or more reconstructed frames are resampled.

[0101] 2. The method according to Clause 1, further comprising: For multiple frame pairs, calculate multiple frame-level MIOUs, where each frame pair includes two frames from the video sequence; and The sequence-level MIOU is calculated based on the multiple frame-level MIOUs.

[0102] 3. The method according to Clause 2, further comprising: The sequence-level MIOU is calculated based on the encoding configuration used to encode the video sequence.

[0103] 4. The method described in accordance with Clause 2 or 3, further comprising: In response to a random access (RA) coding configuration, multiple selected frames are identified to calculate the multiple frame-level MIOUs, wherein the frame pairs correspond to multiple interval levels; and The sequence-level MIOU is calculated based on the weighted sum of the multiple frame-level MIOUs.

[0104] 5. The method according to Clause 4, wherein, for the case where the first interval level of the first frame pair is greater than the second interval level of the second frame pair, the first weight of the first frame level MIOU of the first frame pair is greater than the second weight of the second frame level MIOU of the second frame pair.

[0105] 6. The method according to any one of clauses 2 to 5, further comprising: In response to a low-latency (LD) coding configuration, multiple selected frames are identified at fixed intervals to calculate the multiple frame-level MIOUs, wherein each frame pair includes two adjacent selected frames; and The sequence-level MIOU is calculated by averaging the multiple frame-level MIOUs.

[0106] 7. The method according to any one of clauses 1 to 6, further comprising: Obtain the number of multiple bounding boxes corresponding to multiple groups with different target region sizes; Obtain multiple sequence-level MIOUs from the multiple groups; and The decision index is obtained based on the number of bounding boxes in the multiple groups and the multiple sequence-level MIOUs.

[0107] 8. A method for encoding a video sequence into a bitstream, the method comprising: Based on one or more frames of the video sequence, determine the sequence-level average intersection-over-union ratio (MIOU). Decision metrics are obtained based on the sequence-level MIOU to determine the resampling ratio and whether to perform time resampling. Based on the decision metrics, the one or more frames are resampled; and The one or more resampled frames are encoded to generate a bitstream.

[0108] 9. The method described in Clause 8, further comprising: For multiple frame pairs, calculate multiple frame-level MIOUs, where each frame pair includes two frames from the video sequence; and The sequence-level MIOU is calculated based on the multiple frame-level MIOUs.

[0109] 10. The method according to Clause 9, further comprising: The sequence-level MIOU is calculated based on the encoding configuration used to encode the video sequence.

[0110] 11. The method according to Clause 9 or 10, further comprising: In response to a random access (RA) coding configuration, multiple selected frames are identified to calculate the multiple frame-level MIOUs, wherein the frame pairs correspond to multiple interval levels; and The sequence-level MIOU is calculated based on the weighted sum of the multiple frame-level MIOUs.

[0111] 12. The method according to Clause 11, wherein, for the case where the first interval level of the first frame pair is greater than the second interval level of the second frame pair, the first weight of the first frame level MIOU of the first frame pair is greater than the second weight of the second frame level MIOU of the second frame pair.

[0112] 13. The method according to any one of clauses 9 to 12, further comprising: In response to a low-latency (LD) coding configuration, multiple selected frames are identified at fixed intervals to calculate the multiple frame-level MIOUs, wherein each frame pair includes two adjacent selected frames; and The sequence-level MIOU is calculated by averaging the multiple frame-level MIOUs.

[0113] 14. The method according to any one of clauses 8 to 13, further comprising: Obtain the number of multiple bounding boxes corresponding to multiple groups with different target region sizes; Obtain multiple sequence-level MIOUs from the multiple groups; and The decision index is obtained based on the number of bounding boxes in the multiple groups and the multiple sequence-level MIOUs.

[0114] 15. A method for transmitting a bit stream, the method comprising: Receive a video sequence comprising one or more frames; The video sequence is encoded using the following operations: Based on the one or more frames of the video sequence, determine the sequence-level average intersection-over-union ratio (MIOU). Decision metrics are obtained based on the sequence-level MIOU to determine the resampling ratio and whether to perform time resampling. Based on the decision metrics, the one or more frames are resampled; and Encode the one or more resampled frames; and Send the bit stream generated based on the encoding.

[0115] 16. The method according to Clause 15, wherein encoding the video sequence further comprises: For multiple frame pairs, calculate multiple frame-level MIOUs, where each frame pair includes two frames from the video sequence; and The sequence-level MIOU is calculated based on the multiple frame-level MIOUs.

[0116] 17. The method according to Clause 16, wherein encoding the video sequence further comprises: The sequence-level MIOU is calculated based on the encoding configuration used to encode the video sequence.

[0117] 18. The method according to clause 16 or 17, wherein encoding the video sequence further comprises: In response to a random access (RA) coding configuration, multiple selected frames are identified to calculate the multiple frame-level MIOUs, wherein the frame pairs correspond to multiple interval levels; and The sequence-level MIOU is calculated based on the weighted sum of the multiple frame-level MIOUs.

[0118] 19. The method according to Clause 18, wherein, for the case where the first interval level of the first frame pair is greater than the second interval level of the second frame pair, the first weight of the first frame level MIOU of the first frame pair is greater than the second weight of the second frame level MIOU of the second frame pair.

[0119] 20. The method according to any one of clauses 16 to 19, wherein encoding the video sequence further comprises: In response to a low-latency (LD) coding configuration, multiple selected frames are identified at fixed intervals to calculate the multiple frame-level MIOUs, wherein each frame pair includes two adjacent selected frames; and The sequence-level MIOU is calculated by averaging the multiple frame-level MIOUs.

[0120] It should be understood that the above embodiments can be implemented in hardware, software (program code), or a combination of hardware and software. If implemented in software, it can be stored in the above-described computer-readable medium. When executed by a processor, the software can perform the disclosed methods. The computing units and other functional units described in this disclosure can be implemented in hardware, software, or a combination of hardware and software. Those skilled in the art should also understand that multiple modules / units described above can be combined into one module / unit, and each module / unit described above can be further divided into multiple sub-modules / sub-units.

[0121] In the foregoing specification, numerous specific details have been described with reference to embodiments, which may vary depending on the implementation. Certain adjustments and modifications may be made to the described embodiments. Other embodiments will be apparent to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the appended claims. The sequence of steps shown in the figures is also to be considered for illustrative purposes only and is not intended to be limited to any particular sequence of steps. Therefore, those skilled in the art will understand that these steps may be performed in a different order while implementing the same method.

[0122] Exemplary embodiments have been disclosed in the accompanying drawings and description. However, many variations and modifications can be made to these embodiments. Therefore, although specific terminology has been used, it is used only in a general and descriptive sense and not for limiting purposes.

Claims

1. A method for decoding a bitstream associated with a video sequence, the method comprising: Decompress the bitstream associated with the video sequence; Based on the decompressed bitstream, reconstruct one or more frames of the video sequence; Determine the sequence-level average intersection-union ratio (MIOU) based on one or more reconstructed frames; Decision metrics are obtained based on the sequence-level MIOU to determine the resampling ratio and whether to perform time resampling. as well as Based on the decision metrics, the one or more reconstructed frames are resampled.

2. The method according to claim 1, further comprising: For multiple frame pairs, calculate multiple frame-level MIOUs, where each frame pair includes two frames from the video sequence; and The sequence-level MIOU is calculated based on the multiple frame-level MIOUs.

3. The method according to claim 2, further comprising: The sequence-level MIOU is calculated based on the encoding configuration used to encode the video sequence.

4. The method of claim 2, further comprising: In response to a random access RA coding configuration, multiple selected frames are identified to calculate the multiple frame-level MIOUs, wherein the frame pairs correspond to multiple interval levels; and The sequence-level MIOU is calculated based on the weighted sum of the multiple frame-level MIOUs.

5. The method according to claim 4, wherein, If the first interval level of the first frame pair is greater than the second interval level of the second frame pair, the first weight of the first frame level MIOU of the first frame pair is greater than the second weight of the second frame level MIOU of the second frame pair.

6. The method of claim 2, further comprising: In response to a low-latency LD coding configuration, multiple selected frames are identified at fixed intervals to calculate the multiple frame-level MIOUs, wherein each frame pair includes two adjacent selected frames; and The sequence-level MIOU is calculated by averaging the multiple frame-level MIOUs.

7. The method of claim 1, further comprising: Obtain the number of multiple bounding boxes corresponding to multiple groups with different target region sizes; Obtain multiple sequence-level MIOUs for the multiple groups; as well as The decision metric is obtained based on the number of bounding boxes in the multiple groups and the multiple sequence-level MIOUs.

8. A method for encoding a video sequence into a bitstream, the method comprising: Based on one or more frames of the video sequence, determine the sequence-level average intersection-union ratio (MIOU). Decision metrics are obtained based on the sequence-level MIOU to determine the resampling ratio and whether to perform time resampling. Based on the decision indicators, the one or more frames are resampled; as well as The one or more resampled frames are encoded to generate a bitstream.

9. The method of claim 8, further comprising: For multiple frame pairs, calculate multiple frame-level MIOUs, where each frame pair includes two frames from the video sequence; and The sequence-level MIOU is calculated based on the multiple frame-level MIOUs.

10. The method of claim 9, further comprising: The sequence-level MIOU is calculated based on the encoding configuration used to encode the video sequence.

11. The method of claim 9, further comprising: In response to a random access RA coding configuration, multiple selected frames are identified to calculate the multiple frame-level MIOUs, wherein the frame pairs correspond to multiple interval levels; and The sequence-level MIOU is calculated based on the weighted sum of the multiple frame-level MIOUs.

12. The method according to claim 11, wherein, If the first interval level of the first frame pair is greater than the second interval level of the second frame pair, the first weight of the first frame level MIOU of the first frame pair is greater than the second weight of the second frame level MIOU of the second frame pair.

13. The method of claim 9, further comprising: In response to a low-latency LD coding configuration, multiple selected frames are identified at fixed intervals to calculate the multiple frame-level MIOUs, wherein each frame pair includes two adjacent selected frames; and The sequence-level MIOU is calculated by averaging the multiple frame-level MIOUs.

14. The method of claim 8, further comprising: Obtain the number of multiple bounding boxes corresponding to multiple groups with different target region sizes; Obtain multiple sequence-level MIOUs from the multiple groups; as well as The decision index is obtained based on the number of bounding boxes in the multiple groups and the multiple sequence-level MIOUs.

15. A method for transmitting a bit stream, the method comprising: Receive a video sequence comprising one or more frames; The video sequence is encoded using the following operations: Based on the one or more frames of the video sequence, determine the sequence-level average intersection-union ratio (MIOU). Decision metrics are obtained based on the sequence-level MIOU to determine the resampling ratio and whether to perform time resampling. Based on the decision indicators, the one or more frames are resampled; as well as Encode the one or more resampled frames; as well as Send the bit stream generated based on the encoding.

16. The method according to claim 15, wherein, The encoding of the video sequence further includes: For multiple frame pairs, calculate multiple frame-level MIOUs, where each frame pair includes two frames from the video sequence; and The sequence-level MIOU is calculated based on the multiple frame-level MIOUs.

17. The method according to claim 16, wherein, The encoding of the video sequence further includes: The sequence-level MIOU is calculated based on the encoding configuration used to encode the video sequence.

18. The method according to claim 16, wherein, The encoding of the video sequence further includes: In response to a random access RA coding configuration, multiple selected frames are identified to calculate the multiple frame-level MIOUs, wherein the frame pairs correspond to multiple interval levels; and The sequence-level MIOU is calculated based on the weighted sum of the multiple frame-level MIOUs.

19. The method according to claim 18, wherein, If the first interval level of the first frame pair is greater than the second interval level of the second frame pair, the first weight of the first frame level MIOU of the first frame pair is greater than the second weight of the second frame level MIOU of the second frame pair.

20. The method of claim 16, wherein, The encoding of the video sequence further includes: In response to a low-latency LD coding configuration, multiple selected frames are identified at fixed intervals to calculate the multiple frame-level MIOUs, wherein each frame pair includes two adjacent selected frames; and The sequence-level MIOU is calculated by averaging the multiple frame-level MIOUs.

Images