Systems and methods for object-based redundancy processing in video coding for machine-based applications

Abstract

Systems and methods of video encoding for machine-based applications are provided which include redundancy reduction processing on a resampled video signal. The redundancy reduction processing generally includes a first processing operation to determine pixelwise information ranking, and a second processing operation to perform pixelwise information reduction. Pixels that are determined to be of less importance can be eliminated or transformed prior to encoding to reduce the encoding overhead and improve encoding performance.

Description

SYSTEMS AND METHODS FOR OBJECT-BASED REDUNDANCY PROCESSING IN VIDEO CODING FOR MACHINE-BASED APPLICATIONSCross Reference to Related Applications

[0001] This application claims the benefit of priority to U.S. provisional application serial number 63 / 668,339, entitled Systems and Methods for Object-based Redundancy Removal for Video Coding for Machines, filed on July 8, 2024, the disclosure of which is hereby incorporated herein by reference.Field of the Disclosure

[0002] The present invention generally relates to the field of video encoding and decoding. In particular, the present disclosure is directed to video coding and decoding of video for machines.Background of the Disclosure

[0003] In recent times, a significant portion of all the images and videos that are recorded in the field are consumed by machines only, without ever reaching human eyes. Those machines process images and videos with the goal of completing tasks such as object detection, object tracking, segmentation, event detection etc. Recognizing that this trend is prevalent and will only accelerate in the future, international standardization bodies established efforts to standardize image and video coding that is primarily optimized for machine consumption. For example, standards like JPEG Al and Video Coding for Machines are initiated in addition to already established standards such as Compact Descriptors for Visual Search, and Compact Descriptors for Video Analytics. Solutions that improve efficiency compared to the classical image and video coding techniques are needed.

[0004] Video Coding for Machines (VCM) is the process of compressing image / video information for machine consumption. As used herein, VCM is not limited to any particular protocol or standard and is intended to broadly convey compression and decompression of data for machine consumption. Machine consumption is the process of machines consuming information, in this case in the form of images / video. This can include object detection and segmentation tasks. Current VCM systems typically follow the following architecture: video / images are captured by a digital camera or recording device, video / image information is compressed using a device or software called a codec (compression-decompression). The compressed image / video information is sent to a receiving device where the image / video information is decompressed and interpreted by a machine. The compression can be performed using traditional block-based video encoders such as versatilevideo coding (VVC), neural -network based compression, or hybrid of traditional coding and neural -network based compression.

[0005] In systems which have a many-to-one relationship between the encoder and decoder sites, such as machine-based systems with multiple cameras, conventional approaches to video encoding may require massive video transmission requirements. The use of conventional encoding in these applications may take significant time for efficient and fast real-time analysis and decision-making. In certain embodiments, a VCM approach may resolve this problem by both encoding video and extracting some features at a transmitter site and then transmitting a resultant encoded bit stream to a VCM decoder. At a decoder site, a hybrid system may be employed in which video may be decoded for human vision and features may be decoded for machines.

[0006] A “feature,” as used in this disclosure, is a specific structural and / or content attribute of data. Examples of features may include SIFT, audio features, color hist, motion hist, speech level, loudness level, or the like. Features may be time stamped. Each feature may be associated with a single frame of a group of frames. Features may include high level content features such as timestamps, labels for persons and objects in the video, coordinates for objects and / or regions-of- interest, frame masks for region-based quantization, and / or any other feature that may occur to persons skilled in the art upon reviewing the entirety of this disclosure. As a further non-limiting example, features may include features that describe spatial and / or temporal characteristics of a frame or group of frames. Examples of features that describe spatial and / or temporal characteristics may include motion, texture, color, brightness, edge count, blur, blockiness, or the like.

[0007] At a decoder site it will be appreciated that video may be decoded for human vision and features may be decoded for machines. Systems which provide video for human vision and machine consumption are sometimes called hybrid systems. The systems and methods disclosed herein are intended to apply to machine-based and hybrid systems.

[0008] Error! Reference source not found, is a block diagram illustrating the architecture of a known system for encoding and decoding video for machine-based applications, such as VCMRS. In general, frames from decoded videos at the receiver are used as input to the trained neural networks to perform tasks such as object detection, object segmentation, and object tracking. In general, applying compression degrades the quality of the compressed video relative to the source and may impact the performance of machine tasks. Higher compression will be more efficient for transmission but could lead to larger degradation in quality and degradation in machine taskperformance. Methods to improve the task performance without increasing the size of the compressed video will improve VCM applications and services.

[0009] The following is an overview of the components described in the known system of Fig. 1, the details of which are provided in the working group document, "Algorithm description of tools in VCM reference software," ISO / IEC JTC 1 / SC 29 / WG 04, which is incorporated in its entirety herein by reference.

[0010] Referring to Fig. 1, the input video is applied to a temporal resample / restoration module 105. The temporal resample module 105 reduces frame numbers before encoding and restores dropped frames with picture interpolation. The temporal resample module 105 is coupled to a spatial resample module 110 which generates optimal scale factors for input pictures, which is later passed to the inner encoder incorporated with reference picture resampling to generate a bitstream consisting of adaptably down-scaled pictures.

[0011] The spatial resample module 110 is coupled to an ROI-based processing / restoration module 115, which performs region of interest ("ROI") extraction, margin dilation, object region grouping, ROI accumulation, and ROI retargeting. ROI based processing module 115 is coupled to a b / 7 truncation module 120. Since certain machine vision tasks like object tracking and detection exhibit less sensitivity to bit depth, a lower sample value range, represented by fewer bits, may provide better balance bitrate and machine task score. The bit truncation module is coupled to the video encoder 130 of the inner codec 125. The inner codec 125 takes the processed video and encodes a bitstream, such as using a known coding protocol, such as AVI, HEVC, Versatile Video Coding, or other similar encoding standard. Additional information from each of these blocks may be signaled to the decoder in the bitstream, such as header information, or supplemental information. Such information may be signalled, for example and without limitation, in the sequence parameter set (SPS), picture parameter set (PPS), or in supplemental messages, such as SEI.

[0012] At the decoder side, the complementary operations a performed. The inner video decoder 135 receives and decodes the bitstream and provides the decoded bitstream to an ROI restoration module 140, which performs complimentary operations to the ROI processing module 115 at the encoder site. Similarly, spatial restoration module 145 performs complimentary operations to those performed by the spatial resample module 110 and the temporal restoration module 150 performs complimentary operations to those of temporal resample module 105. A bit truncation module 255 is coupled to the temporal restoration module. Despite its name, the bit truncation module 255performs an inverse operation to that of bit truncation module 220 at the encoder to reconstruct the previously truncated bits.Summary of the Disclosure

[0013] Systems and methods for encoding video for machine-based systems are provided which include redundancy reduction processing to transform or eliminate information on pixels that are determined to be of less importance to the machine task being performed at the decoder site.

[0014] A method of encoding a bitstream for machine-based applications with redundancy reduction includes receiving a video signal, performing temporal resampling and spatial resampling on the input video signal, and performing redundancy reduction processing on the resampled video signal. The redundancy reduction processing preferably includes a first processing operation to determine pixelwise information ranking and a second processing operation to perform pixelwise information reduction. Following redundancy reduction processing, performing region of interest processing and encoding the processed video signal to generate an encoded bitstream. In some embodiments, a bit reduction processing operation can be included prior to encoding.

[0015] In some embodiments, the pixelwise information ranking is a binary ranking indicating one of retention or removal. In some embodiments, the pixelwise information ranking may include generating an importance map indicating the relative importance of each pixel. Preferably, the pixelwise information ranking process determines importance based on a specific machine task being performed at a decoder site. In some cases, the importance map may be quantized into a plurality of levels of relative importance. In certain embodiments, the quantization is binary and generates to a bimodal map.

[0016] In certain embodiments, the importance map is generated for each frame in a video. Alternatively, the importance map may be generated for each channel.

[0017] In some embodiments, the pixelwise information reduction processing identifies pixels indicated as being less important and replaces the pixel value with a predetermined constant. In alternate embodiments, the pixelwise information reduction processing identifies pixels indicated as being less important and transforms those pixels by blurring.

[0018] A video encoder may be provided comprising circuitry configured to perform the described methods. A non-transitory computer readable media may be programmed with instructions for a computer processor to perform the methods described herein.

[0019] In some embodiments of the present disclosure, a video encoder is provided with a temporal resampling module receiving an input video signal, a spatial resampling module coupled to the temporal resampling module, and a redundancy reduction module coupled to the spatial resampling module. The redundancy reduction module is configured to perform a first processing operation to determine pixelwise information ranking, and a second processing operation to perform pixelwise information reduction. A region of interest processing module coupled to the redundancy reduction module. A bit truncation module is optionally coupled to the region of interest module. An inner video encoder coupled to the bit truncation module and generating an encoded bitstream.

[0020] In the video encoder, the pixelwise information ranking may provide a binary ranking indicating one of retention or removal of each pixel. In some embodiments, the pixelwise information ranking module may generate an importance map indicating the relative importance of each pixel. In certain embodiments, the relative importance is determined based upon a machine task being performed at a decoder site. This may be performed by a variety of processes. In certain embodiments, the redundancy reduction module includes a neural network trained in accordance with the machine task being performed at a decoder site.

[0021] In certain embodiments of the video encoder, the importance map is quantized into a plurality of levels of relative importance. In some cases, the quantization is binary and generates a bimodal map.

[0022] The importance map may be generated for each frame in a video. In some embodiments, the importance map may be generated for each channel. In some embodiments, an importance map may be shared by a plurality of channels, including up to all channels.

[0023] In some embodiments of the video encoder, pixelwise information reduction processing identifies pixels indicated as being less important and replaces the pixel value of unimportant pixels with a predetermined constant. Alternatively, the pixelwise information reduction processing identifies pixels indicated as being less important and transforms those pixels by blurring.

[0024] Preferable features of the present disclosure include importance maps which introduce s levels of importance to every pixel in a video frame. The levels of importance may be continuous. Importance maps may be discretized using thresholds. Importance may be binarized to reflect pixels to keep or exclude. Importance maps may utilize multiple source frames to reduce temporal discrepancies during residual coding. Importance maps may be optimized for the end task network.Less important pixels may be removed or transformed to reduce the cost of encoding. Removal methods using importance maps may be parameterizable and include masking, blurring, fading,resampling, and quantizing. Preferably, the present systems and methods provide observable BD- Rate improvement over prior methods.Brief Description of the Figures

[0025] For the purpose of illustrating the embodiments of the disclosure, the drawings show aspects of one or more embodiments of the present systems and methods. However, it should be understood that the present invention is not limited to the precise arrangements and instrumentalities shown in the drawings, wherein:

[0026] Fig. l is a block diagram illustrating the architecture of a known VCM encoder-decoder system;

[0027] Fig. 2 is a block diagram illustrating the architecture of a VCM encoder-decoder system in accordance with the present disclosure;

[0028] Fig. 3, comprising Fig. 3A and Fig 3B, illustrates a comparison between a source frame (Fig. 3 A) and an importance map generated in accordance with the present disclosure (Fig. 3B);

[0029] Fig. 4, comprising Fig. 4A and 4B, illustrates a comparison of a decoded frame from a conventional VCM decoder (Fig. 4A) compared to a decoded frame in accordance with the present disclosure in which masking has been applied (Fig. 4B); and

[0030] Fig. 5, comprising Fig. 5A and 5B, illustrates a comparison of a decoded frame from a conventional VCM decoder (Fig. 5A) compared to a decoded frame in accordance with the present disclosure in which blurring has been applied (Fig. 5B).

[0031] The drawings are not necessarily to scale and may be illustrated by phantom lines, diagrammatic representations and fragmentary views. In certain instances, details that are not necessary for an understanding of the embodiments or that render other details difficult to perceive may have been omitted.Detailed Description of Disclosed Embodiments

[0032] Figure 2 illustrates an embodiment of the present disclosure which extends a VCM-like encoder-decoder (CODEC), such as described in connection with Fig. 1 and its derivatives, with a redundancy removal / reduction processing module at the encoder which is preferably split into two phases: pixelwise information ranking, and pixelwise information removal / transformation. Each phase along with suggested implementations are described in greater detail below.

[0033] Fig. 2 is a simplified block diagram of an encoder / decoder system in accordance with the present disclosure. As illustrated in Fig. 2, a redundancy processing module 212 is interposed between the spatial resample module 210 and the ROI processing module 215, which are described above in connection with the known architecture of Fig. 1.

[0034] Referring to Fig. 2, the input video is applied to a temporal resample / restoration module 205. The temporal resample module 205 reduces frame numbers before encoding and restores dropped frames with picture interpolation. The temporal resample module 205 is coupled to a spatial resample module 210 which generates optimal scale factors for input pictures, which are later passed to the inner encoder 230 incorporated with reference picture resampling to generate a bitstream consisting of adaptably down-scaled pictures.

[0035] The spatial resample module 210 is coupled to a redundancy processing module 212. The redundancy processing module 212 performs redundancy removal or reduction processing as described in further detail herein. The redundancy processing module 212 is then coupled to an ROI-based processing / restoration module 215, which performs region of interest ("ROI") extraction, margin dilation, object region grouping, ROI accumulation, and ROI retargeting. ROI based processing module 215 is coupled to a bit truncation module 220. Since certain machine vision tasks, such as object detection and tracking typically exhibit less sensitivity to bit depth, a lower sample value range, represented by fewer bits, may provide a better balance of bitrate and machine task score. The bit truncation module is coupled to the video encoder 230 of the inner codec 225. The inner codec 225 receives the processed video signal and generates an encoded bitstream, such as using a known coding protocol, such as AVI, HEVC, Versatile Video Coding, or other similar encoding standard, for transmission to a decoder.

[0036] Additional information from each of the blocks in the encoder useful for signal reconstruction may be signaled to the decoder in the bitstream, such as header information, or supplemental information. Such information may be signalled, for example and without limitation, in the sequence parameter set (SPS), picture parameter set (PPS), or in supplemental messages, such as SEI. This may include information on resampling, ROI processing, truncation and the like.

[0037] At the decoder side, complementary operations to those performed by the encoder are performed. The inner video decoder 235 receives and decodes the bitstream and provides the decoded bitstream to an ROI restoration module 240, which performs complimentary operations to the ROI processing module 215 at the encoder site. Similarly, spatial restoration module 245 performs complimentary operations to those performed by the spatial resample module 210 and thetemporal restoration module 250 performs complimentary operations to those of temporal resample module 105. A bit truncation module 255 is coupled to the temporal restoration module. Despite its name, the bit truncation module 255 performs an inverse operation to that of bit truncation module 220 at the encoder to reconstruct the previously truncated bits.

[0038] The redundancy processing module 212 preferably performs redundancy removal / reduction processing which is preferably split into two phases: pixelwise information ranking, and pixelwise information removal / reduction.Pixelwise Information Ranking

[0039] The present methods assume that sources of information in a video typically have varying levels of importance to the end machine task. By identifying pixels of little importance to the machine task and removing them or transforming them, bandwidth may be reduced without impacting machine performance. ROI encoder implementations [1] are known to erase frame pixels not included in regions recognized by computer vision methods [3], [4], Object detection methods identify locations of individual objects, typically through bounding boxes. In the context of ROI processing, pixels not located in any bounding box may be deemed less important. The shortcomings with bounding boxes are their imprecision since rectangles cannot capture irregular shapes without keeping extra pixels and occlusion handling. Instance segmentation methods resolve both issues by creating bimodal masks marking object locations per pixel. However, both object detection and instance segmentation only allow binary decision making at the pixel level for ROI - namely whether a pixel is important or unimportant.

[0040] In accordance with the present disclosure, pixel wise importance ranking phase processing may introduce an importance map which enables finer levels of importance with a continuous or discretized interval. It can be represented, for example, in a normalized range as an image with pixel values ranging from 0 (least important) to 1 (most important). Error! Reference source not found.A depicts a source frame with Figure 3B depicting its corresponding importance map in a gray scale, where the brighter pixels inside of objects are deemed more important. An importance map may be created for each frame in a video and for each channel. Alternatively, one importance map may be shared across several or all channels. In the case of channels with different resolutions (for example, YUV 4:2:0), a shared importance map may be resampled to fit each channel.

[0041] Note that the importance map can later be discretized to different levels using a thresholding scheme. In a specific case, performing binary thresholding on the importance map can create bimodal masks with matching semantics to the previously mentioned methods.

[0042] Systems and methods to generate importance maps will typically depend heavily on the end task they are used for. Preferably, these systems and methods take a source frame as input and produce an importance map as an output. Alternatively, systems and methods in accordance with the present disclosure may take multiple reference frames when producing one or more importance maps, in order to minimize temporal discrepancies. Convolutional neural networks can be applied, especially with architectures inspired by object segmentation solutions [5], Systems can additionally optimize importance maps for the end task network, such as by using class activation maps and intermediate feature maps [6], [7], [8],Pixelwise Information Removal / Transform

[0043] Importance maps can be applied to decide which regions in the encoder’s input may either be discarded or reconstructed less faithfully on the decoder. The expected outcome is better compression with minimal end performance degradation. The following provides a non-exhaustive list of techniques to perform pixel removal in accordance with the present systems and methods.

[0044] An importance map may be binarized into two levels significing Kept and Erased. Pixels marked as Kept are preserved from the input without transformation. Pixels marked as Erased are instead replaced with another value. The erased value is arbitrary, though it should be consistent with other erased pixels in the frame. Examples include pre-set constant values across frames or an average of erased pixels or all pixels per frame. Defining the erased value scheme and binary threshold are typically encoding parameters.

[0045] Fig. 4 illustrates a comparison of the exemplary output of the decoder from Fig. 1 (Fig. 4A), as compared to the present method in which the same decoded frame is shown after an importance map with masking is applied in Fig. 4B.Blur

[0046] In some embodiments, rather than remove less important pixels, regions with less importance may be blurred with a parameterized Gaussian convolutional kernel. One option is to binarize the importance map like masking in 3.2.1, and then blur pixels marked Erased with a set kernel size. An alternative option is for importance levels to correlate with kernel size at their location, where lower levels use larger kernels. Blurring reduces texture and high-frequency details without fully eliminating content as is the case in masking.

[0047] Fig. 5 illustrates a comparison of the exemplary output of the decoder from Fig. 1 (Fig. 5A), as compared to the same frame processed by the present method in which an importance map with blurring is applied in Fig. 5B.Other methods for reduction / removal

[0048] Some techniques to remove pixels may perform better than others depending on a range of factors including the input content, encoder implementation, and end task algorithm. For example, each technique produces different residuals during residual coding. Additional alternate techniques for pixel removal may include on or more of the following:• Fade: Interpolate between source pixels and another arbitrary value, parameterized by the importance map.• Resample'. Pixels can be resampled by first down-sampling source pixels and then up- sampling to original resolution, causing degradation. The degree of down-sampling is parameterized by the importance map.• Quantize-. For encoders with quantizer schemes implemented, the importance map can signal regions where higher quantization should be used, by either increasing the quantization step size or using an alternative quantization table.4. Preliminary Results

[0049] The results in Table 1 compare the performance of the known system of Fig. 1 against an exemplary implementation of the present systems and methods employing masking described herein. Evaluation was performed using a Detectron2 object detection network [3] on the SFU- HW-Tracks dataset [9], Even using a naive parameter selection provides improvement across most test cases, and it is expected that further refined implementations may provide further performance improvements.Table 1. Exemplary improvement over conventional systems

[0050] While the description above has discussed the present systems and methods in the context of video coding for machines, these systems and methods are applicable to a broader range of applications. For example, redundancy removal systems can similarly be applied to Feature Coding from Machines ("FCM") codecs. In such a case, information ranking may take place on the source video in the same manner as described for VCM. Afterwards, information removal may take place on either the source video or the feature maps of the first partial network. The former's degradation before feeding through the network may produce feature maps with compressionfriendly semantics. The latter, assuming the feature maps hold spatial relevance to the source video as seen with convolutional neural networks, can resample the importance map to the feature dimensions before applying removal. Alternatively, both information ranking and removal can take place on the feature maps only. Importance maps may either be shared or unique to each feature map.

[0051] Some embodiments may include non-transitory computer program products (i.e., physically embodied computer program products) that store instructions, which when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform operations herein.

[0052] Embodiments may include circuitry configured to implement any operations as described above in any embodiment, in any order and with any degree of repetition. For instance, modules, such as encoder or decoder, may be configured to perform a single step or sequence repeatedly until a desired or commanded outcome is achieved; repetition of a step or a sequence of steps may be performed iteratively and / or recursively using outputs of previous repetitions as inputs to subsequent repetitions, aggregating inputs and / or outputs of repetitions to produce an aggregate result, reduction or decrement of one or more variables such as global variables, and / or division of a larger processing task into a set of iteratively addressed smaller processing tasks. Encoders and decoders described herein may perform any step or sequence of steps as described in this disclosure in parallel, such as simultaneously and / or substantially simultaneously performing a step two or more times using two or more parallel threads, processor cores, or the like; division of tasks between parallel threads and / or processes may be performed according to any protocol suitable for division of tasks between iterations. Persons skilled in the art, upon reviewing the entirety of this disclosure, will be aware of various ways in which steps, sequences of steps, processing tasks, and / or data may be subdivided, shared, or otherwise dealt with using iteration, recursion, and / or parallel processing.

[0053] Non-transitory computer program products (i.e., physically embodied computer program products) may store instructions, which when executed by one or more data processors of one or more computing systems, causes at least one data processor to perform operations, and / or steps thereof described in this disclosure, including without limitation any operations described above and / or any operations decoder and / or encoder may be configured to perform. Similarly, computer systems are also described that may include one or more data processors and memory coupled to the one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. In addition, methods can be implemented by one or more data processors either within a single computing system or distributed among two or more computing systems. Such computing systems can be connected and can exchange data and / or commands or other instructions or the like via one or more connections, including a connection over a network (e.g. the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, or the like), via a direct connection between one or more of the multiple computing systems, or the like.

[0054] It is to be noted that any one or more of the aspects and embodiments described herein may be conveniently implemented using one or more machines (e.g., one or more computing devices that are utilized as a user computing device for an electronic document, one or more server devices, such as a document server, etc.) programmed according to the teachings of the present specification, as will be apparent to those of ordinary skill in the computer art. Appropriate software coding can readily be prepared by skilled programmers based on the teachings of the present disclosure, as will be apparent to those of ordinary skill in the software art. Aspects and implementations discussed above employing software and / or software modules may also include appropriate hardware for assisting in the implementation of the machine executable instructions of the software and / or software module.

[0055] Such software may be a computer program product that employs a machine-readable storage medium. A machine-readable storage medium may be any medium that is capable of storing and / or encoding a sequence of instructions for execution by a machine (e.g., a computing device) and that causes the machine to perform any one of the methodologies and / or embodiments described herein. Examples of a machine-readable storage medium include, but are not limited to, a magnetic disk, an optical disc (e.g., CD, CD-R, DVD, DVD-R, etc.), a magneto-optical disk, a read-only memory “ROM” device, a random-access memory “RAM” device, a magnetic card, an optical card, a solid-state memory device, an EPROM, an EEPROM, and any combinations thereof. A machine-readable medium, as used herein, is intended to include a single medium as well as acollection of physically separate media, such as, for example, a collection of compact discs or one or more hard disk drives in combination with a computer memory. As used herein, a machine- readable storage medium does not include transitory forms of signal transmission.

[0056] Such software may also include information (e.g., data) carried as a data signal on a data carrier, such as a carrier wave. For example, machine-executable information may be included as a data-carrying signal embodied in a data carrier in which the signal encodes a sequence of instruction, or portion thereof, for execution by a machine (e.g., a computing device) and any related information (e.g., data structures and data) that causes the machine to perform any one of the methodologies and / or embodiments described herein.

[0057] Examples of a computing device include, but are not limited to, an electronic book reading device, a computer workstation, a terminal computer, a server computer, a handheld device (e.g., a tablet computer, a smartphone, etc.), a web appliance, a network router, a network switch, a network bridge, any machine capable of executing a sequence of instructions that specify an action to be taken by that machine, and any combinations thereof. In one example, a computing device may include and / or be included in a kiosk.

[0058] The foregoing has been a detailed description of illustrative embodiments of the invention. Various modifications and additions can be made without departing from the spirit and scope of this invention. Features of each of the various embodiments described above may be combined with features of other described embodiments as appropriate in order to provide a multiplicity of feature combinations in associated new embodiments. Furthermore, while the foregoing describes a number of separate embodiments, what has been described herein is merely illustrative of the application of the principles of the present invention. Additionally, although particular methods herein may be illustrated and / or described as being performed in a specific order, the ordering is highly variable within ordinary skill to achieve methods, systems, and software according to the present disclosure. Accordingly, this description is meant to be taken only by way of example, and not to otherwise limit the scope of this invention.

[0059] Exemplary embodiments have been disclosed above and illustrated in the accompanying drawings. It will be understood by those skilled in the art that various changes, omissions and additions may be made to that which is specifically disclosed herein without departing from the spirit and scope of the present invention.ReferencesThe following documents referenced herein are hereby incorporated by reference in their entireties:[1] WG 04 MPEG Video Coding and Jin Young Lee, “[N00466] Algorithm description of tools in VCM reference software.” ISO / IEC JTC 1 / SC 29 / WG 04, Mar. 15, 2024.[2] Jianle Chen, Yan Ye, and Seung Hwan Kim, “Algorithm description for Versatile Video Coding and Test Model 12 (VTM 12).” ISO / IEC JTC 1 / SC 29 / WG 5, Jan. 15, 2021.[3] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick, “Detectron2.” 2019. [Online], Available: https: / / github.com / facebookresearch / detectron2[4] Z. Wang, L. Zheng, Y. Liu, and S. Wang, “Towards Real-Time Multi-Object Tracking,” Eur. Conf. Comput. Vis. ECCV, 2020.[5] C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao, “Y0L0v7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.[6] S. Desai and H. G. Ramaswamy, “Ablation-CAM: Visual Explanations for Deep Convolutional Network via Gradient-free Localization,” in 2020 IEEE Winter Conference on Applications of Computer Vision (WACVf 2020, pp. 972-980. doi: 10.1109 / WACV45572.2020.9093360.[7] M. B. Muhammad and M. Yeasin, “Eigen-cam: Class activation map using principal components,” in 2020 international joint conference on neural networks (IJCNNf IEEE, 2020, pp. 1-7.[8] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: visual explanations from deep networks via gradient-based localization,” I nt. J. Comput. Vis., vol. 128, pp. 336-359, 2020.[9] T. Tanaka, H. Choi, and I. V. Bajic, “SFU-HW-Tracks-vl : Object tracking dataset on raw video sequences,” ArXiv Prepr. ArXiv211214934, 2021.

Claims

What is claimed is:

1. A method of encoding a bitstream for machine-based applications comprising: receiving a video signal; performing temporal resampling on the input video signal; performing spatial resampling of the video signal; performing redundancy reduction processing on the resampled video signal the redundancy reduction processing comprising a first processing operation to determine pixelwise information ranking, and a second processing operation to perform pixelwise information reduction; performing region of interest processing; and encoding the processed video signal to generate an encoded bitstream.

2. The method of claim 1, wherein the pixelwise information ranking comprises a binary ranking indicating one of retention or removal.

3. The method of claim 1, wherein the pixelwise information ranking comprises generating an importance map indicating the relative importance of each pixel.

4. The method of claim 3, wherein the pixelwise information ranking process determines importance based on a specific machine task being performed at a decoder site.

5. The method of claim 4, wherein the importance map is quantized into a plurality of levels of relative importance.

6. The method of claim 5, wherein the quantization is binary and generates to a bimodal map.

7. The method of claim 3, wherein an importance map is generated for each frame in a video.

8. The method of claim 3, wherein an importance map is generated for each channel.

9. The method of claim 3, wherein an importance map is shared by a plurality of channels.

10. The method of any of claims 2-9 wherein the pixelwise information reduction processing identifies pixels indicated as being less important and replaces the pixel value with a predetermined constant.

11. The method of any of claims 2-9 wherein the pixelwise information reduction processing identifies pixels indicated as being less important and transforms those pixels by blurring.

12. A video encoder comprising circuitry configured to perform the method of claims 1-11.

13. A non-transitory computer readable media programmed with instructions for a computer processor to perform the method of claims 1-12.

14. A video encoder comprising: a temporal resampling module receiving an input video signal; a spatial resampling module coupled to the temporal resampling module; a redundancy reduction module coupled to the spatial resampling module; the redundancy reduction module performing a first processing operation to determine pixelwise information ranking, and a second processing operation to perform pixelwise information reduction; a region of interest processing module coupled to the redundancy reduction module; a bit truncation module; and an inner video encoder coupled to the bit truncation module and generating an encoded bitstream.

15. The video encoder of claim 14, wherein the pixelwise information ranking comprises a binary ranking indicating one of retention or removal.

16. The video encoder of claim 14, wherein the pixelwise information ranking comprises generating an importance map indicating the relative importance of each pixel.

17. The video encoder of claim 16, wherein the relative importance is determined based upon a machine task being performed at a decoder site.

18. The video encoder of claim 17, wherein the redundancy reduction module includes a neural network trained in accordance with the machine task being performed at a decoder site.

19. The video encoder of any of claims 16-18, wherein the importance map is quantized into a plurality of levels of relative importance.

20. The video encoder of claim 19, wherein the quantization is binary and generates to a bimodal map.

21. The video encoder of claim 19, wherein an importance map is generated for each frame in a video.

22. The video encoder claim 19, wherein an importance map is generated for each channel.

23. The video encoder of claim 19, wherein an importance map is shared by a plurality of channels.

24. The video encoder of any of claims 14-23 wherein the pixelwise information reduction processing identifies pixels indicated as being less important and replaces the pixel value with a predetermined constant.

25. The video encoder of any of claims 14-23 wherein the pixelwise information reduction processing identifies pixels indicated as being less important and transforms those pixels by blurring.

Images