Systems and methods for content adaptive multi-scale feature layer filtering

Abstract

A system and method are provided for encoding and decoding video for machine consumption, wherein bandwidth is reduced by filtering feature layers determined to be redundant or less relevant at an encoder site. The video encoder includes a neural network front end that receives image data and generates multiple feature layers. A layer context processor and a redundancy layer identifyer work together to evaluate the context of objects in the image data and determine the relevance of multiple feature layers at the decoder site to the machine task. The layer filter then performs at least one of the following to generate a filtered set of layers: removing redundant layers and scaling layers identified as less relevant to the machine task.

Description

[0001] Related application statement

[0002] This application claims priority to U.S. Provisional Application No. 63 / 537,927, filed on September 12, 2023, entitled “Content Adaptive Multi-Scale Feature Layer Filtering”, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This application generally relates to the field of video encoding and decoding. Specifically, the present invention relates to systems and methods for reducing bandwidth in machine-oriented video encoding systems by selectively removing or modifying selected feature layers from the encoded bitstream. Background Technology

[0004] As the number and scale of deployed video sensors / devices increase, it is expected that more and more video will be processed by machines. Systems or solutions employing thousands of cameras generate massive amounts of video that cannot be monitored cost-effectively by humans. Machine or computing systems capable of capturing and analyzing video offer an effective solution for the implementation of decision support systems and analytics engines. Machines designed to perform analytical tasks are not as sensitive to video quality and resolution as human operators. Video Coding for Machines (VCM) seizes this opportunity to transform and represent video to minimize the computation, storage, and streaming of video data while ensuring that machine tasks are performed with high efficiency.

[0005] Feature Coding based Video Coding for Machines (FCVCM) is based on the observation that video analysis based on convolutional neural networks (CNNs) is the most prominent solution, but it requires a large amount of computational resources. The FCVCM method attempts to compress and transmit video features extracted from CNNs, and then use the decompressed features on the receiver side to continue processing on the CNN performing the machine task.

[0006] Figure 1This diagram illustrates a typical CNN used for object detection. A CNN trained using training data such as video, images, audio, LiDAR, thermal imaging, or even text is used to perform the task and provide information about the data input to the network. The input image is first processed by a Feature Pyramid Network (FPN) 105, which generates feature maps at layers P2, P3, P4, and P5. The feature maps from layers P2, P3, P4, and P5 are further processed by a Region Proposal Network (RPN) 110 and a Box Head 115. The Box Head 115 may include a Fast R-CNN (ConvFCHead) 120 and a second Fast R-CNN for the output layer 125, which determines the region of interest (ROI) and the detected label for the ROI. In this architecture, FPN 105, RPN 110, and Box Head 115 represent computationally complex processes.

[0007] By splitting the computations performed in a CNN between the transmitter / camera and receiver sides, system complexity and data usage can be reduced. In this architecture, the camera or video source side of the system consists of the first part or front end of the CNN, such as FPN 105. The output of FPN, feature maps P2, P3, P4, and P5, are then compressed and transmitted to the receiver. The decompressed feature maps received at the receiver are then fed into the second part of the network, which includes components such as RPN 110 and Box Head 115, to complete the network execution and produce the network output. Such CNNs can be trained to perform tasks such as object detection, segmentation, action detection, and object tracking.

[0008] Figure 2 This illustrates a typical approach to handling such split networks. Part 1 of the neural network, 205, represents a camera / video source where the input video or image is processed by Part 1 to produce multiple feature maps, such as four feature maps P2, P3, P4, and P5. The number and structure of the feature maps depend on the neural network. For example, some networks may only have three feature maps instead of four. The size or dimension of each feature map also depends on the neural network. The feature maps are then packed and quantized to prepare them for compression.

[0009] NN Part 1

[0010] Figure 3Part 1 of the NN section, 305, further explains this. In this stage, features are extracted from the neural network (e.g., Res-Net in FPN 105). These features can be composed of four layers: P2 (310a), P3 (310b), P4 (310c), and P5 (310d), each with 256 channels. Each channel in each layer represents a convolution with a kernel and represents a feature of the input image. Typically, all channels in a given layer have the same dimension. For example, as... Figure 4 As shown, in layer P2, each channel has two dimensions, P2cW and P2cH. These two dimensions represent the width and height of the channel, depending on the input image and the neural network. Similarly, the channel dimensions of the other three layers in this example network can be represented by the corresponding channel widths and heights as P3cW x P3cH, P4cW x P4cH, and P5cW x P5cH.

[0011] In the case of Feature Pyramid Network 105, due to information loss caused by the pooling function of the neural network (which reduces the spatial size of information to a more abstract representation), the size of each layer is smaller than the previous layer. For example, the size of each layer may be half that of the previous layer, where p2 is the largest layer and p5 is the smallest layer.

[0012] The pixels of the image input to the network are passed through the network. The extracted feature maps are the transformed input pixels in each channel of each feature map. The features in each extracted channel are essentially floating-point values ​​representing the features of the input image. Therefore, compressing or encoding feature maps requires compressing all values ​​representing the features of the input image.

[0013] Packing (Tiling) & 10-bit uniform quantization

[0014] Return to reference Figure 2 Packing 210 and quantization 215 are performed to convert the floating-point feature values ​​of the neural network into an image representation and to enhance the data used for the encoder, for example, by using 10-bit quantization. However, it is understood that other quantization levels can also be used.

[0015] For the packing and tiling stage 210, features are converted to an image format by mapping channel data to pixels and assembling them layer by layer to form a complete image. For example, consider a P2 layer with 256 channels, each with a width and height dimension of P2cW x P2cH. Figure 4 As shown, channel data in this type of layer can be packaged into tiles of size P2cW x P2cH, which are laid out as a rectangular grid.

[0016] The feature map data from each of these 256 channels can be tiled into a grid of N x M pieces, each piece being P2cW x P2cH in size. This yields the equivalent feature map for layer P2, with a width of P2W = P2cW x N and a height of P2H = P2cH x M. The original floating-point feature map data for layer P2 has a dimension of P2w x P2H. Normalizing this two-dimensional feature map data to a 10-bit representation yields the 10-bit image corresponding to layer P2. Similarly, tiling and normalization produce the 10-bit layer images for layers P3, P4, and P5. This normalization is also called uniform 10-bit quantization because the floating-point feature map data is mapped to 1024 levels in the 10-bit representation of that data.

[0017] Taking the R-CNN neural network as an example, the size of each channel in layer P3 is half the size of each channel in layer P2 in every dimension. Similarly, the channel dimension in P4 is half that of P3, and the channel dimension in P5 is half that of P4. Therefore, the amount of data in layers and channels in P5 is significantly less than that in P2.

[0018] P2 layer:

[0019] Width, P2W = P2cW x N, and

[0020] Height, P2H = P2cH x M

[0021] M x N = 256;

[0022] P3 layer:

[0023] P3cW = P2cW / 2; P3cH = P2cH / 2;

[0024] Width, P3W = P3cW x 2N, and

[0025] Height, P3H = P3cH x M / 2

[0026] P4 layer:

[0027] P4cW=P3cW / 2=P2cW / 4; P4cH=P3cH / 2=P2cH / 4;

[0028] Width, P4W = P4cW x 4N, and

[0029] Height, P4H = P4cH x M / 4

[0030] P5 layer:

[0031] P5cW=P4cW / 2=P2cW / 8; P5cH=P4cH / 2=P2cH / 8;

[0032] Width, P5W = P5cW x 8N, and

[0033] Height, P5H = P5cH x M / 8

[0034] Channel data from all four layers are packaged into a single image and sent to the encoder input. Typically, the channels are tiled so that the channels of layer p2 are arranged in a 16 x 16 grid according to a predetermined channel order. Channels in subsequent layers are tiled to fit the width of the P2 channel arrangement. Figure 5 Examples of tiled channel data from layers P2, P3, P4, and P5 are shown.

[0035] compression

[0036] Continue to refer to Figure 2 The tiled and quantized images are compressed using image and video compression 220 (such as versatile video coding (VVC) or any other image or video compression). The compressed data is then transmitted to the receiver via a communication network.

[0037] Decompress

[0038] The compressed channel image data is received and decompressed using an appropriate decompression system 225 to produce a decoded tiled image. If lossy compression is used, the decoded image may contain distortion due to lossy compression.

[0039] Inverse quantization & unpacking

[0040] The received image is dequantized (denormalized) 230 and converted into floating-point feature map data, which is then unpacked 235 into the corresponding channels of the original network features, which can be fed into the latter part of the neural network. In these operations, the processes in the packing (tiling) 210 and quantization 215 steps are reversed.

[0041] NN Part 2 Processing

[0042] Once the original layer information is reconstructed from the decompressed data, it can be directly fed into the second part of the neural network (e.g., RPN and Box Head in the R-CNN example) NN Part 240, which will output the machine task output (e.g., object detection or segmentation information).

[0043] Machine task performance

[0044] Machine task performance (e.g., the accuracy of object detection or segmentation information) can be affected by the method used to compress feature map data. Lossless compression faithfully reconstructs feature maps and does not affect machine task performance at the receiver. Any lossy compression (e.g., using video or image compression tools) can impact machine task performance. Higher compression levels are expected to introduce greater distortion into the reconstructed feature maps, resulting in a more significant degradation in machine task performance. Summary of the Invention

[0045] In some embodiments of this disclosure, a video encoder is provided in a system for machine-oriented video coding. The video encoder includes a neural network front end that receives image data and generates multiple feature layers. A layer context processor evaluates the context of objects in the image data, which affects the importance of layers of the feature maps to a machine task. A redundancy layer identifyer is coupled to the context processor and determines the relevance of the multiple feature layers to a machine task at a decoder site. A layer filter receives the multiple feature layers from the neural network front end and the output of the redundancy layer identifyer, and performs at least one of the following to generate a filtered set of layers: removing redundant layers and scaling layers identified as having low relevance to the machine task. The encoder receives the filtered set of layers and generates an encoded bitstream of the filtered set of layers.

[0046] In some embodiments, the video encoder provides signaling information in the encoded bitstream that indicates which of a plurality of layers are removed or modified by layer filters. Preferably, the plurality of feature layers from the neural network front end have decreasing sizes. In some embodiments, the neural network front end can be implemented using a feature pyramid network.

[0047] This disclosure also includes embodiments of a decoder in a system for machine-oriented video encoding. The decoder receives an encoded bitstream generated by an encoder, which generates multiple feature layers and selectively filters these feature layers prior to encoding. The bitstream preferably includes the filtered feature layers and signaling information identifying which layers are affected by filtering at the encoder. The decoder decompresses the encoded bitstream, applies the signaling information to the decompressed bitstream, and generates layers that were filtered out at the encoder. The decoded and reconstructed feature layers are applied to a neural network backend, which is preferably trained for a machine task.

[0048] The disclosure also describes a bitstream and a method for transmitting an encoded bitstream for machine-oriented video encoding. These methods include applying a neural network front-end to receive image data and generate multiple feature layers. The method also includes applying a layer context processor to evaluate the context of objects in the image data, which affects the importance of layers of the feature maps to the machine task; and applying a redundancy layer identifyer to apply the output of the context processor and determine the relevance of multiple feature layers to the machine task. A layer filter is applied and at least one of the following is performed to generate a filtered set of layers: removing redundant layers and scaling layers identified as having low relevance to the machine task. An encoder is applied to generate an encoded bitstream of the filtered set of layers for transmission. Preferably, the encoded bitstream includes signaling information indicating which layers among the multiple layers were removed or modified by the layer filter, and, where appropriate, information for reversing the filtering process at the decoder (e.g., scaling information).

[0049] These and other aspects and features of the non-limiting embodiments of the present invention will become apparent to those skilled in the art after reading the following description of specific non-limiting embodiments of the present invention in conjunction with the accompanying drawings. Attached Figure Description

[0050] For the purpose of illustrating the invention, the accompanying drawings show various aspects of one or more embodiments of the invention. However, it should be understood that the invention is not limited to the precise devices and apparatus shown in the drawings, wherein:

[0051] Figure 1 This is a block diagram of a generalized R-CNN known in this field;

[0052] Figure 2 This is a block diagram of the FCVCM system;

[0053] Figure 3 This is a further explanation Figure 2 A block diagram of an example layer in NN Part 1;

[0054] Figure 4 It is a graphical representation of the feature map being tiled according to this disclosure;

[0055] Figure 5 It is a graphical representation of a tiled image based on this disclosure;

[0056] Figure 6 This is a simplified block diagram of an FCVCM system with layered filtering according to the present disclosure;

[0057] Figure 7 This is a simplified block diagram of an FCVCM system with layered filtering according to the present disclosure;

[0058] Figure 8 This is an explanation Figure 7A graph illustrating the performance of exemplary layer removal in the embodiments;

[0059] Figure 9 This is a block diagram illustrating an exemplary embodiment of the video decoder;

[0060] Figure 10 This is a block diagram illustrating an exemplary embodiment of a video encoder; and

[0061] Figure 11 This is a block diagram of a computing system that can be used to implement any one or more methods disclosed herein and any one or more parts thereof.

[0062] The accompanying drawings are not necessarily drawn to scale and may use dashed lines, schematic diagrams, and partial views to illustrate the figures. In some cases, details that are not essential for understanding the embodiments, or that make other details difficult to understand, may have been omitted. Detailed Implementation

[0063] refer to Figure 2 The layers extracted from the NN Part 1 205 of the neural network are packaged 210, quantized 215, compressed 220, and transmitted to the receiver at the decoder station. At the receiver, the layer data is decoded, reconstructed into a format acceptable to the NN Part 2 240, and then input into the NN Part 2 to complete the machine task.

[0064] Multi-scale feature networks use layers of different scales to preserve information at different levels. Generally, larger layers retain more information, while smaller layers (depending on the network architecture) retain less information, but allow the network to process larger and more complex objects by focusing on its main features.

[0065] In some machine learning tasks, processing a given image or video frame may not require all layers. Depending on the frame's content, some layers may be less important or completely redundant for a given task network. In such cases, redundant layers can be removed without impacting machine learning performance. For example, an object processed through multiple feature layers might be reduced to a negligible size for the machine learning task. Since removing layers reduces the amount of layer data (feature maps), the storage and bandwidth requirements of the compressed feature maps can be reduced without adversely affecting machine learning performance. Similarly, less important layers can be compressed at a higher level with a greater compression penalty without causing substantial degradation in machine learning performance.

[0066] refer to Figure 6 By using contextual information from frame preprocessing and an understanding of the applicable task network, the layer size can be reduced, or layers can be completely removed from the system, thereby reducing the information passed to the encoder with almost no negative impact on system performance.

[0067] Neural Networks Part 1, Section 605: In this stage, multi-scale feature layers are extracted from the neural network, as described above. Figure 1 and Figure 2 As stated above.

[0068] Layer activation context based on object size 615: In this processing block, the context that influences the importance of feature maps in a given task network includes frame content, encoding parameters, object type, number of objects of a given type, and object overlap. Such context can be obtained using object detection networks such as YOLO.

[0069] Redundant Layer Identifier (RLI) 620: The RLI 620 algorithm uses the output of the layer activation context 615 based on object size to identify which layers in Part 1 605 of the neural network are most important for a given machine task. RLI 620 can be embedded in a trained machine learning model that takes one or more of object size, count, and type as input and outputs layer importance. For example, importance can be represented by a value between 0 and 100, where 0 means completely redundant and 100 means completely necessary. The RLI function can also be implemented as a set of rules based on the size of objects in a frame. Object size can be described by the pixel area covered by the object's bounding box, or by a combination of pixel area and the smaller of width and height. For example, the presence of smaller objects in a frame will mark larger layers (e.g., P2 and P3) as necessary and smaller layers (e.g., P4 and P5) as redundant. Similarly, the presence of objects larger than a threshold size S2 will cause larger layers to be marked as redundant.

[0070] Example 1:

[0071]

[0072] Example 2:

[0073]

[0074] Layer Filter 610: Layer filter 610 uses the output of RLI 620 to process layers from the original feature layer set n from NN Part 1 605 to identify which layers among the n layers should be removed or reduced. The methods employed by layer filter 610 may include completely removing layers, reducing the layer size, or performing other processing on the layers to reduce the bitstream of a given selection of layers in RLI 620.

[0075] Layer Removal: Layers determined to be completely redundant for the machine task can be completely removed at the transmitter (encoder input). The layer is removed at the encoder input via layer filter 610, and the removed layer identifier is signaled to the receiver, for example, by including the layer identifier parameter in the compressed bitstream. The receiver uses this signaling in the compressed bitstream to replace the removed layer with a default value (e.g., all zeros).

[0076] Scaling Layers: Layers identified as less important but not entirely redundant can be scaled down (reduced layer size / resolution) at the transmitter side (encoder input) via layer filter 610. The scaled-down layers are encoded at the encoder input, and the layer identifier and scaling parameters (e.g., scaling factor and scaler type) are signaled to the receiver, for example, by including parameters in the compressed bitstream. The receiver uses these signaled parameters from the bitstream to rescale the decoded layers to the desired dimension.

[0077] Data Processing and Transmission 625: The output of layer filter 610 is applied to further encoding processing steps, which may involve quantizing, packing, and compressing the processed layer data, roughly as a combination of... Figure 2 The discussion then proceeds as follows: The processed information is then encoded, transmitted, decoded, post-processed, and sent to the second part of the task network, roughly as follows... Figure 2 As shown.

[0078] Neural Networks Part 2 630: with Figure 2 The second part of the NN is similar. The second part of the NN, 630, outputs the machine task output (e.g., object detection or segmentation information) given the machine task.

[0079] Figure 7 This is a simplified block diagram of a system for implementing the proposed layered filtering method. For example... Figure 7 What is described:

[0080] NN Part 1 705: In this stage, features are extracted from a CNN implemented using a Res-Net 101 neural network. These features consist of multiple layers, each with multiple channels. In one embodiment, four layers are used, each with 256 channels.

[0081] Redundancy layer identifier implementation: In this embodiment, RLL 720 employs an algorithm that utilizes object size-based layer activation context 730 to identify which layers are most important for a given machine task. In this case, the context implies that the task network is aware of activation layers of certain object sizes. The redundancy layer identifier combines this context with object detection 725, which can be implemented, for example, using a lightweight detector such as YOLOv3 object detection, to mark layers as redundant.

[0082] Layer filter 740: As described above, layer filter 740 processes the data used by the encoder. Filtering methods may include complete layer removal, resizing, or other processing to reduce the bitstream from a given selected layer of the redundant layer recognizer 720. In this case, data in the image data (potentially a packed and quantized YUV file) will be completely removed.

[0083] Compression 745: Once all processing is complete, the image can be encoded. In this case, encoding can be performed using known compression methods, such as VTM-12.0 implemented using VVC video compression.

[0084] Decompression: After the receiving device acquires the encoded image containing network features, it decompresses it by 750 and then decodes these features using a compatible decoder (such as the VTM-12.0 implementation of the VVC decoder).

[0085] Dequantization & Unpacking 755: The received image is dequantized and unpacked into the original network features, which can be fed into the latter part of the neural network. The dequantization and unpacking 755 operation is to reverse the process applied in the packing and quantization operation 710 during encoding.

[0086] NN Part 2 Processing 760: Once the raw layer information is obtained, it can be directly fed into the second part of the neural network NN Part 2, from which object detection and / or segmentation information is output to perform relevant machine tasks.

[0087] Layers of feature maps excluded from the bitstream are preferably notified via signaling within the bitstream so that the decoder can use this information to recreate the missing or modified layers at the receiver. Excluded layers can be implicitly notified via signaling, for example, by notifying included layers via signaling. Therefore, any layers not included in this signaling are not encoded and will be created at the receiver.

[0088] In video, layer selection / inclusion can vary from frame to frame. As the video content changes, some layer information included in the Feature-Coded Video (fc-VCM) bitstream used for machine-oriented video coding may also change. This included layer information may remain constant across a set of consecutive frames in the video. In some applications, layer selection information may not change at all (e.g., a fixed camera monitoring a parking lot to count vehicles or people). A flexible bitstream syntax is preferable informing different possibilities via signaling.

[0089] Frame layer information signaling can be included in the image header to signal the layers in the image; when the layer information is the same for all frames in a group of pictures (GOP), it can be included in the group of pictures (GOP) header; or when the layers remain unchanged for all frames in a video sequence, it can be included in the sequence header or session header. Other methods of signaling layer information are also possible.

[0090] Example bitstream syntax for using signaling to notify layer information:

[0091]

[0092] fli_num_layers: Number of layers per frame

[0093] fli_layer_id: The layer ID. Using a 3-bit value, up to 8 layers can be notified via signaling.

[0094] fli_layer_included: A flag indicating whether the feature map data of a specific layer is included in the bitstream.

[0095] `fli_layer_fill_method`: The method used to fill missing layers at the receiver. A value of 0 indicates that all feature maps not included in the bitstream must be set to 0 (zero-fill layer). When `fli_layer_fill_method` is set to 1, the average value of the next larger layer is used for filling. When `fli_layer_fill_method` is set to 2, the average value of the next smaller layer is used for filling.

[0096] `const_fill_value`: If `fli_layer_fill_method` is 3, a constant value is included to fill missing layer data. This value is expressed in the same units as the encoded feature map data. If the encoded layer data is a 10-bit quantized value, the constant padding is also encoded as a 10-bit quantized value.

[0097] The bitstream syntax shown is for illustrative purposes only; the above functions can also be used with different bitstream syntaxes for signaling notification.

[0098] against Figure 7 An illustration of exemplary results from the system is shown in Figure 8 As shown in the diagram.

[0099] Figure 7The system described is an exemplary implementation using layer removal and layer reduction. The system preferably uses a lightweight detector 725 (e.g., YOLOv3) to measure the relative size of objects. This information is passed through an algorithm that uses network context regarding the layer importance of object size to make decisions about which layers to mark as redundant. Preferably, redundant layers are removed from the bitstream. In this regard, the system performance was evaluated using the openImages5k dataset, such as... Figure 8 As shown.

[0100] The disclosed method can generate bitstreams that conform to known video standards (such as VVC) and can be decoded by VVC-compliant decoders. The disclosed method can also be used to reduce the complexity of other video coding standards (such as HEVC and AV1).

[0101] Figure 9 This is a system block diagram illustrating an example of a decoder 900 capable of implementing this method. Decoder 900 may include an entropy decoder processor 904, an inverse quantization and inverse transform processor 908, a deblocking filter 912, a frame buffer 916, a motion compensation processor 920, and / or an intra-frame prediction processor 924.

[0102] During operation, still refer to Figure 9 The bitstream 928 can be received by the decoder 900 and input to the entropy decoder processor 904, which can decode a portion of the bitstream's entropy into quantization coefficients. The quantization coefficients can be provided to the inverse quantization and inverse transform processor 908, which can perform inverse quantization and inverse transform to create a residual signal. Depending on the processing mode, this residual signal can be added to the output of the motion compensation processor 920 or the intra-frame prediction processor 924. The outputs of the motion compensation processor 920 and the intra-frame prediction processor 924 can include block predictions based on previously decoded blocks. The sum of the predictions and residuals can be processed by the deblocking filter 912 and stored in the frame buffer 916.

[0103] In one embodiment, still referencing Figure 9The decoder 900 may include circuitry configured to perform any of the operations described in any of the above embodiments in any order and with any degree of repetition. For example, the decoder 900 may be configured to repeatedly execute a single step or sequence of steps until a desired or commanded result is achieved; may iteratively and / or recursively repeat a step or a series of steps, using the output of a previously repeated step as input for a subsequent repeated step, aggregating repeated inputs and / or outputs to produce an aggregated result, reducing or decrementing one or more variables (e.g., global variables), and / or decomposing a large processing task into a set of smaller processing tasks that are iteratively addressed. The decoder may execute any step or sequence of steps described in this disclosure in parallel, for example, by using two or more parallel threads, processor cores, etc., to execute a step two or more times simultaneously and / or substantially simultaneously; the partitioning of tasks among parallel threads and / or processes may be performed according to any protocol suitable for partitioning tasks between iterations. Those skilled in the art, upon reading the full contents of this disclosure, will appreciate various methods for using iterative, recursive, and / or parallel processing to subdivide, share, or otherwise process steps, sequences of steps, processing tasks, and / or data.

[0104] Figure 10 This is a system block diagram illustrating an example video encoder 1000 capable of adaptive cropping. The example video encoder 1000 can receive input video 1004, which can be initially segmented or partitioned according to a processing scheme (e.g., a tree-structured macroblock partitioning scheme, such as a quadtree plus a binary tree). Examples of tree-structured macroblock partitioning schemes might involve dividing image frames into large blocks called coding tree units (CTUs). In some implementations, each CTU can be further divided once or multiple times to form several sub-blocks called coding units (CUs). The final result of this partitioning may include a set of sub-blocks, which can be called predictive units (PUs). Transform units (TUs) can also be used.

[0105] Still referencing Figure 10 The example video encoder 1000 may include an intra-frame prediction processor 1008, a motion estimation / compensation processor 1012 (also known as an inter-frame prediction processor, capable of constructing a motion vector candidate list, including adding global motion vector candidates to the motion vector candidate list), a transform / quantization processor 1016, an inverse quantization / inverse transform processor 1020, a loop filter 1024, a decoded image buffer 1028, and / or an entropy coding processor 1032. Bitstream parameters may be input to the entropy coding processor 1032 for inclusion in the output bitstream 1036.

[0106] During operation, and continue to refer to Figure 10 For each block of the input video frame, it can be determined whether the block will be processed via intra-frame prediction or motion estimation / compensation. The block can be provided to either the intra-frame prediction processor 1008 or the motion estimation / compensation processor 1012. If the block is to be processed via intra-frame prediction, the intra-frame prediction processor 1008 can perform processing to output predicted values. If the block is to be processed via motion estimation / compensation, the motion estimation / compensation processor 1012 can perform processing including constructing a list of motion vector candidates and, if applicable, adding global motion vector candidates to the list of motion vector candidates.

[0107] Further reference Figure 10 The residual can be formed by subtracting the predicted value from the input video. The residual can be received by a transform / quantization processor 1016, which performs transform processing (e.g., discrete cosine transform (DCT)) to produce coefficients that can be quantized. The quantization coefficients and any associated signaling information can be provided to an entropy coding processor 1032 for entropy coding and inclusion in the output bitstream 1036. The entropy coding processor 1032 can support encoding signaling information associated with encoding the current block. Furthermore, the quantization coefficients can be provided to an inverse quantization / inverse transform processor 1020, which reconstructs pixels that can be combined with the predicted value and processed by a loop filter 1024, the output of which can be stored in a decoded image buffer 1028 for use by a motion estimation / compensation processor 1012, which is capable of constructing a motion vector candidate list, including adding global motion vector candidates to the motion vector candidate list.

[0108] Continue to refer to Figure 10 While some changes have been described in detail above, other modifications or additions are possible. For example, in some implementations, the current block may include any symmetric block (8 x 8, 16 x 16, 32 x 32, 64 x 64, 128 x 128, etc.) as well as any asymmetric block (8 x 4, 16 x 8, etc.).

[0109] In some implementations, the reference is still made. Figure 10 This allows for the implementation of a quadtree plus binary decision tree (QTBT). In QTBT, at the coding tree unit level, the partitioning parameters can be dynamically derived to adapt to local characteristics without any transmission overhead. Subsequently, at the coding unit level, the joint classifier decision tree structure eliminates unnecessary iterations and controls the risk of incorrect predictions. In some implementations, the LTR frame block update mode can be provided as an additional option available at each leaf node of the QTBT.

[0110] In some implementations, the reference is still made. Figure 10 Additional syntax elements can be signaled at different levels of the bitstream. For example, an enabled flag can be included for the entire sequence by including an encoded enable flag in the Sequence Parameter Set (SPS). Furthermore, CTU flags can be encoded at the Code Tree Unit (CTU) level.

[0111] Some embodiments may include non-transitory computer program products (i.e. physically embodied computer program products) that store instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform the operations described herein.

[0112] Still referencing Figure 10 The encoder 1000 may include circuitry configured to perform any of the operations described above in any embodiment, in any order, and with any degree of repetition. For example, the encoder 1000 may be configured to repeatedly execute a single step or sequence until a desired or commanded result is achieved; it may iteratively and / or recursively repeat a step or a series of steps, using the output of a previously repeated step as input for a subsequent repetition, aggregating repeated inputs and / or outputs to produce an aggregated result, reducing or diminishing one or more variables (e.g., global variables), and / or decomposing a large processing task into a set of smaller processing tasks processed iteratively. The encoder 1000 may execute any step or sequence of steps described in this disclosure in parallel, for example, by using two or more parallel threads, processor cores, etc., to execute a step two or more times simultaneously and / or substantially simultaneously; the partitioning of tasks among parallel threads and / or processes may be performed according to any protocol suitable for partitioning tasks between iterations. Those skilled in the art, upon reading the full contents of this disclosure, will appreciate various methods for using iteration, recursion, and / or parallel processing to subdivide, share, or otherwise process steps, sequences of steps, processing tasks, and / or data.

[0113] Continue to refer to Figure 10Non-transitory computer program products (i.e., physically embodied computer program products) may store instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform the operations and / or steps described herein, including but not limited to any of the foregoing operations and / or any operations that decoder 900 and / or encoder 1000 may be configured to perform. Similarly, computer systems are also described, which may include one or more data processors and memory coupled to said one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. Furthermore, the methods may be implemented by one or more data processors, either within a single computing system or distributed between two or more computing systems. Such computing systems may be connected via one or more connections and exchange data and / or commands or other instructions, including connections via networks (e.g., the Internet, wireless wide area networks, local area networks, wide area networks, wired networks, etc.), direct connections between one or more computing systems, etc.

[0114] It should be noted that any one or more aspects and embodiments described herein can be readily implemented using one or more machines (e.g., one or more computing devices used as user computing devices for electronic documents, one or more server devices, such as document servers, etc.) programmed according to the teachings of this specification, as will be apparent to those skilled in the art of computer science. Based on the teachings of this disclosure, skilled programmers can easily prepare suitable software code, as will be apparent to those skilled in the art of software science. The various aspects and implementations of the software and / or software modules discussed above may also include suitable hardware to assist in implementing the machine-executable instructions of the software and / or software modules.

[0115] Such software may be a computer program product employing a machine-readable storage medium. A machine-readable storage medium can be any medium capable of storing and / or encoding a series of instructions for execution by a machine (e.g., a computing device) and enabling the machine to perform any of the methods and / or embodiments described herein. Examples of machine-readable storage media include, but are not limited to: magnetic disks, optical disks (e.g., CDs, CD-Rs, DVDs, DVD-Rs, etc.), magneto-optical disks, read-only memory "ROM" devices, random access memory "RAM" devices, magnetic cards, optical cards, solid-state storage devices, EPROMs, EEPROMs, and any combinations thereof. The term "machine-readable storage medium" as used herein is intended to include both single media and collections of physically separate media, such as a batch of optical disks or a combination of one or more hard disk drives and computer memory. The term "machine-readable storage medium" as used herein does not include transient signal transmissions.

[0116] Such software may also include information (e.g., data) carried as data signals on a data carrier (e.g., a carrier wave). For example, machine-executable information may be included as data-carrying signals embodied in a data carrier, which encode a series of instructions or a portion thereof for execution by a machine (e.g., a computing device), as well as any related information (e.g., data structures and data) that causes the machine to perform any of the methods and / or embodiments described herein.

[0117] Examples of computing devices include, but are not limited to: e-book reading devices, computer workstations, terminal computers, server computers, handheld devices (such as tablets, smartphones, etc.), network devices, network routers, network switches, network bridges, any machine capable of executing a set of instructions specifying the actions to be taken by the machine, and any combination thereof. In one example, a computing device may include, or be included in, a self-service terminal.

[0118] Figure 11 An illustration of one embodiment of a computing device is shown, exemplarily represented as a computer system 1100, wherein a set of instructions can be executed to cause a control system to perform any one or more aspects and / or methods of this disclosure. It is also contemplated to utilize multiple computing devices to implement specially configured instruction sets to cause one or more devices to perform one or more aspects and / or methods of this disclosure. The computer system 1100 includes a processor 1104 and a memory 1108, which communicate with each other and with other components via a bus 1112. The bus 1112 may include any of a variety of bus architectures, including but not limited to a memory bus, a memory controller, a peripheral bus, a local bus, and any combination thereof, and may use any of a variety of bus architectures.

[0119] Processor 1104 may include any suitable processor, such as, but not limited to, a processor containing logic circuitry for performing arithmetic and logical operations, such as an arithmetic and logic unit (ALU), which may be regulated by a state machine and guided by operational inputs from memory and / or sensors; as a non-limiting example, processor 1104 may be organized according to a von Neumann architecture and / or a Harvard architecture. Processor 1104 may include, be incorporated into, and / or be incorporated into (but is not limited to) a microcontroller, microprocessor, digital signal processor (DSP), field programmable gate array (FPGA), complex programmable logic device (CPLD), graphics processing unit (GPU), general-purpose GPU, tensor processing unit (TPU), analog or mixed signal processor, trusted platform module (TPM), floating-point unit (FPU), and / or system on a chip (SoC).

[0120] Memory 1108 may include various components (e.g., machine-readable media), including but not limited to random access memory components, read-only components, and any combination thereof. In one example, a basic input / output system 1116 (BIOS), including basic routines (e.g., during startup) that facilitate the transfer of information between elements within computer system 1100, may be stored in memory 1108. Memory 1108 may also include (e.g., stored on one or more machine-readable media) instructions (e.g., software) 1120 embodying any one or more aspects and / or methods of this disclosure. In another example, memory 1108 may also include any number of program modules, including but not limited to an operating system, one or more application programs, other program modules, program data, and any combination thereof.

[0121] Computer system 1100 may also include storage device 1124. Examples of storage devices (e.g., storage device 1124) include, but are not limited to, hard disk drives, disk drives, optical disc drives combined with optical media, solid-state storage devices, and any combination thereof. Storage device 1124 may be connected to bus 1112 via a suitable interface (not shown). Example interfaces include, but are not limited to, SCSI, advanced technology attachment (ATA), serial ATA, universal serial bus (USB), IEEE 1394 (FIREWIRE), and any combination thereof. In one example, storage device 1124 (or one or more components thereof) may interact with computer system 1100 detachably (e.g., via an external port connector (not shown)). Specifically, storage device 1124 and associated machine-readable medium 1128 may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for computer system 1100. In one example, software 1120 may reside wholly or partially on machine-readable medium 1128. In another example, software 1120 may reside entirely or partially in processor 1104.

[0122] Computer system 1100 may also include input device 1132. In one example, a user of computer system 1100 may input commands and / or other information to computer system 1100 via input device 1132. Examples of input device 1132 include, but are not limited to, alphanumeric input devices (e.g., keyboards), pointing devices, joysticks, game controllers, audio input devices (e.g., microphones, voice response systems, etc.), cursor control devices (e.g., mice), touchpads, optical scanners, video capture devices (e.g., still cameras, video cameras), touchscreens, and any combination thereof. Input device 1132 may interact with bus 1112 via any of a variety of interfaces (not shown), including but not limited to serial interfaces, parallel interfaces, game ports, USB interfaces, firewire interfaces, direct interfaces to bus 1112, and any combination thereof. Input device 1132 may include a touchscreen interface, which may be part of or separate from display 1136, as will be discussed further below. Input device 1132 may be used as a user selection device for selecting one or more graphical representations in a graphical interface, as described above.

[0123] Users can also input commands and / or other information to computer system 1100 via storage device 1124 (e.g., removable disk drive, flash drive, etc.) and / or network interface device 1140. Network interface devices (e.g., network interface device 1140) can be used to connect computer system 1100 to one or more networks (e.g., network 1144) and one or more remote devices 1148 connected thereto. Examples of network interface devices include, but are not limited to, network interface cards (e.g., mobile network interface cards, LAN cards), modems, and any combination thereof. Examples of networks include, but are not limited to, wide area networks (e.g., the Internet, corporate networks), local area networks (e.g., networks associated with offices, buildings, campuses, or other relatively small geographical spaces), telephone networks, data networks associated with telephone / voice providers (e.g., data and / or voice networks of mobile communication providers), direct connections between two computing devices, and any combination thereof. Networks like network 1144 can employ wired and / or wireless communication modes. Generally, any network topology can be used. Information (such as data, software 1120, etc.) can communicate with and / or be communicated from computer system 1100 via network interface device 1140.

[0124] Computer system 1100 may also include a video display adapter 1152 for transmitting displayable images to a display device (e.g., display device 1136). Examples of display devices include, but are not limited to, liquid crystal displays (LCDs), cathode ray tubes (CRTs), plasma displays, light-emitting diode (LED) displays, and any combination thereof. Display adapter 1152 and display device 1136 may be used in conjunction with processor 1104 to provide graphical representations of various aspects of this disclosure. In addition to display devices, computer system 1100 may also include one or more other peripheral output devices, including but not limited to audio speakers, printers, and any combinations thereof. Such peripheral output devices may be connected to bus 1112 via peripheral interface 1156. Examples of peripheral interfaces include, but are not limited to, serial ports, USB connections, firewire connections, parallel connections, and any combinations thereof.

[0125] The foregoing is a detailed description of illustrative embodiments of the present invention. Various modifications and additions can be made without departing from the spirit and scope of the invention. Features of each of the various embodiments described above can be suitably combined with features of other described embodiments to provide multiple combinations of features in related new embodiments. Furthermore, while many individual embodiments have been described above, the description herein is merely an illustration of the application of the principles of the invention. Moreover, although specific methods herein may be described and / or described as being performed in a specific order, the order of execution can be flexibly varied by those skilled in the art to achieve the methods, systems, and software according to this disclosure. Therefore, this description is intended to be understood as illustrative only and should not otherwise limit the scope of the invention.

[0126] Exemplary embodiments have been disclosed above and illustrated in the accompanying drawings. Those skilled in the art will understand that various changes, omissions, and additions can be made to the specific disclosure herein without departing from the spirit and scope of the invention.

Claims

1. A video encoder for a machine-oriented video encoding system, comprising: A neural network front-end, which receives image data and generates multiple feature layers; A layer context processor evaluates the context of objects in the image data, which affects the importance of layers of the feature map for machine tasks; A redundancy layer identifier applies the output of the context processor and determines the relevance of the plurality of feature layers to the machine task; A layer filter receives the output of the plurality of feature layers from the front end of the neural network and the redundant layer identifier, and performs at least one of the following to generate a filtered set of layers: removing redundant layers and scaling layers identified as having low relevance to the machine task. An encoder that receives the filtered layer set and generates an encoded bitstream of the filtered layer set.

2. The video encoder of claim 1 further includes signaling information in the encoded bitstream, the signaling information indicating which of the plurality of layers are removed or modified by the layer filter.

3. The video encoder according to claim 1, wherein, Multiple layers have decreasing layer sizes.

4. The video encoder according to claim 1, wherein, The neural network front end includes a feature pyramid network.

5. The video encoder according to claim 1, wherein, The redundancy layer identifier also includes a lightweight object detector.

6. A decoder in a system for machine-oriented video encoding, comprising circuitry configured to: Receive an encoded bitstream generated by an encoder, the encoder generating multiple feature layers and selectively filtering the multiple feature layers before encoding, the bitstream including the filtered feature layers and signaling information identifying which layers are affected by the filtering; Decompress the encoded bitstream; The signaling information is applied to the decompressed bitstream, and a layer that is filtered out at the encoder is generated. The reconstructed feature layers are applied to neural networks trained for machine tasks.

7. The decoder according to claim 6, wherein, The signaling information explicitly informs which layers were removed during encoding.

8. The decoder according to claim 6, wherein, The signaling information implicitly informs which layers were removed during encoding.

9. A method for transmitting a encoded bitstream for machine-oriented video encoding, comprising: A neural network front-end is applied, which receives image data and generates multiple feature layers; An application-layer context processor evaluates the context of objects in the image data, which affects the importance of layers of the feature map for machine tasks; A redundancy layer identifier is applied, which uses the output of the context processor and determines the relevance of the plurality of feature layers to the machine task; An application layer filter is used, which receives the outputs from the plurality of feature layers from the front end of the neural network and the redundant layer identifier, and performs at least one of the following to generate a filtered set of layers: removing redundant layers and scaling layers identified as having low relevance to the machine task. An encoder is applied that receives the filtered layer set and generates an encoded bit stream of the filtered layer set for transmission.

10. The method according to claim 9, wherein, The encoded bitstream also includes signaling information in the encoded bitstream, which indicates which of the multiple layers are removed or modified by the layer filter.

11. The method according to claim 10, wherein, The bitstream explicitly notifies via signaling which layers are removed by the layer filter.

12. The method according to claim 10, wherein, The bitstream implicitly uses signaling to notify which layers are removed by the layer filter.

Images