Systems, methods and bitstreams for decoder-side adaptive reconstruction refinement

Abstract

Systems and methods are provided to improve encoding and decoding of video in a system for feature coding for machines (FCM). At the encoder, a calculate distribution properties process is applied to the feature maps output by a split NN front end to determine at least one probability distribution property of tensors of the feature maps. A user-defined refinement period may be provided at the encoder for use in decoder-side processing. At a decoder, a reconstruction refinement process uses the probability distribution property and the refinement period encoded in a bitstream to align the probability distribution properties of decoded and restored feature maps with the original feature maps signaled in the bitstream. The decoder employs an inverse quantization method which reuses the distribution properties of reconstruction refinements to perform inverse quantization. The refined and restored feature maps are applied to the split neural network back end for inferencing.

Description

SYSTEMS, METHODS AND BITSTREAMS FOR DECODER-SIDE ADAPTIVE RECONSTRUCTION REFINEMENT CROSS REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims the benefit of priority to U.S. Provisional Application serial no.63 / 663,293 filed on June 24, 2024, and entitled “SYSTEMS AND METHODS FOR DECODER-SIDE ADAPTIVE RECONSTRUCGTION REFINEMENT,” the disclosure of which is hereby incorporated by reference in its entirety. BACKGROUND OF THE DISCLOSURE

[0002] As the number and scale of deployed video sensors / devices increases, an increasing amount of video is expected to be processed by machines and for machine use rather than human viewing. Indeed, an exemplary system or solution employing thousands of cameras would produce significant amounts of video that cannot be monitored by humans in a cost- effective manner. Because of the significant volume of the data, an efficient compression system is desirable for Machine-to-Machine (M2M) communication. To standardize the coding for machines to facilitate a Machine-to-Machine (M2M) communication system more efficiently, MPEG issued a Call for Proposal (CfP) for Coding for Machines in 2022.

[0003] Currently, M2M communications are typically managed by two distinct systems, namely, edge computing or local computing systems and remote computing or Video Coding for Machines (VCM) systems. In the context of edge computing systems, the complete Convolutional Neural Network (CNN) is typically executed on edge devices. However, end devices often lack the computational capacity to run extensive networks, requiring billions of operations for inference result calculations. In contrast, in video coding for machine systems, the video is typically compressed initially and transmitted to a base server or cloud server with more computing resources to execute the full CNN. The calculated results are then subsequently relayed back to the edge devices. However, video coding for machine systems may not be able to support compute offload and may fail to fully leverage the potential of end devices, as they execute the complete CNN on the base server. To overcome these limitations for M2M communication and to enable collaborative intelligence and split computing for M2M connections, MPEG has issued a Call for Proposal (CfP) for Feature Coding for Machines (FCM) technology in 2023. As used herein, the term Feature Coding for Machines or FCM is intended to apply generally to systems and methods for feature coding of video and similarinput data for use in transferring information in systems primarily intended for machine consumption and / or machine to machine applications without regard to any one particular standard.

[0004] Figure 1 is a simplified block diagram providing a high-level illustration of a feature coding for machine (FCM) system. It will be appreciated that, as used herein, the terms feature coding for machines and FCM refer generally to any machine to machine application using feature representation of data, such as video, and is not intended to be limited to any particular coding standard. A resource limited edge device receives input video 110 and uses a split neural network part 1115 (also referred to herein as NN front end or head) to extract features of the video which are encoded by FCM encoder 120. The encoded features are transmitted over a channel to a base server which decodes the features. The base server includes an FCM decoder 130 which provides decoded feature maps 135 to a neural network part 2140 (also referred to herein as NN back end or tail) which provides output to inference results 145.

[0005] Referring to Figure 1, in FCM systems, a Neural Network is first split into two parts, i.e., NN Part 1 (Split CNN - Head) 115 and NN Part 2 (Split CNN - Tail) 140. Instead of running the full CNN on the end devices, only the NN Part 1115 is executed on the resource-limited devices, and the NN Part 2140 is preferably executed on the base server which is expected to have more computing resources. Since the NN Part 1115 is generally executed on the edge devices, instead of transmitting videos to the base server (like in Video Coding for Machine), the intermediate feature data extracted from NN Part 1 is sent to the base server. However, the volume of intermediate feature data is usually much larger than the video itself and needs efficient compression before sending it to the base server. An example of such a system is depicted in Error! Reference source not found..

[0006] Referring to Fig. 2, which is a block diagram further illustrating an exemplary FCM system, an FCM Encoder 205 which receives feature maps 210 from an NN part 1115 consists of three main modules, e.g., feature reduction 215, feature conversion 220, and inner codec 225 for encoding sequences.

[0007] The FCM architecture employs a split neural network, typically a CNN. In the context of CNNs, input data, output data and transformations with the neural network are generally represented using tensors. A tensor is the primary data structure used by a CNN and is a mathematical representation of a concept in the form of an n-dimensional array. For example, tensors can represent concepts such as scalar (0-dimensional tensor), vector (1-dimensional tensor), matrix (2 dimensional tensor), and an nd-array (n-dimensional tensor). Typical tensorsoccurring in the CNNs are 3-dimensional, e.g. [c, w, h], where c represents the number of channels, and w and h represent the width and height of each channel.

[0008] As noted in Figure 1, the FCM encoder provides an encoded bitstream over a communication link (e.g., wired or wireless communication link) to a base receiver including an FCM decoder. Referring to Fig. 2, FCM decoder 235 consists of three principal modules, e.g., an inner codec 240, inverse feature conversion 245, and feature restoration 250 for decoding the sequences with feature maps 255. The input of the FCM encoder 205 is intermediate feature data (^^௧) extracted from NN Part 1115 (Fig.1), and the output of the FCM decoder is reconstructed intermediate feature data (^^௧ᇱ), as shown in Error! Reference source not found., which is applied to NN Part 2140 (Fig.1). After reconstruction of the intermediate feature data (^^௧ᇱ), the NN Part 2 is executed to generate the inference results. Feature data usually consists of multiple layers. And each layer usually consists of multiple channels. For instance, in one example, the backbone of detectron2 (NN Part 1) may consist of 4 layers and each layer consists of 256 channels. The details of all the FCM Encoder and Decoder modules are described in more detail below.

[0009] The feature reduction module 215 of the FCM encoder 205 takes original feature maps 210 and / or layers as input and outputs a reduced number of feature maps and / or layers representing the original data. Essentially feature reduction module 215 takes the data from many layers and conveys it as a single layer. The feature reduction module 215 can be a neural network, auto-encoder, or just classical statistical approaches like PCA or clustering methods to reduce the number of feature maps and / or layers.

[0010] The feature conversion module 220 of the FCM encoder 205 receives reduced feature maps and / or layers as input and converts the feature tensor data from floating point 32-bit to unsigned integers, such as 8bit or 10bit integers. During feature conversion from Floating Point 32-bit to 8-bit or 10-bit, min-max normalization may be applied, followed by uniform quantization according to bit width. In some cases, depending on the data, it may also pack the feature maps into one frame. Then, the packed frame or converted feature maps and / or layers are converted by the feature conversion module to a suitable video format, such as YUV 4:0:0, before sending it to the inner codec 225.

[0011] The Inner Codec (Encoder) module 225 takes the video, such as YUV 4:0:0, representing the feature maps as input and outputs a bitstream to transmit over a communication channel 230 to the receiver 235. The Inner Codec (Encoder) 225 could be aNeural Network encoder or any traditional encoder like HEVC, VVC, VVenC, or a combination of both Neural Network encoder and traditional encoder.

[0012] Turning to the FCM decoder 235, the Inner Codec (Decoder) module 240 receives the encoded bitstream as input and decompresses the bitstream file to a suitable format, such as YUV 4:0:0. Like the inner codec (encoder) module 225, the inner codec (decoder) 240 can be implemented by a Neural Network based decoder or any traditional decoder like HEVC, VVC, VVdeC or the combination of both Neural Network based decoder and traditional decoder.

[0013] The inverse feature conversion module 245 takes the decoded YUV 4:0:0 from Inner Codec 240 as input and unpacks (if necessary) and converts the unsigned integers to floating point values. Min-max value sent from the encoder may be used to convert the unsigned integer to Floating Point 32-bit. The feature restoration module 250 receives feature tensors from the inverse feature conversion module 245 as input and restores the feature tensors from reduced size to the original size. Like the feature reduction module 215, the feature restoration module 250 can also be a Neural Network, Auto-Decoder, or just classical statistical approaches like Inverse PCA or de-clustering to restore the original size of feature maps and layers.

[0014] Documents which may provide relevant background information for the present disclosure include: Yuan Zhang, Manouchehr Rafie, Shan Liu, Christopher Hollmann, “[N00164] Call for Proposals on Video Coding for Machines.” ISO / IEC JTC 1 / SC 29 / WG 2, Jan.2022; C. Rosewarne (Canon) and Y. Zhang (China Telecom), “[N00282] Call for Proposals on Feature Compression for Video Coding for Machines.” ISO / IEC JTC 1 / SC 29 / WG 2, Jan. 2023; WG 04 MPEG Video coding, “[N00460] Algorithm description of FCTM.” Jan. 2024; and Y. Wu, A. Kirillov, F. Massa, W. Y. Lo, and R. Girshick, “Detectron2,” 2019, Feb. 2024 (Online). SUMMARY OF THE DISCLOSURE

[0015] Systems and methods for improved FCM encoding and decoding are provided. Examples of embodiments, features and advantages of the presently disclosed encoder and decoder embodiments include, but are not limited to, the following summary.

[0016] In one embodiment, an encoder for feature coding for machines (FCM) includes a split neural network front end, the split neural network front end receiving an input image signal and generating a plurality of feature maps comprising a plurality of feature tensors representing the input image signal. A calculate distribution properties module is provided which receives the plurality of feature maps and determines at least one probability distribution property of the plurality of feature tensors, including an RMS calculated from the feature tensors. A featurereduction module coupled to the calculate distribution properties module receives the at least one feature map and at least one distribution property, and generates at least one reduced feature map representing the original plurality of feature maps. A feature conversion module receives the reduced feature maps, converts the tensors of the reduced feature maps from floating point to a fixed integer format, and provides the converted feature maps and probability distribution properties in a video format. An encoder receives the video format signal from the feature conversion module and generates an encoded bitstream representing the plurality of feature maps and the at least one probability distribution property including the calculated RMS.

[0017] The probability distribution properties are calculated on a per layer basis or a per channel basis. In some embodiments, a second calculate distribution properties module can be provided after the feature reduction module and generates at least one probability distribution property of the reduced feature map.

[0018] In another embodiment, an encoder for feature coding for machines (FCM) includes a split neural network front end, the split neural network front end receiving an input image input image signal and generating a plurality of feature maps comprising a plurality of feature tensors representing the input image signal. A feature reduction module receives a plurality of feature maps and generates at least one reduced feature map including a plurality of tensors representing the plurality of feature maps. A calculate distribution properties module receives the reduced feature maps, at least one reduced feature map having at least one feature tensor. The calculate distribution properties module receives at least one feature tensor and determines at least one probability distribution property therefrom, the at least one probability distribution property including an RMS calculated from the feature tensors of the reduced feature map. A feature conversion module is coupled to the calculated distribution properties module and converts the reduced feature map and at last one probability distribution property to a video format. An inner encoder receives the video format signal from the feature conversion module and generates an encoded bitstream representing the plurality of feature maps and probability distribution properties.

[0019] The probability distribution properties can be calculated on a per layer basis or on a per channel basis.

[0020] In a further embodiment, a decoder for feature coding for machines (FCM) includes an inner decoder receiving a coded bitstream including at least one reduced feature map representing feature maps output from a neural network front end at an encoder and at leastone probability distribution property of the feature map including an RMS calculated by the encoder and provides a decoded video signal representing the at least one reduced feature map and probability distribution property including an RMS calculated by the encoder. An inverse feature conversion module receives the decoded video signal and provides feature tensors of layers of at least one feature map, the inverse feature conversion module receiving the at least one probability distribution property and calculates min and max properties of a quantization parameter therefrom. A feature restoration module receives feature tensors of the reduced feature map and reconstructs tensors for a plurality of feature maps representing a signal processed by a neural network front end. A reconstruction refinement module receives the probability distribution property from the bitstream and the tensors for a plurality of reconstructed feature maps from the feature restoration module and aligns a probability distribution property of the restored feature maps with the at least one probability distribution property in the bitstream.

[0021] In some embodiments, the decoder may include a second reconstruction refinement module interposed between the inverse feature conversion module and the feature restoration module, wherein the bitstream includes an RMS of a reduced feature map calculated by the encoder, and wherein the second reconstruction refinement module aligning an RMS of reconstructed reduced feature maps with the RMS of the reduced feature map in the bitstream. It will be appreciated that additional distribution properties, such as mean and standard deviation, may be included in addition to RMS.

[0022] The probability distribution properties can be calculated on a per layer basis or on a per channel basis. In some embodiments, reconstruction refinement may be performed on a periodic basis. In some embodiments, reconstruction refinement is performed on every frame in the bitstream.

[0023] In an other embodiment, a decoder for feature coding for machines (FCM), includes an inner decoder receiving a coded bitstream including at least one reduced feature map representing feature maps output from a neural network front end at an encoder and at least one probability distribution property of the feature map and providing a decoded video signal representing the at least one reduced feature map and probability distribution property. An inverse feature conversion module receives the decoded video signal and provides feature tensors of layers of at least one feature maps, the inverse feature conversion module receives the at least one probability distribution property and calculates min and max properties of aquantization parameter therefrom. A feature restoration module receives feature tensors of the reduced feature map and generating tensors for a plurality of feature maps represents a signal processed by a neural network front end used an encoder site from the reduced feature map. A reconstruction refinement module receives the probability distribution property from the bitstream and the tensors for a plurality of restored feature maps from the feature restoration module and aligns a probability distribution property of the restored feature maps with the at least one probability distribution property in the bitstream.

[0024] The probability distribution property may include a mean value of feature tensors, μ and a standard deviation of the feature tensors in the reduced feature map, σ, and the min =μ- 3σ and the max =μ+3σ.

[0025] The probability distribution properties are calculated on a per layer basis or on a per channel basis. The reconstruction refinement may be performed on a periodic basis. Alternatively, reconstruction refinement may be performed on every frame in the bitstream.

[0026] In a further embodiment, an encoder for feature coding for machines (FCM), includes a split neural network front end, the split neural network front end receiving an input image signal and generating a plurality of feature maps comprising a plurality of feature tensors representing the input image signal. An activity region extractor module, the activity region extractor receiving the input image signal and the plurality of feature maps and identifying regions of interest in the feature maps based on activity detected in the input images. A calculate distribution properties module receives the plurality of feature maps and receives parameters of the detected regions of interest from the activity region extractor and determines at least one probability distribution property of the plurality of feature tensors within each detected region of interest. A feature reduction module is coupled to the calculate distribution properties module and receives the at least one feature map and at least one distribution property for each region of interest and generates at least one reduced feature map representing the original plurality of feature maps within the regions of interest. A feature conversion module receives the reduced feature maps and converts the reduced feature maps and probability distribution properties of the regions of interest to a video format. An inner encoder receives the video format signal from the feature conversion module and generates an encoded bitstream representing the plurality of feature maps and the at least one probability distribution property of the regions of interest and encodes signaling information with parameters of the regions of interest.

[0027] The probability distribution properties can be calculated on a per layer basis or on a per channel basis. In some embodiments, a second calculate distribution properties module is provided after the feature reduction module and generates at least one probability distribution property of the reduced feature map.

[0028] An encoder for feature coding for machines (FCM) includes a split neural network front end, the split neural network front end receiving an input image input image signal and generates a plurality of feature maps comprising a plurality of feature tensors representing the input image signal. An activity region extractor module receives the input image signal and the plurality of feature maps and identifying regions of interest in the feature maps based on activity detected in the input images. A feature reduction module receives a plurality of feature maps and generates at least one reduced feature map including a plurality of tensors representing the plurality of feature maps. A calculate distribution properties module receives the reduced feature maps, and at least one reduced feature map having at least one feature tensor. The calculate distribution properties module receives at least one feature tensor and receiving parameters of the detected regions of interest from the activity region extractor and determining at least one probability distribution property of the regions of interest. A feature conversion module coupled to the calculated distribution properties module converts the reduced feature map and at last one probability distribution property to a video format. An inner encoder receives the video format signal from the feature conversion module and generates an encoded bitstream representing the plurality of feature maps and probability distribution properties.

[0029] In this further embodiment, the probability distribution properties may be calculated on a per layer basis or on a per channel basis.

[0030] In some embodiments, a decoder for feature coding for machines (FCM) includes an inner decoder receiving a coded bitstream including at least one reduced feature map representing feature maps output from a neural network front end at an encoder and at least one probability distribution property of the feature map and providing a decoded video signal representing the at least one reduced feature map and probability distribution property. An inverse feature conversion module receives the decoded video signal and provides feature tensors of layers of at least one feature maps, the inverse feature conversion module receives the at least one probability distribution property. A feature restoration module receives feature tensors of the reduced feature map and reconstructs tensors for a plurality of feature mapsrepresenting a signal processed by a neural network front end used an encoder site from the reduced feature map. A reconstruction refinement module receives the probability distribution property from the bitstream, the tensors for a plurality of reconstructed feature maps from the feature restoration module, and parameters of regions of interest signaled in the bitstream. The reconstruction refinement module aligns a probability distribution property of the restored feature maps within the regions of interest with the at least one probability distribution property in the bitstream for the corresponding regions of interest.

[0031] Some embodiments may include a second reconstruction refinement module interposed between the inverse feature conversion module and the feature restoration module. The second reconstruction refinement module receives tensors for a plurality of reduced feature maps and parameters of regions of interest signaled in the bitstream and aligning a probability distribution property of the reconstructed reduced feature maps within the regions of interest with the at least one probability distribution property in the bitstream for the corresponding regions of interest.

[0032] In some cases, reconstruction refinement may be performed on a periodic basis. Alternatively, reconstruction refinement may be performed on every frame in the bitstream.

[0033] In the various embodiments disclosed herein, reconstruction refinement can be performed using Method 1 and / or Method 2 disclosed herein, alone or in combination. Additional methods of reconstruction refinement which similarly align the distribution properties are also suitable for use in the present systems and methods.

[0034] In some embodiments, an inverse quantization method is employed which reuses the distribution properties of reconstruction refinements on the decoder side to perform inverse quantization.

[0035] In some embodiments decoder side inverse quantization can be performed using the equation:

[0036] In certaindetermines how often distribution properties will be extracted form the feature tensor. In some cases, the refinement period may be user-defined.

[0037] In some embodiments, distribution properties of the layer can be predicted from the distribution properties of the channels or vice versa. In some embodiments, a distributionproperty of the reduced feature tensor can be predicted using a linear combination of the original feature tensor distribution or vice versa.

[0038] Features and elements of the embodiments described above may be combined in some embodiments.

[0039] Exemplary bitstream syntax is provided to support the features disclosed herein. BRIEF DESCRIPTION OF THE FIGURES

[0040] For the purpose of illustrating the invention, the drawings show aspects of one or more embodiments of the invention. However, it should be understood that the present invention is not limited to the precise arrangements and instrumentalities shown in the drawings, wherein:

[0041] Figure 1 is a simplified block diagram of a system for Feature Coding for Machines (FCM);

[0042] Figure 2 is block diagram of a system for Feature Coding for Machines (FCM) of Fig. 1 and further illustrating the components of the encoder and decoder systems;

[0043] Figures 3A, 3B and 3C are graphs illustrating exemplary layer distribution after application of a first method disclosed herein;

[0044] Figures 4A, 4B, and 4C are graphs illustrating exemplary layer distribution after application of a second method disclosed herein;

[0045] Figures 5A and 5B are graphs illustrating a comparison of the first and second layer distribution methods disclosed herein;

[0046] Figure 6 is block diagram of a first embodiment of a system for Feature Coding for Machines (FCM) in accordance with the present disclosure;

[0047] Figure 7 is block diagram of a second embodiment of a system for Feature Coding for Machines (FCM) in accordance with the present disclosure;

[0048] Figure 8 is block diagram of a third embodiment of a system for Feature Coding for Machines (FCM) in accordance with the present disclosure;

[0049] Figure 9 is block diagram of a further embodiment of a system for Feature Coding for Machines (FCM) in accordance with the present disclosure;

[0050] Figures 10A and 10B are graphs illustrating the rate that distribution properties will be extracted from the feature tensor based on the user-defined refinement period;

[0051] Figure 11 is block diagram illustrating the prediction of reduced feature tensors from the original feature tensor distribution properties; and

[0052] Figure 12 is a block diagram of a computing system that can be used to implement any one or more of the methodologies disclosed herein and any one or more portions thereof.

[0053] The drawings are not necessarily to scale and may be illustrated by phantom lines, diagrammatic representations and fragmentary views. In certain instances, details that are not necessary for an understanding of the embodiments or that render other details difficult to perceive may have been omitted. DETAILED DESCRIPTION OF DISCLOSED EMBODIMENTS

[0054] The present systems and methods provide improvements over known FCM systems. It will be appreciated that while the embodiments are typically described in connection with the encoding and decoding of video signals, the present systems and methods may be applicable ot other high volume / bandwidth data, including but not limited to Lidar, Radar, sensor data, and the like.

[0055] Figure 6 is a simplified block diagram illustrating a first exemplary embodiment of an FCM system in accordance with the present disclosure. Referring to Fig.6, the FCM encoder includes a calculate distribution properties module 612 which is placed before the feature reduction module 615, which has been described in connection with Figure 2. At the FCM decoder 635, a reconstruction refinement module 653 is placed after the feature restoration module 650 previously described in connection with Figure 2. It should be noted that the architecture of Fig.6 is suitable for both channel level or layer lever reconstruction refinement or combinations of both.

[0056] The calculate distribution properties module 612 and reconstruction refinement module 653 are described in further detail below. The remaining modules illustrated in Fig.6 within the FCM decoder 635 and FCM encoder 605 operate in substantially the same manner as the corresponding blocks described above in connection with Fig.2.

[0057] The calculate distribution properties module 612 receives the original feature tensors of feature maps 610 (generated by an NN Part 1, not shown in Fig. 6) as input, calculates properties of the probability distribution of the feature tensors such as, but not limited to, mean (mu) and standard deviation (sigma), and sends those distribution properties, the refinement period, and the number of refinements, to the inner codec 625 for inclusion in a bitstream sent to the decoder 635. Properties of the probability distribution can be calculated for the features from the last layer of the NN part 1 of a split neural network. Properties such as mu and sigma can be calculated per-channel or per-layer. Calculating properties on a per-layer basis may beadvantageous in that only a single value or pair of values needs to be signaled for the whole layer.

[0058] The probability distribution properties are sent to the decoder as part of the bitstream, in lossless or lossy form, depending on the settings and system preferences. The probability distribution properties of channels or layers can also be derived using the reference frame’s distribution properties. In the case of channel level refinement, a subset of the channel’s distribution property could be sent to the decoder, and the other distribution properties could be derived on the decoder side.

[0059] The reconstruction refinement module 653 uses the signaled probability distribution properties from bitstream to refine the reconstructed feature tensors. Using the distribution properties sent from the FCM encoder 605, the reconstruction refinement module 653 will align the reconstructed tensor’s distribution as close to the original tensor’s distribution as possible. This alignment can be performed on different levels, e.g., layer level or channel level.

[0060] It should also be noted that the refinement could be done for every frame or periodically, depending on the system requirements. In the case of periodic refinement, the distribution properties from reference frames or derived distribution properties using reference frames could be used for distribution alignment of the current frame.

[0061] Reconstruction refinement may be performed on the decoder system using the following two exemplary methods described below. Typically, the input and output of all these methods is usually a tensor (or N-dimensional matrix). Method.1. Using only ^^ of the original distribution: ^^^^^^^^^^ௗ ൌ ^^^^^^^^^^ ൈ ఙ^^^ఙ^^^^^^Eq. 1.^^^^^^^^^^ௗ= refined layer or^^^^^^^^^= reconstructed layer or channel at the decoder side [input] ^^^^^= sigma of the original channel or layer distribution ^^^^^^^= sigma of the reconstructed channel or layer distribution Method.2. Using only (^^,^^) of the original distribution: A)^^^^^^^^^^ௗ ൌ ^^^^^^^^^^ ൈ ఙ^^^ఙ^^^^^^ Eq.2.^^^^^^^^^^ௗ= refined layer or channel [output] ^^^^^^^^^= reconstructed layer or channel at the decoder side [input] ^^^^^= sigma of the original channel or layer distribution ^^^^^^^= sigma of the reconstructed channel or layer distribution ^^^^^^^ ^^^௧= mu of reconstructed channel or layer distribution after applying Eq.2.of original channel or layer distribution B) Here, ^^^^^^^^^^^^^^^^௬^^ , ^^^^^^^^^^^^^^^^^^^^_^^^^^^^^^^^^௬^^ , ^^^^^^^^^^^^^^_^^^^^^^^^^^^௬^^ are tensors or (N-dimensionalmatrix). ^^^^^^^^^^^^^^^^^^^_^^^^^^^^^^^௧^^^^ ^^^^^ ^ ^^ೌ^^^ିఙ^ೌ^^^^^௬^^ ൌఙ^^^^^, for layer = 1, 2, …, n (4) n(5) where ^^^^^^^^^^ ^^^^^^^^^^^^ : Mean of the layer during reconstruction.^^^^^^^^^^ ^^^^^^^^^^^^ : Standard deviation of the layer during reconstruction.ftensorlayer : The feature tensor corresponding to the layer. ^^^^^^ ^^^^^^^^^^^^: Original mean of the layer.^^^^^^ ^^^^^^^^^^^^: Original standard deviation of the layer.normalized_layerlayer : The normalized layer obtained by subtracting the mean and dividing by the standard deviation during reconstruction. refined_layerslayer : The final refined layer obtained by multiplying the normalized layer by the original standard deviation and adding the original mean.

[0062] Both Method 1 and Method.2 can be applied at either the channel level or layer level. Further, Eq.2 and Eq.3 set forth above can be applied in any order and in any combination.

[0063] An exemplary distribution after applying both Method 1 and Method 2 is shown in Error! Reference source not found., Error! Reference source not found. and Error! Reference source not found.. The graphs in Figures 3 and 4 show histograms of the values in the feature layers overlayed with the fitted distributions. The Figs.3A and 4A are plots showing values of the input, original feature tensors. Figs 3B and 4B illustrate the values of the tensor values after reconstruction. Figs. 3C and 4C are graphs which show values after the present methods of refinement are applied. As evident from Figs 3C and 4C, the refinement transforms the distribution of Figs 3B and 4B into something more closely aligned with the original probability distribution. These figures illustrate that the reconstruction refinement method operates to align the reconstructed tensor’s distribution property as close to the distribution property of the original tensors. Figure 5A and 5B depicts the overlay of all three distributions (original, reconstructed, and refined).

[0064] Figure 7 is a block diagram illustrating a second embodiment of the presently disclosed FCM system. In the embodiment of Figure 7, the calculate distribution properties module 712 is placed after the feature reduction module 715 in the FCM system that has otherwise been described in connection with Figure 2. In this configuration, the calculate distribution properties module is determining the probability distribution property of the reduced feature maps rather than the original feature maps 710 from the neural network front end, i.e., NN Part 1. The calculate distribution properties module 712 is preferably coupled to the inner codec 725 and provides data such as the distribution properties, refinement period and number of refinements, as applicable, to be included in the encoded bitstream. Further, the reconstruction refinement module 747 may preferably be placed after the inverse feature conversion module 745. The reconstruction refinement module 747 is preferably coupled to the decoder 740 and receives parameters from the calculate distribution properties module 712 that were encoded in the bitstream, such as distribution properties, refinement period and number of refinements. The architecture of Figure 7 is suitable for both channel level or layer lever reconstruction refinement or combinations of both.

[0065] Figure 8 is a block diagram of a further embodiment of an FCM codec system in accordance with the present disclosure which combines the features and benefits of the embodiments of Fig. 6 and Fig. 7. Referring to Fig. 8, the calculate distribution properties function is performed by the calculate distribution properties 1812 at the input of FCM encoder 805 and calculate distribution properties 2 module 817 at the output of the feature reduction module 815. This allows the distribution properties to be determined both before feature reduction module 815 and after feature reduction module 815. Although this requires additional encoder complexity and overhead in the bitstream in that two sets of distribution properties are encoded in the bitstream, this may be advantageous in certain applications since the layers of the original features maps (typically five layers each in the original feature maps from NN Part 1) have different distribution properties than the single layer that results from feature reduction module 815 and having the distribution properties for both the original and reduced feature tensors available may provide for further refinement of the reconstructed tensors at the decoder. Similarly, at the decoder side, the reconstruction refinement properties process is divided among reconstruction refinement 1 module 852 and reconstruction refinement 2 module 847. The reconstruction refinement 1 module 852 preferably operates with the distribution properties of the original feature tensors provided by inner codec 840 and the reconstruction refinement 2 module preferably operates with the reduced feature tensors provided by inner codec 840. It should be noted that the exemplary embodiment of Figure 8 is also suitable for both channel level or layer lever reconstruction refinement or the combinations of both.

[0066] Preferably, both the calculate distribution properties 1 module 812 and the calculate distribution properties 2 module are preferably coupled to the encoder 825 to provide data, such as distribution properties, refinement period and number of refinements to be encoded in the bitstream. Similarly, at the FCM decoder 835, the reconstruction refinement 1 module and reconstruction refinement 2 modules can be coupled to the decoder 840 to receive data such as distribution properties, refinement period and number of refinements, that were encoded in the bitstream.

[0067] Embodiments of encoders in accordance with the present disclosure may send layers or channel distribution properties to the decoder and the decoder will use those properties to align the reconstructed layers' or channel distributions' properties close to the original tensor's distribution property.

[0068] Known FCM systems may use min-max normalization followed by uniform quantization to convert the Floating Point 32-bit precision data to unsigned integers of 8-bit or10-bit. This may be performed by the feature conversion module and the min-max parameters provided to the inner codec for inclusion in the bitstream. Forward Quantization (at Encoder): Eq. 6 Inverse Quantization (atEq.7

[0069] In this disclosed preferred embodiment, we present the following Inverse Quantization method, which reuses the distribution properties of reconstruction refinements on the decoder side to perform inverse quantization.Present Inverse Quan za on (at Decoder):

[0070] The refinementproperties will be extracted from the feature tensor, and it is user-defined. For example, as illustrated in Figs.10A and 10B, if the refinement period is 32, distribution properties will be extracted for the first frame (I Frame), and the same distribution properties will be reused for the refinement of the remaining 31 frames (P Frames) on the decoder side. If the refinement period is 1, distribution properties will be extracted for every frame, as illustrated in Fig.10B.

[0071] If the refinement period is low and the process takes the average of all the distribution properties, the delay at the decoder will be minimal. Conversely, if the refinement period is high and the process take the average of all the distribution properties, the precision of the distribution properties will be higher, enhancing the end-task accuracy. Distribution Property Prediction

[0072] The distribution properties of the layer can be predicted from the distribution properties of the channels or vice versa. For instance, in the following figure, there are n channels, and the distribution properties of the layer can be predicted from the distribution properties of the n channels

[0073] The distribu on Property of the Reduced Feature Tensor can be predicted using a linearcombination of the Original Feature Tensor distribution or vice versa. An example is depicted in Figure 11 with prediction 1105 receiving the original tensor distribution properties and predicting the reduced tensor distribution properties. 73.1. An exemplary bitstream syntax for use with min-max calculation at the decoder of theDescriptor ^i bit ^

[0001] Syntax Elements: ^ end_task_type ൌ 0: Object Detection, 1: Object Segmentation, 2: Object Tracking ^unsigned integer: 2 bits^^ reϐinement_period_for_org_ft ൌ how often reconstruction reϐinement will be applied to the original feature tensor ^unsigned integer: 8 bits^ Range ^1 to 256^ ^ reϐinement_period_for_reduced_ft ൌ how often reconstruction reϐinement will be applied to the reduced feature tensor ^unsigned integer: 8 bits^ Range ^1 to 256^ ^ poc ൌ picture of coding ^ layer_sigma^i^ ൌ sigma of the ith layer’s distribution ^ϐloating point: 32 bits^ ^ layer_mu^i^ ൌ mu of the ith layer’s distribution ^ϐloating point: 32 bits^ Example Bitstream Syntax 2 Descriptor ^ ^Descriptor…^

[0002] Syntax Elements: ^ is_apply_on_org_ft ൌ ϐlag to signal whether the reconstruction reϐinement will be applied to the original feature tensor or not ^unsigned integer: 1 bit^ ^ reϐinement_period_for_org_ft ൌ how often reconstruction reϐinement will be applied to the original feature tensor ^unsigned integer: 8 bits^ ^ is_apply_on_reduced_ft ൌ ϐlag to signal whether the reconstruction reϐinement will be applied to the reduced feature tensor or not ^unsigned integer: 1 bit^^ reϐinement_period_for_reduced_ft ൌ how often reconstruction reϐinement will be applied to the reduced feature tensor ^unsigned integer: 8 bits^ ^ poc ൌ picture of coding ^ layer_sigma^i^ ൌ sigma of the ith layer’s distribution ^ϐloating point: 32 bits^ ^ layer_mu^i^ ൌ mu of the ith layer’s distribution ^ϐloating point: 32 bits^

[0074]

[0075]

[0076] Figure 9 is a block diagram of an alternate embodiment of an FCM system in accordance with the present disclosure. Referring to Fig. 9, the FCM encoder includes a calculate distribution properties module 912 which receives both the original feature maps 910 and the reduced feature tensors from feature reduction module 915. A user defined input may be provide at 905 to further define the End Task 907, such as detection, segmentation and training. User defined input may also include a user-defined refinement period. The output of end task 907 is provided to the inner codec and this data is included in the coded bitstream. The feature reduction module 915, feature conversion module 920 are substantially the same as described above, such as in connection with Fig.6.

[0077] In the embodiment of Fig.9, the calculate distribution properties module 912 calculates a different set of properties than described in connection with Fig. 6. One example of such a distribution property could a root mean square (RMS) calculation of the original tensors. These RMS distribution properties could also be extracted from different levels, e.g., layer level and channel level. Depending on the system requirements, these distribution properties could be encoded in a lossy or lossless way in the bitstreams. It should be noted that it’s also possible to derive the distribution properties of channels or layers using the reference frame’s distribution properties. In the case of channel level refinement, a subset of the channels’ distribution property could be sent to the decoder, and the other distribution properties could be derived on the decoder side.

[0078] The FCM decoder of Fig. 9, includes an inner codec 940, inverse feature conversion module 945, and feature restoration module 950 as previously described. The decoder of Fig. 9 further includes a reconstruction refinements 2 module 947 and a reconstruction refinements 1 module 952.

[0079] It should also be noted that the refinement performed by the reconstruction refinements modules may be performed for every frame or periodically, depending on the system requirements. In the case of periodic refinement, the distribution properties from referenceframes or derived distribution properties using reference frames could be used for distribution alignment of the current frame.

[0080] Some embodiments of the present disclosure may include / and or be embodied by non- transitory computer program products (i.e., physically embodied computer program products) that store instructions, which when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform operations herein. Such non- transitory computer program products (i.e., physically embodied computer program products) may store instructions, which when executed by one or more data processors of one or more computing systems, causes at least one data processor to perform operations, and / or steps thereof described in this disclosure, including without limitation any operations described above and / or any operations of the FCM decoder and / or FCM encoder may be configured to perform. Similarly, computer systems are also described that may include one or more data processors and memory coupled to the one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. In addition, methods can be implemented by one or more data processors either within a single computing system or distributed among two or more computing systems. Such computing systems can be connected and can exchange data and / or commands or other instructions or the like via one or more connections, including a connection over a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, or the like), via a direct connection between one or more of the multiple computing systems, or the like.

[0081] Any one or more of the aspects and embodiments described herein may be conveniently implemented using one or more machines (e.g., one or more computing devices that are utilized as a user computing device for an electronic document, one or more server devices, such as a document server, etc.) programmed according to the teachings of the present specification, as will be apparent to those of ordinary skill in the computer art. Appropriate software coding can readily be prepared by skilled programmers based on the teachings of the present disclosure, as will be apparent to those of ordinary skill in the software art. Aspects and implementations discussed above employing software and / or software modules may also include appropriate hardware for assisting in the implementation of the machine executable instructions of the software and / or software module.

[0082] Such software may be a computer program product that employs a machine-readable storage medium. A machine-readable storage medium may be any medium that is capable of storing and / or encoding a sequence of instructions for execution by a machine (e.g., acomputing device) and that causes the machine to perform any one of the methodologies and / or embodiments described herein. Examples of a machine-readable storage medium include, but are not limited to, a magnetic disk, an optical disc (e.g., CD, CD-R, DVD, DVD-R, etc.), a magneto-optical disk, a read-only memory “ROM” device, a random-access memory “RAM” device, a magnetic card, an optical card, a solid-state memory device, an EPROM, an EEPROM, and any combinations thereof. A machine-readable medium, as used herein, is intended to include a single medium as well as a collection of physically separate media, such as, for example, a collection of compact discs or one or more hard disk drives in combination with a computer memory. As used herein, a machine-readable storage medium does not include transitory forms of signal transmission.

[0083] Such software may also include information (e.g., data) carried as a data signal on a data carrier, such as a carrier wave. For example, machine-executable information may be included as a data-carrying signal embodied in a data carrier in which the signal encodes a sequence of instruction, or portion thereof, for execution by a machine (e.g., a computing device) and any related information (e.g., data structures and data) that causes the machine to perform any one of the methodologies and / or embodiments described herein.

[0084] Examples of a computing device include, but are not limited to, an electronic book reading device, a computer workstation, a terminal computer, a server computer, a handheld device (e.g., a tablet computer, a smartphone, etc.), a web appliance, a network router, a network switch, a network bridge, any machine capable of executing a sequence of instructions that specify an action to be taken by that machine, and any combinations thereof. In one example, a computing device may include and / or be included in a kiosk.

[0085] FIG. 12 shows a diagrammatic representation of one embodiment of a computing device in the exemplary form of a computer system 1200 within which a set of instructions for causing a control system to perform any one or more of the aspects and / or methodologies of the present disclosure may be executed. It is also contemplated that multiple computing devices may be utilized to implement a specially configured set of instructions for causing one or more of the devices to perform any one or more of the aspects and / or methodologies of the present disclosure. Computer system 1200 includes a processor 1204 and a memory 1208 that communicate with each other, and with other components, via a bus 1212. Bus 1212 may include any of several types of bus structures including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combinations thereof, using any of a variety of bus architectures.

[0086] Processor 1204 may include any suitable processor, such as without limitation a processor incorporating logical circuitry for performing arithmetic and logical operations, such as an arithmetic and logic unit (ALU), which may be regulated with a state machine and directed by operational inputs from memory and / or sensors; processor 1204 may be organized according to Von Neumann and / or Harvard architecture as a non-limiting example. Processor 1204 may include, incorporate, and / or be incorporated in, without limitation, a microcontroller, microprocessor, digital signal processor (DSP), Field Programmable Gate Array (FPGA), Complex Programmable Logic Device (CPLD), Graphical Processing Unit (GPU), general purpose GPU, Tensor Processing Unit (TPU), analog or mixed signal processor, Trusted Platform Module (TPM), a floating-point unit (FPU), and / or system on a chip (SoC).

[0087] Memory 1208 may include various components (e.g., machine-readable media) including, but not limited to, a random-access memory component, a read only component, and any combinations thereof. In one example, a basic input / output system 1216 (BIOS), including basic routines that help to transfer information between elements within computer system 1200, such as during start-up, may be stored in memory 1208. Memory 1208 may also include (e.g., stored on one or more machine-readable media) instructions (e.g., software) 1220 embodying any one or more of the aspects and / or methodologies of the present disclosure. In another example, memory 1208 may further include any number of program modules including, but not limited to, an operating system, one or more application programs, other program modules, program data, and any combinations thereof.

[0088] Computer system 1200 may also include a storage device 1224. Examples of a storage device (e.g., storage device 1224) include, but are not limited to, a hard disk drive, a magnetic disk drive, an optical disc drive in combination with an optical medium, a solid-state memory device, and any combinations thereof. Storage device 1224 may be connected to bus 1212 by an appropriate interface (not shown). Example interfaces include, but are not limited to, SCSI, advanced technology attachment (ATA), serial ATA, universal serial bus (USB), IEEE 1394 (FIREWIRE), and any combinations thereof. In one example, storage device 1224 (or one or more components thereof) may be removably interfaced with computer system 1200 (e.g., via an external port connector (not shown)). Particularly, storage device 1224 and an associated machine-readable medium 1228 may provide nonvolatile and / or volatile storage of machine- readable instructions, data structures, program modules, and / or other data for computer system 1200. In one example, software 1220 may reside, completely or partially, within machine- readable medium 1228. In another example, software 1220 may reside, completely or partially, within processor 1204.

[0089] Computer system 1200 may also include an input device 1232. In one example, a user of computer system 1200 may enter commands and / or other information into computer system 1200 via input device 1232. Examples of an input device 1232 include, but are not limited to, an alpha-numeric input device (e.g., a keyboard), a pointing device, a joystick, a gamepad, an audio input device (e.g., a microphone, a voice response system, etc.), a cursor control device (e.g., a mouse), a touchpad, an optical scanner, a video capture device (e.g., a still camera, a video camera), a touchscreen, and any combinations thereof. Input device 1232 may be interfaced to bus 1212 via any of a variety of interfaces (not shown) including, but not limited to, a serial interface, a parallel interface, a game port, a USB interface, a FIREWIRE interface, a direct interface to bus 1212, and any combinations thereof. Input device 1232 may include a touch screen interface that may be a part of or separate from display 1236, discussed further below. Input device 1232 may be utilized as a user selection device for selecting one or more graphical representations in a graphical interface as described above.

[0090] A user may also input commands and / or other information to computer system 1200 via storage device 1224 (e.g., a removable disk drive, a flash drive, etc.) and / or network interface device 1240. A network interface device, such as network interface device 1240, may be utilized for connecting computer system 1200 to one or more of a variety of networks, such as network 1244, and one or more remote devices 1248 connected thereto. Examples of a network interface device include, but are not limited to, a network interface card (e.g., a mobile network interface card, a LAN card), a modem, and any combination thereof. Examples of a network include, but are not limited to, a wide area network (e.g., the Internet, an enterprise network), a local area network (e.g., a network associated with an office, a building, a campus or other relatively small geographic space), a telephone network, a data network associated with a telephone / voice provider (e.g., a mobile communications provider data and / or voice network), a direct connection between two computing devices, and any combinations thereof. A network, such as network 1244, may employ a wired and / or a wireless mode of communication. In general, any network topology may be used. Information (e.g., data, software 1220, etc.) may be communicated to and / or from computer system 1200 via network interface device 1240.

[0091] Computer system 1200 may further include a video display adapter 1252 for communicating a displayable image to a display device, such as display device 1236. Examples of a display device include, but are not limited to, a liquid crystal display (LCD), a cathode ray tube (CRT), a plasma display, a light emitting diode (LED) display, and any combinations thereof. Display adapter 1252 and display device 1236 may be utilized in combination withprocessor 1204 to provide graphical representations of aspects of the present disclosure. In addition to a display device, computer system 1200 may include one or more other peripheral output devices including, but not limited to, an audio speaker, a printer, and any combinations thereof. Such peripheral output devices may be connected to bus 1212 via a peripheral interface 1256. Examples of a peripheral interface include, but are not limited to, a serial port, a USB connection, a FIREWIRE connection, a parallel connection, and any combinations thereof.

[0092] The foregoing has been a detailed description of illustrative embodiments of the invention. Various modifications and additions can be made without departing from the spirit and scope of this invention. Features of each of the various embodiments described above may be combined with features of other described embodiments as appropriate in order to provide a multiplicity of feature combinations in associated new embodiments. Furthermore, while the foregoing describes a number of separate embodiments, what has been described herein is merely illustrative of the application of the principles of the present invention. Additionally, although particular methods herein may be illustrated and / or described as being performed in a specific order, the ordering is highly variable within ordinary skill to achieve methods, systems, and software according to the present disclosure. Accordingly, this description is meant to be taken only by way of example, and not to otherwise limit the scope of this invention.

[0093] Exemplary embodiments have been disclosed above and illustrated in the accompanying drawings. It will be understood by those skilled in the art that various changes, omissions and additions may be made to that which is specifically disclosed herein without departing from the spirit and scope of the present invention.

Claims

What is Claimed is:

1. An encoder for feature coding for machines (FCM), comprising: a split neural network front end, the split neural network front end receiving an input image signal and generating a plurality of feature maps comprising a plurality of feature tensors representing the input image signal; an end user input providing entry of a user-defined refinement period; a calculate distribution properties module receiving the plurality of feature maps and determining at least one probability distribution property of the plurality of feature tensors, the at least one probability distribution property including an RMS calculated from the feature tensors; a feature reduction module coupled to the calculate distribution properties module and receiving the at least one feature map and at least one distribution property, and generating at least one reduced feature map representing the original plurality of feature maps; a feature conversion module receiving the reduced feature maps and converting the reduced feature maps and probability distribution properties to a video format; and an inner encoder receiving the video format signal from the feature conversion module, the user-defined refinement period, and generating an encoded bitstream representing the plurality of feature maps and the at least one probability distribution property.

2. The encoder of claim 1, wherein the probability distribution properties are calculated on a per layer basis.

3. The encoder of claim 1, wherein the probability distribution properties are calculated on a per channel basis.

4. The encoder of claim 1, wherein a second calculate distribution properties module is provided after the feature reduction module and generating at least one probability distribution property of the reduced feature map. [0034] In some embodiments, an inverse quantization method is employed which reuses the distribution properties of reconstruction refinements on the decoder side to perform inverse quantization. [0035] In some embodiments decoder side inverse quantization can be performed using the equation:[0036] In certain embodiements, the refinement period determines how often distribution properties will be extracted form the feature tensor. In some cases, the refinement period may be user-defined. [0037] In some embodiments, distribution properties of the layer can be predicted from the distribution properties of the channels or vice versa. In some embodiments, a distribution property of the reduced feature tensor can be predicted using a linear combination of the original feature tensor distribution or vice versa.

5. A decoder for feature coding for machines (FCM), comprising: an inner decoder receiving a coded bitstream including at least one reduced feature map representing feature maps output from a neural network front end at an encoder, at least one probability distribution property of the feature map, and a user- defined refinement period and providing a decoded video signal representing the at least one reduced feature map and probability distribution property, wherein the inner codec reuses distribution properties of reconstruction refinements to perform inverse quantization; an inverse feature conversion module receiving the decoded video signal and providing feature tensors of layers of at least one feature maps, the inverse feature conversion module receiving the at least one probability distribution property, and calculating min and max properties of a quantization parameter therefrom; a feature restoration module receiving feature tensors of the reduced feature map and generating tensors for a plurality of feature maps representing a signal processed by a neural network front end used an encoder site from the reduced feature map; and a reconstruction refinement module receiving the probability distribution property from the bitstream and the tensors for a plurality of restored feature maps from the feature restoration module and aligning a probability distribution property of the restored feature maps with the at least one probability distribution property in the bitstream.

6. The decoder of claim 5, wherein the user-defined refinement period determines how often distribution properties will be extracted from the feature tensor.

7. The decoder of claim 5 wherein inverse quantization is performed using the equation8. The decoder of claim 8, wherein the probability distribution properties are calculated on a per layer basis.

9. The decoder of claim 8, wherein the probability distribution properties are calculated on a per channel basis.

10. The decoder of claim 8, wherein reconstruction refinement is performed on a periodic basis.

11. The decoder of claim 8, wherein reconstruction refinement is performed on every frame in the bitstream.

Images