Systems, methods and bitstreams for reconstruction refinement in feature coding for machines (FCM)

Abstract

Systems and methods are provided to improve encoding and decoding of video in a system for feature coding for machines (FCM). At the encoder site, a calculate distribution properties process is applied to the feature maps output by a split neural network front end to determine at least one probability distribution property of the tensors of the feature maps. At a decoder site, a reconstruction refinement process uses the probability distribution property encoded in a bitstream to align the probability distribution properties of decoded and restored feature maps with the original feature maps. The refined and restored feature maps are applied to the split neural network back end for inferencing.

Description

SYSTEMS, METHODS AND BITSTREAMS FOR RECONSTRUCTION REFINEMENT IN FEATURE CODING FOR MACHINES (FCM) CROSS REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims the benefit of priority to U.S. Provisional Application serial no. 63 / 568,207 filed on March 21, 2024, and entitled SYSTEMS, METHODS AND BITSTREAMS FOR RECONSTRUCTION REFINEMENT IN FEATURE CODING FOR MACHINES (FCM). BACKGROUND OF THE DISCLOSURE

[0002] As the number and scale of deployed video sensors / devices increases, an increasing amount of video is expected to be processed by machines and for machine use rather than human viewing. Indeed, an exemplary system or solution employing thousands of cameras would produce significant amounts of video that cannot be monitored by humans in a cost- effective manner. Because of the significant volume of the data, an efficient compression system is desirable for Machine-to-Machine (M2M) communication. To standardize the coding for machines to facilitate a Machine-to-Machine (M2M) communication system more efficiently, MPEG issued a Call for Proposal (CfP) for Coding for Machines in 2022.

[0003] Currently, M2M communications are typically managed by two distinct systems, namely, edge computing or local computing systems and remote computing or Video Coding for Machines (VCM) systems. In the context of edge computing systems, the complete Convolutional Neural Network (CNN) is typically executed on edge devices. However, end devices often lack the computational capacity to run extensive networks, requiring billions of operations for inference result calculations. In contrast, in Video Coding for Machine systems, the video is typically compressed initially and transmitted to a base server or cloud server with more computing resources to execute the full CNN. The calculated results are then subsequently relayed back to the edge devices. However, Video Coding for machine systemsmay not be able to support compute offload and may fail to fully leverage the potential of end devices, as they execute the complete CNN on the base server. To overcome these limitations for M2M communication and to enable collaborative intelligence and split computing for M2M connections, MPEG has issued a Call for Proposal (CfP) for Feature Coding for Machines (FCM) technology in 2023.

[0004] Figure 1 is a simplified block diagram providing a high-level illustration of a Feature Coding for Machine (FCM) system. A resource limited device receives input video 110 and uses a split neural network part 1115 (also referred to herein as NN front end or head) to extract features of the video which are encoded by FCM encoder 120. The encoded features are transmitted over a channel to a base server which decodes the features. The base server includes an FCM decoder 130 which provides decoded feature maps 135 to a neural network part 2140 (also referred to herein as NN back end or tail) which provides output to inference results 145.

[0005] Referring to Figure 1, in Feature Coding for Machine (FCM) systems, a Neural Network is first split into two parts, i.e., NN Part 1 (Split CNN - Head) 115 and NN Part 2 (Split CNN - Tail) 140. Instead of running the full CNN on the end devices, only the NN Part 1115 is executed on the resource-limited devices, and the NN Part 2140 is preferably executed on the base server which is expected to have more computing resources. Since the NN Part 1 115 is generally executed on the edge devices, instead of transmitting videos to the base server (like in Video Coding for Machine), the intermediate feature data extracted from NN Part 1 is sent to the base server. However, the volume of intermediate feature data is usually much larger than the video itself and needs efficient compression before sending it to the base server. An example of such a system is depicted in Error! Reference source not found..

[0006] Referring to Fig. 2, which is a block diagram further illustrating an exemplary FCM system, an FCM Encoder 205 which receives feature maps 210 from an NN part 1115 consistsof three main modules, e.g., feature reduction 215, feature conversion 220, and inner codec 225 for encoding sequences.

[0007] The FCM architecture employs a split neural network, typically a CNN. In the context of CNNs, input data, output data and transformations with the neural network are generally represented using tensors. A tensor is the primary data structure used by a CNN and is a mathematical representation of a concept in the form of an n-dimensional array. For example, tensors can represent concepts such as scalar (0 dimensional tensor), vector (1 dimensional tensor), matrix (2 dimensional tensor), and an nd-array (n dimensional tensor). Typical tensors occurring in the CNNs are 3-dimensional, e.g. [c, w, h], where c represents the number of channels, and w and h represent the width and height of each channel.

[0008] As noted in Figure 1, the FCM encoder provides an encoded bitstream over a communication link (e.g., wired or wireless communication link) to a base receiver including an FCM decoder. Referring to Fig. 2, FCM decoder 235 consists of three principal modules, e.g., an inner codec 240, inverse feature conversion 245, and feature restoration 250 for decoding the sequences with feature maps 255. The input of the FCM encoder 205 is intermediate feature data (^^௧) extracted from NN Part 1115 (Fig.1), and the output of the FCM decoder is reconstructed intermediate feature data (^^௧ᇱ), as shown in Error! Reference source not found., which is applied to NN Part 2140 (Fig.1). After reconstruction of the intermediate feature data (^^௧ᇱ), the NN Part 2 is executed to generate the inference results. Feature data usually consists of multiple layers. And each layer usually consists of multiple channels. For instance, in one example, the backbone of detectron2 (NN Part 1) may consist of 4 layers and each layer consists of 256 channels. The details of all the FCM Encoder and Decoder modules are described in more detail below.

[0009] The feature reduction module 215 of the FCM encoder 205 takes original feature maps 210 and / or layers as input and outputs a reduced number of feature maps and / or layersrepresenting the original data. Essentially feature reduction module 215 takes the data from many layers and conveys it as a single layer. The feature reduction module 215 can be a neural network, auto-encoder, or just classical statistical approaches like PCA or clustering methods to reduce the number of feature maps and / or layers.

[0010] The feature conversion module 220 of the FCM encoder 205 receives reduced feature maps and / or layers as input and converts the feature tensor data from floating point to unsigned integers, such as 8bit or 10bit integers. In some cases, depending on the data, it may also pack the feature maps into one frame. Then, the packed frame or converted feature maps and / or layers are converted by the feature conversion module to a suitable video format, such as YUV 4:0:0, before sending it to the inner codec 225.

[0011] The Inner Codec (Encoder) module 225 takes the video, such as YUV 4:0:0, representing the feature maps as input and outputs a bitstream to transmit over a communication channel 230 to the receiver 235. The Inner Codec (Encoder) 225 could be a Neural Network encoder or any traditional encoder like HEVC, VVC, VVenC, or a combination of both Neural Network encoder and traditional encoder.

[0012] Turning to the FCM decoder 235, the Inner Codec (Decoder) module 240 receives the encoded bitstream as input and decompresses the bitstream file to a suitable format, such as YUV 4:0:0. Like the inner codec (encoder) module 225, the inner codec (decoder) 240 can be implemented by a Neural Network based decoder or any traditional decoder like HEVC, VVC, VVdeC or the combination of both Neural Network based decoder and traditional decoder.

[0013] The inverse feature conversion module 245 takes the decoded YUV 4:0:0 from Inner Codec 240 as input and unpacks (if necessary) and converts the unsigned integers to floating point values.

[0014] The feature restoration module 250 receives feature tensors from the inverse feature conversion module 245 as input and restores the feature tensors from reduced size to theoriginal size. Like the feature reduction module 215, the feature restoration module 250 can also be a Neural Network, Auto-Decoder, or just classical statistical approaches like Inverse PCA or de-clustering to restore the original size of feature maps and layers.

[0015] Documents which may provide relevant background information for the present disclosure include: [1] Yuan Zhang, Manouchehr Rafie, Shan Liu, Christopher Hollmann, “[N00164] Call for Proposals on Video Coding for Machines.” ISO / IEC JTC 1 / SC 29 / WG 2, Jan. 2022; [2] C. Rosewarne (Canon) and Y. Zhang (China Telecom), “[N00282] Call for Proposals on Feature Compression for Video Coding for Machines.” ISO / IEC JTC 1 / SC 29 / WG 2, Jan. 2023; [3] WG 04 MPEG Video coding, “[N00460] Algorithm description of FCTM.” Jan. 2024; and [4] Y. Wu, A. Kirillov, F. Massa, W. Y. Lo, and R. Girshick, “Detectron2,” 2019, Feb.2024 (Online). SUMMARY OF THE DISCLOSURE

[0016] Systems and methods for improved FCM encoding and decoding are provided. Examples of embodiments, features and advantages of the presently disclosed encoder and decoder embodiments include, but are not limited to, the following summary.

[0017] An encoder for feature coding for machines (FCM) includes a calculate distribution properties module which receives a plurality of feature maps output by a split neural network front end. The feature maps include feature tensors, and the calculate distribution properties module receives at least one feature tensor and determines at least one probability distribution property from the feature tensor(s). The probability distribution property(ies) are included in a bitstream provided to a decoder site and may be used to align the probability distribution properties of restored feature maps input to a backend of the split neural network at the decoder site.

[0018] The encoder may include a feature reduction module coupled to the calculate distribution properties module and receives the at least one feature map and at least onedistribution property and generates at least one reduced feature map representing the original plurality of feature maps output by the neural network front end. A feature conversion module may be provided to receive the reduced feature maps and convert the reduced feature maps and probability distribution properties to a video format. An inner encoder receives the video format signal from the feature conversion module and generates an encoded bitstream representing the plurality of feature maps and probability distribution property / properties.

[0019] Alternatively, the calculate distribution properties module can be placed after the feature reduction module and generate at least one probability distribution property of the reduced feature map rather than the original plurality of feature maps.

[0020] In yet another alternative embodiment a first calculate distribution properties module can be placed at the input of the feature reduction motion and a second calculate distribution properties module can be placed after the feature reduction module. In this case probability distribution properties of both the original feature maps and reduced feature maps can be determined and included in the bitstream.

[0021] The encoders described above can be part of an encoding system for feature coding for machines (FCM) which includes a neural network front end receiving an input image signal and generating a plurality of feature maps representing the input image signal.

[0022] The probability distribution property can be a mean value of feature tensors in a feature map. The probability distribution property can be a standard deviation of feature tensors in a feature map. The probability distribution property can be other statistical property or a combination of properties. Multiple statistical properties for a layer of channel can be used individually or in combination. The probability distribution properties of the NN front end layers can be calculated on a per layer basis, a per channel basis or both. A second calculate distribution properties module can optionally be provided after the feature reduction moduleand generates a probability distribution property of the reduced feature map to be signaled in the bitstream.

[0023] A decoder for feature coding for machines (FCM) preferably includes an inner decoder which receives a coded bitstream including at least one reduced feature map representing feature maps output from a split neural network front end used at an encoder and at least one probability distribution property of the feature maps. The inner decoder provides a decoded video signal representing the reduced feature map(s) and probability distribution property / properties in the bitstream. An inverse feature conversion module may be provided to receive the decoded video signal and provide feature tensors of at least one reduced feature map in the bitstream.

[0024] A feature restoration module receives the feature tensors of the reduced feature map and generates tensors for a plurality of restored feature maps representing the output layer of a neural network front end used an encoder site. A reconstruction refinement module is provided which receives the probability distribution property from the bitstream and the tensors for a plurality of restored feature maps from the feature restoration module and aligns a probability distribution property of the restored feature maps with the at least one probability distribution property in the bitstream. The resulting realignment can reduce the impact of the feature reduction and feature restoration processes which may alter the probability distribution of the original feature maps.

[0025] The probability distribution property is determined at the encoder site and may be a mean value, a standard deviation, or other statistical measure of feature tensors in the feature maps or reduced feature maps. The probability distribution property can be multiple statistical measurements used in combination.

[0026] The encoder may send layers distribution properties or channel distribution properties to the decoder, and the decoder will use those properties to align the reconstructed layers' or channel distributions' properties as close to the original tensor's distribution property.

[0027] At the decoder, refinement can be performed periodically or for every frame. In the case of periodic refinement, distribution properties from available reference frames could be used.

[0028] Distribution properties between consecutive frames may be derived or predicted from other available frames.

[0029] Distribution properties between layers or channels may be derived or predicted from other information available on the decoder side.

[0030] Distribution properties between channels of different layers or the same layer could also be derived or predicted.

[0031] Reconstruction refinement may be done on different levels, e.g., layer level and channel level.

[0032] Distribution properties may be provided in the bitstream in either lossless or a lossy encoding method.

[0033] Calculate distribution properties and reconstruction refinement methods disclosed herein may be placed at various points on the encoder or the decoder if there is a presence of lossy coding or neural network bottleneck. BRIEF DESCRIPTION OF THE FIGURES

[0034] For the purpose of illustrating the invention, the drawings show aspects of one or more embodiments of the invention. However, it should be understood that the present invention is not limited to the precise arrangements and instrumentalities shown in the drawings, wherein:

[0035] Figure 1 is a simplified block diagram of a system for Feature Coding for Machines (FCM);

[0036] Figure 2 is block diagram of a system for Feature Coding for Machines (FCM) of Fig. 1 and further illustrating the components of the encoder and decoder systems;

[0037] Figure 3 are graphs illustrating exemplary layer distribution after application of a first method disclosed herein;

[0038] Figure 4 are graphs illustrating exemplary layer distribution after application of a second method disclosed herein;

[0039] Figure 5 are graphs illustrating a comparison of the first and second layer distribution methods disclosed herein;

[0040] Figure 6 is block diagram of a first embodiment of a system for Feature Coding for Machines (FCM) in accordance with the present disclosure;

[0041] Figure 7 is block diagram of a second embodiment of a system for Feature Coding for Machines (FCM) in accordance with the present disclosure;

[0042] Figure 8 is block diagram of a third embodiment of a system for Feature Coding for Machines (FCM) in accordance with the present disclosure;

[0043] FIG.9 is a block diagram of a computing system that can be used to implement any one or more of the methodologies disclosed herein and any one or more portions thereof.

[0044] The drawings are not necessarily to scale and may be illustrated by phantom lines, diagrammatic representations and fragmentary views. In certain instances, details that are not necessary for an understanding of the embodiments or that render other details difficult to perceive may have been omitted. DETAILED DESCRIPTION OF DISCLOSED EMBODIMENTS

[0045] The present systems and methods provide improvements over known FCM systems. Exemplary embodiments of the presently disclosed FCM Codec include a calculate distribution properties module / process on the encoder side or edge device and a reconstruction refinement module / process on decoder side, typically at the base receiver.

[0046] Figure 6 is a simplified block diagram illustrating a first exemplary embodiment of an FCM system in accordance with the present disclosure. Referring to Fig.6, the FCM encoder includes a calculate distribution properties module 612 which is placed before the feature reduction module 615, which has been described in connection with Figure 2. At the FCM decoder 635, a reconstruction refinement module 653 is placed after the feature restoration module 650 previously described in connection with Figure 2. It should be noted that the architecture of Fig.6 is suitable for both channel level or layer lever reconstruction refinement or combinations of both.

[0047] The calculate distribution properties module 612 and reconstruction refinement module 653 are described in further detail below. The remaining modules illustrated in Fig.6 within the FCM decoder 635 and FCM encoder 605 operate in substantially the same manner as the corresponding blocks described above in connection with Fig.2.

[0048] The calculate distribution properties module 612 receives the original feature tensors of feature maps 610 as input, calculates properties of the probability distribution of features such as, but not limited to, mean (mu) and standard deviation (sigma), and sends those distribution properties in a bitstream to the decoder. Properties of the probability distribution can be calculated for the features from the last layer of the NN part 1 of a split neural network. Properties such as mu and sigma can be calculated per-channel or per-layer. Calculating properties on a per-layer basis may be advantageous in that only a single value or pair of values needs to be signaled for the whole layer. The probability distribution properties are sent to the decoder as part of the bitstream, in lossless or lossy form, depending on the settings and system preferences. The probability distribution properties of channels or layers can also be derived using the reference frame’s distribution properties. In the case of channel level refinement, a subset of the channel’s distribution property could be sent to the decoder, and the other distribution properties could be derived on the decoder side.

[0049] The reconstruction refinement module 653 uses the signaled probability distribution properties from bitstream to refine the reconstructed feature tensors. Using the distribution properties sent from the FCM encoder 605, the reconstruction refinement module 653 will align the reconstructed tensor’s distribution as close to the original tensor’s distribution as possible. This alignment can be performed on different levels, e.g., layer level or channel level.

[0050] It should also be noted that the refinement could be done for every frame or periodically, depending on the system requirements. In the case of periodic refinement, the distribution properties from reference frames or derived distribution properties using reference frames could be used for distribution alignment of the current frame.

[0051] Reconstruction refinement may be performed on the decoder system using the following two exemplary methods described below. Typically, the input and output of all these methods is usually a tensor (or N-dimensional matrix). Method.1. Using only ^^ of the original distribution: ^^^^^^^^^^ௗ ൌ ^^^^^^^^^^ ൈ ఙ^^^ఙ^^^^^^Eq. 1.^^^^^^^^^^ௗ= refined layer or^^^^^^^^^= reconstructed layer or channel at the decoder side [input] ^^^^^= sigma of the original channel or layer distribution ^^^^^^^= sigma of the reconstructed channel or layer distribution Method.2. Using only (^^,^^) of the original distribution: A)^^^^^^^^^^ௗ= refined layer or channel [output] ^^^^^^^^^= reconstructed layer or channel at the decoder side [input] ^^^^^= sigma of the original channel or layer distribution^^^^^^^= sigma of the reconstructed channel or layer distribution ^^^^^^^ ^^^௧= mu of reconstructed channel or layer distribution after applying Eq.2. ^^^^^= mu of original channel or layer distribution B) Here, ^^^^^^^^^^^^^^^^௬^^ , ^^^^^^^^^^^^^^^^^^^^_^^^^^^^^^^^^௬^^ , ^^^^^^^^^^^^^^_^^^^^^^^^^^^௬^^ are tensors or (N-dimensionalmatrix). Eq. ^4^ Eq. ^5^Further, Eq.2 and Eq.3 set forth above can be applied in any order and in any combination.

[0053] An exemplary distribution after applying Method 1 & Method 2 is shown in Error! Reference source not found., Error! Reference source not found. and Error! Reference source not found.. The graphs in Figures 3 and 4 show histograms of the values in the feature layers overlayed with the fitted distributions. The Figs.3A and 4A are plots showing values of the input, original feature tensors. Figs 3B and 4B illustrate the values of the tensor values after reconstruction. Figs. 3C and 4C are graphs which show values after the present methods of refinement are applied. As evident from Figs 3C and 4C, the refinement transforms the distribution of Figs 3B and 4B into something more closely aligned with the original probability distribution. These figures illustrate that the reconstruction refinement method operates to align the reconstructed tensor’s distribution property as close to the distribution property of the original tensors. Figure 5A and 5B depicts the overlay of all three distributions (original, reconstructed, and refined).

[0054] Figure 7 is a block diagram illustrating a second embodiment of the presently disclosed FCM system. In the embodiment of Figure 7, the calculate distribution properties module 712 is placed after the feature reduction module 715 in the FCM system that has otherwise been described in connection with Figure 2. In this configuration, the calculate distribution properties module is determining the probability distribution property of the reduced feature maps rather than the original feature maps 710 from the neural network front end, i.e., NN Part 1. Further, the reconstruction refinement module 747 may preferably be placed after the inverse feature conversion module 745. The architecture of Figure 7 is suitable for both channel level or layer lever reconstruction refinement or combinations of both.

[0055] Figure 8 is a block diagram of a further embodiment of an FCM system in accordance with the present disclosure which combines the benefits of the embodiments of Fig.6 and Fig. 7. Referring to Fig.8 the calculate distribution properties function is performed by the calculate distribution properties 1 812 at the input of FCM encoder 805 and calculate distribution properties 2 module 817 at the output of the feature reduction module 815. This allows the distribution properties to be determined both before feature reduction 815 and after feature reduction 815. Although this requires additional overhead in the bitstream in that two sets of distribution properties are encoded in the bitstream, this may be advantageous in certain applications since the layers of the features maps (typically five layers each in the original feature maps from NN Part 1) have different distribution properties than the single layer that results from feature reduction module 815 and may provide for further refinement. Similarly, at the decoder side, the reconstruction refinement properties process is divided among reconstruction refinement 1 module 847 and reconstruction refinement 2 module 852. It should be noted that the exemplary embodiment of Figure 8 is also suitable for both channel level or layer lever reconstruction refinement or the combinations of both.

[0056] Embodiments of encoders in accordance with the present disclosure may send layers or channel distribution properties to the decoder and the decoder will use those properties to align the reconstructed layers' or channel distributions' properties close to the original tensor's distribution property.

[0057] The present systems and methods of encoding and decoding FCM transmit an encoded bitstream from an encoder site, typically an edge device, and a decoder site, e.g., typically a base receiver. The present disclosure further provides exemplary bitstream syntax for implementing the present systems and methods.

[0058] The following illustrates a first exemplary bitstream syntax (Bitstream Syntax 1): reconstruction_refinement_informations() { Descriptor refinement period u(1)

[0059] Syntax Elements: ^ refinement_period = how often the reconstruction refinement methods will be applied on the decoder side (unsigned integer: 1 byte) ^ n_refinements = number of refinement elements in bitstream (unsigned integer: variable length) ^ layer_sigma[i] = sigma of the ithlayer’s distribution (floating point: 4 bytes) ^ layer_mu[i] = mu of the ithlayer’s distribution (floating point: 4 bytes)

[0060] [The following illustrates a first exemplary bitstream syntax (Bitstream Syntax 2): reconstruction_refinement_informations() { Descriptor} }^ refinement_period = how often the reconstruction refinement methods will be applied on the decoder side (unsigned integer: 1 byte) ^ n_refinements = number of refinement elements in bitstream (unsigned integer: variable length) ^ layer_sigma[i] = sigma of the ithlayer’s distribution (floating point: 4 bytes)

[0062] The following illustrates a first exemplary bitstream syntax (Bitstream Syntax 3): reconstruction_refinement_informations() { Descriptor refinement period u(1)

[0063] Syntax Elements: ^ refinement_period = how often the reconstruction refinement methods will be applied on the decoder side (unsigned integer: 1 byte) ^ n_refinements = number of refinement elements in bitstream (unsigned integer: variable length) ^ layer_mu[i] = mu of the ithLayer’s distribution (floating point: 4 bytes)

[0064] The following illustrates a first exemplary bitstream syntax (Bitstream Syntax 4): reconstruction_refinement_informations() { Descriptor} }^ refinement_period = how often the reconstruction refinement methods will be applied on the decoder side (unsigned integer: 1 byte) ^ n_refinements = number of refinement elements in bitstream (unsigned integer: variable length) ^ channel _sigma[i][j] = sigma of the jthchannel’s distribution of the ithlayer(floating point: 4 bytes) ^ channel _mu[i][j] = mu of the jthchannel’s distribution of the ithlayer (floating point: 4 bytes)

[0066] The following illustrates a first exemplary bitstream syntax (Bitstream Syntax 5): reconstruction_refinement_informations() { Descriptor refinement period u(1)

[0067] Syntax Elements: ^ refinement_period = how often the reconstruction refinement methods will be applied on the decoder side (unsigned integer: 1 byte) ^ n_refinements = number of refinement elements in bitstream (unsigned integer: variable length) ^ channel _sigma[i][j] = sigma of the jthchannel’s distribution of the ithlayer(floating point: 4 bytes)

[0068] The following illustrates a first exemplary bitstream syntax (Bitstream Syntax 6): reconstruction_refinement_informations() { Descriptorfor (j=0; j<=num_of_channels[i]; j++) { channel_mu[i][j] f(4)^ refinement_period = how often the reconstruction refinement methods will be applied on the decoder side (unsigned integer: 1 byte) ^ n_refinements = number of refinement elements in bitstream (unsigned integer: variable length) ^ channel _mu[i][j] = mu of the jthchannel’s distribution of the ithlayer (floating point: 4 bytes)

[0070] The following illustrates a first exemplary bitstream syntax (Bitstream Syntax 7): reconstruction_refinement_informations() { Descriptor refinement eriod u(1)

[0071] Syntax Elements: ^ refinement_period = how often the reconstruction refinement methods will be applied on the decoder side (unsigned integer: 1 byte) ^ n_refinements = number of refinement elements in bitstream (unsigned integer: variable length) ^ layer_sigma[i] = sigma of the ithlayer’s distribution (floating point: 4 bytes) ^ layer_mu[i] = mu of the ithlayer’s distribution (floating point: 4 bytes) ^ channel _sigma[i][j] = sigma of the jthchannel’s distribution of the ithlayer(floating point: 4 bytes)^ channel _mu[i][j] = mu of the jthchannel’s distribution of the ithlayer (floating point: 4 bytes)

[0072] Some embodiments of the present disclosure may include / and or be embodied by non- transitory computer program products (i.e., physically embodied computer program products) that store instructions, which when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform operations herein. Such non- transitory computer program products (i.e., physically embodied computer program products) may store instructions, which when executed by one or more data processors of one or more computing systems, causes at least one data processor to perform operations, and / or steps thereof described in this disclosure, including without limitation any operations described above and / or any operations decoder 900 and / or encoder 800 may be configured to perform. Similarly, computer systems are also described that may include one or more data processors and memory coupled to the one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. In addition, methods can be implemented by one or more data processors either within a single computing system or distributed among two or more computing systems. Such computing systems can be connected and can exchange data and / or commands or other instructions or the like via one or more connections, including a connection over a network (e.g. the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, or the like), via a direct connection between one or more of the multiple computing systems, or the like.

[0073] It is to be noted that any one or more of the aspects and embodiments described herein may be conveniently implemented using one or more machines (e.g., one or more computing devices that are utilized as a user computing device for an electronic document, one or more server devices, such as a document server, etc.) programmed according to the teachings of the present specification, as will be apparent to those of ordinary skill in the computer art.Appropriate software coding can readily be prepared by skilled programmers based on the teachings of the present disclosure, as will be apparent to those of ordinary skill in the software art. Aspects and implementations discussed above employing software and / or software modules may also include appropriate hardware for assisting in the implementation of the machine executable instructions of the software and / or software module.

[0074] Such software may be a computer program product that employs a machine-readable storage medium. A machine-readable storage medium may be any medium that is capable of storing and / or encoding a sequence of instructions for execution by a machine (e.g., a computing device) and that causes the machine to perform any one of the methodologies and / or embodiments described herein. Examples of a machine-readable storage medium include, but are not limited to, a magnetic disk, an optical disc (e.g., CD, CD-R, DVD, DVD-R, etc.), a magneto-optical disk, a read-only memory “ROM” device, a random-access memory “RAM” device, a magnetic card, an optical card, a solid-state memory device, an EPROM, an EEPROM, and any combinations thereof. A machine- readable medium, as used herein, is intended to include a single medium as well as a collection of physically separate media, such as, for example, a collection of compact discs or one or more hard disk drives in combination with a computer memory. As used herein, a machine-readable storage medium does not include transitory forms of signal transmission.

[0075] Such software may also include information (e.g., data) carried as a data signal on a data carrier, such as a carrier wave. For example, machine-executable information may be included as a data-carrying signal embodied in a data carrier in which the signal encodes a sequence of instruction, or portion thereof, for execution by a machine (e.g., a computing device) and any related information (e.g., data structures and data) that causes the machine to perform any one of the methodologies and / or embodiments described herein.

[0076] Examples of a computing device include, but are not limited to, an electronic book reading device, a computer workstation, a terminal computer, a server computer, a handheld device (e.g., a tablet computer, a smartphone, etc.), a web appliance, a network router, a network switch, a network bridge, any machine capable of executing a sequence of instructions that specify an action to be taken by that machine, and any combinations thereof. In one example, a computing device may include and / or be included in a kiosk.

[0077] FIG.9 shows a diagrammatic representation of one embodiment of a computing device in the exemplary form of a computer system 900 within which a set of instructions for causing a control system to perform any one or more of the aspects and / or methodologies of the present disclosure may be executed. It is also contemplated that multiple computing devices may be utilized to implement a specially configured set of instructions for causing one or more of the devices to perform any one or more of the aspects and / or methodologies of the present disclosure. Computer system 900 includes a processor 904 and a memory 908 that communicate with each other, and with other components, via a bus 912. Bus 912 may include any of several types of bus structures including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combinations thereof, using any of a variety of bus architectures.

[0078] Processor 904 may include any suitable processor, such as without limitation a processor incorporating logical circuitry for performing arithmetic and logical operations, such as an arithmetic and logic unit (ALU), which may be regulated with a state machine and directed by operational inputs from memory and / or sensors; processor 904 may be organized according to Von Neumann and / or Harvard architecture as a non-limiting example. Processor 904 may include, incorporate, and / or be incorporated in, without limitation, a microcontroller, microprocessor, digital signal processor (DSP), Field Programmable Gate Array (FPGA), Complex Programmable Logic Device (CPLD), Graphical Processing Unit (GPU), generalpurpose GPU, Tensor Processing Unit (TPU), analog or mixed signal processor, Trusted Platform Module (TPM), a floating-point unit (FPU), and / or system on a chip (SoC).

[0079] Memory 908 may include various components (e.g., machine-readable media) including, but not limited to, a random-access memory component, a read only component, and any combinations thereof. In one example, a basic input / output system 916 (BIOS), including basic routines that help to transfer information between elements within computer system 900, such as during start-up, may be stored in memory 908. Memory 908 may also include (e.g., stored on one or more machine-readable media) instructions (e.g., software) 920 embodying any one or more of the aspects and / or methodologies of the present disclosure. In another example, memory 908 may further include any number of program modules including, but not limited to, an operating system, one or more application programs, other program modules, program data, and any combinations thereof.

[0080] Computer system 900 may also include a storage device 924. Examples of a storage device (e.g., storage device 924) include, but are not limited to, a hard disk drive, a magnetic disk drive, an optical disc drive in combination with an optical medium, a solid-state memory device, and any combinations thereof. Storage device 924 may be connected to bus 912 by an appropriate interface (not shown). Example interfaces include, but are not limited to, SCSI, advanced technology attachment (ATA), serial ATA, universal serial bus (USB), IEEE 1394 (FIREWIRE), and any combinations thereof. In one example, storage device 924 (or one or more components thereof) may be removably interfaced with computer system 900 (e.g., via an external port connector (not shown)). Particularly, storage device 924 and an associated machine-readable medium 928 may provide nonvolatile and / or volatile storage of machine- readable instructions, data structures, program modules, and / or other data for computer system 900. In one example, software 920 may reside, completely or partially, within machine-readable medium 928. In another example, software 920 may reside, completely or partially, within processor 904.

[0081] Computer system 900 may also include an input device 932. In one example, a user of computer system 900 may enter commands and / or other information into computer system 900 via input device 932. Examples of an input device 932 include, but are not limited to, an alpha- numeric input device (e.g., a keyboard), a pointing device, a joystick, a gamepad, an audio input device (e.g., a microphone, a voice response system, etc.), a cursor control device (e.g., a mouse), a touchpad, an optical scanner, a video capture device (e.g., a still camera, a video camera), a touchscreen, and any combinations thereof. Input device 932 may be interfaced to bus 912 via any of a variety of interfaces (not shown) including, but not limited to, a serial interface, a parallel interface, a game port, a USB interface, a FIREWIRE interface, a direct interface to bus 912, and any combinations thereof. Input device 932 may include a touch screen interface that may be a part of or separate from display 936, discussed further below. Input device 932 may be utilized as a user selection device for selecting one or more graphical representations in a graphical interface as described above.

[0082] A user may also input commands and / or other information to computer system 900 via storage device 924 (e.g., a removable disk drive, a flash drive, etc.) and / or network interface device 940. A network interface device, such as network interface device 940, may be utilized for connecting computer system 900 to one or more of a variety of networks, such as network 944, and one or more remote devices 948 connected thereto. Examples of a network interface device include, but are not limited to, a network interface card (e.g., a mobile network interface card, a LAN card), a modem, and any combination thereof. Examples of a network include, but are not limited to, a wide area network (e.g., the Internet, an enterprise network), a local area network (e.g., a network associated with an office, a building, a campus or other relatively small geographic space), a telephone network, a data network associated with a telephone / voiceprovider (e.g., a mobile communications provider data and / or voice network), a direct connection between two computing devices, and any combinations thereof. A network, such as network 944, may employ a wired and / or a wireless mode of communication. In general, any network topology may be used. Information (e.g., data, software 920, etc.) may be communicated to and / or from computer system 900 via network interface device 940.

[0083] Computer system 900 may further include a video display adapter 952 for communicating a displayable image to a display device, such as display device 936. Examples of a display device include, but are not limited to, a liquid crystal display (LCD), a cathode ray tube (CRT), a plasma display, a light emitting diode (LED) display, and any combinations thereof. Display adapter 952 and display device 936 may be utilized in combination with processor 904 to provide graphical representations of aspects of the present disclosure. In addition to a display device, computer system 900 may include one or more other peripheral output devices including, but not limited to, an audio speaker, a printer, and any combinations thereof. Such peripheral output devices may be connected to bus 912 via a peripheral interface 956. Examples of a peripheral interface include, but are not limited to, a serial port, a USB connection, a FIREWIRE connection, a parallel connection, and any combinations thereof.

[0084] The foregoing has been a detailed description of illustrative embodiments of the invention. Various modifications and additions can be made without departing from the spirit and scope of this invention. Features of each of the various embodiments described above may be combined with features of other described embodiments as appropriate in order to provide a multiplicity of feature combinations in associated new embodiments. Furthermore, while the foregoing describes a number of separate embodiments, what has been described herein is merely illustrative of the application of the principles of the present invention. Additionally, although particular methods herein may be illustrated and / or described as being performed in a specific order, the ordering is highly variable within ordinary skill to achieve methods,systems, and software according to the present disclosure. Accordingly, this description is meant to be taken only by way of example, and not to otherwise limit the scope of this invention.

[0085] Exemplary embodiments have been disclosed above and illustrated in the accompanying drawings. It will be understood by those skilled in the art that various changes, omissions and additions may be made to that which is specifically disclosed herein without departing from the spirit and scope of the present invention.

Claims

What is Claimed is:

1. An encoder for feature coding for machines (FCM), comprising: a calculate distribution properties module receiving a plurality of feature maps output from a split neural network front end, at least one feature map having at least one feature tensor, the calculate distribution properties module receiving at least one feature tensor and determining at least one probability distribution property therefrom; a feature reduction module coupled to the calculate distribution properties module and receiving the at least one feature map and at least one distribution property, and generating at least one reduced feature map representing the original plurality of feature maps; a feature conversion module receiving the reduced feature maps and converting the reduced feature maps and probability distribution properties to a video format; and an inner encoder receiving the video format signal from the feature conversion module and generating an encoded bitstream representing the plurality of feature maps and probability distribution properties.

2. The encoder of claim 1, wherein the probability distribution property is a mean value of feature tensors in a feature map.

3. The encoder of claim 1, wherein the probability distribution property is a standard deviation of feature tensors in a feature map.

4. The encoder of claim 1, wherein the probability distribution properties are calculated on a per layer basis.

5. The encoder of claim 1, wherein the probability distribution properties are calculated on a per channel basis.

6. The encoder of claim 1, wherein a second calculate distribution properties module is provided after the feature reduction module and generating at least one probability distribution property of the reduced feature map.

7. An encoder for feature coding for machines (FCM), comprising: a feature reduction module receiving a plurality of feature maps and generating at least one reduced feature map representing the plurality of feature maps; a calculate distribution properties module receiving the reduced feature maps, at least one reduced feature map having at least one feature tensor, the calculatedistribution properties module receiving at least one feature tensor and determining at least one probability distribution property therefrom; a feature conversion module coupled to the calculated distribution properties module and converting the reduced feature map and at last one probability distribution property to a video format; and an inner encoder receiving the video format signal from the feature conversion module and generating an encoded bitstream representing the plurality of feature maps and probability distribution properties.

8. The encoder of claim 1, wherein the probability distribution property is a mean value of feature tensors in the reduced feature map.

9. The encoder of claim 1, wherein the probability distribution property is a standard deviation of feature tensors in the reduced feature map.

10. The encoder of claim 1, wherein the probability distribution properties are calculated on a per layer basis.

11. The encoder of claim 1, wherein the probability distribution properties are calculated on a per channel basis.

12. The encoder of claim 1, wherein: the probability distribution property is one of a mean value or a standard deviation of feature tensors in the reduced feature map; and the probability distribution properties are calculated on one of a per layer basis, a per channel basis, or both a per layer basis and per channel basis.

13. The encoder of claim 12, further comprising a split neural network front end, the split neural network front end receiving an input image signal and generating a plurality of feature maps representing the input image signal; 14. An encoding system for feature coding for machines (FCM), comprising: A split neural network front end, the split neural network front end receiving an input image signal and generating a plurality of feature maps representing the input image signal; a calculate distribution properties module receiving a plurality of feature maps, at least one feature map having at least one feature tensor, the calculate distribution properties module receiving at least one feature tensor and determining at least one probability distribution property therefrom;a feature reduction module coupled to the calculate distribution properties module and receiving the at least one feature map and at least one distribution property, and generating a reduced number of feature maps representing the original plurality of feature maps; a feature conversion module receiving the reduced feature maps and converting the reduced feature maps and probability distribution properties to a video format; and an inner encoder receiving the video format signal from the feature conversion module and generating an encoded bitstream representing the plurality of feature maps and probability distribution properties.

15. A decoder for feature coding for machines (FCM), comprising: an inner decoder receiving a coded bitstream including at least one reduced feature map representing feature maps output from a neural network front end at an encoder and at least one probability distribution property of the feature map and providing a decoded video signal representing the at least one reduced feature map and probability distribution property; an inverse feature conversion module receiving the decoded video signal and providing feature tensors of layers of at least one feature maps; a feature restoration module receiving feature tensors of the reduced feature map and generating tensors for a plurality of feature maps representing a signal processed by a neural network front end used an encoder site from the reduced feature map; and a reconstruction refinement module receiving the probability distribution property from the bitstream and the tensors for a plurality of restored feature maps from the feature restoration module and aligning a probability distribution property of the restored feature maps with the at least one probability distribution property in the bitstream.

16. The decoder of claim 15, wherein the probability distribution property is a mean value of feature tensors in the reduced feature map.

17. The decoder of claim 15, wherein the probability distribution property is a standard deviation of feature tensors in the reduced feature map.

18. The decoder of claim 15, wherein the probability distribution properties are calculated on a per layer basis.

19. The decoder of claim 15, wherein the probability distribution properties are calculated on a per channel basis.

20. The decoder of claim 15 further comprising a second reconstruction refinement module interposed between the inverse feature conversion module and the feature restoration module.

21. The decoder of claim 15, wherein reconstruction refinement is performed on a periodic basis.

22. The decoder of claim 15, wherein reconstruction refinement is performed on every frame in the bitstream.

Images