Systems, methods and bitstreams for feature coding for machine applications with bit-shift

Abstract

Systems and methods are provided to improve encoding and decoding of video in a system for feature coding for machines (FCM). At an encoder site, a right shift operation is performed on tensors of feature maps representing the video signal, removing n least significant bits from each tensor. This reduces the required bandwidth of the resulting encoded bitstream. At a decoder site, a left shift operation is performed on the decoded and reconstructed tensor values to restore the values of the most significant bits of the tensors while replacing the removed LSBs not considered critical to a machine task with zeroes.

Description

SYSTEMS, METHODS AND BITSTREAMS FOR FEATURE CODING FOR MACHINE APPLICATIONS WITH BIT-SHIFT CROSS REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims the benefit of priority to U.S. Provisional Application serial no.63 / 634,426 filed on April 15, 2024, and entitled SYSTEMS AND METHODS FOR BIT-SHIFT FOR FEATURE CODING FOR MACHINES (FCM), the entirety of which is hereby incorporated by reference in its entirety. BACKGROUND OF THE DISCLOSURE

[0002] As the number and scale of deployed video sensors / devices increases, an increasing amount of video is expected to be processed by machines and for machine use rather than human viewing. Indeed, an exemplary system or solution employing thousands of cameras would produce significant amounts of video that cannot be monitored by humans in a cost- effective manner. Because of the significant volume of the data, an efficient compression system is desirable for Machine-to-Machine (M2M) communication. To standardize the coding for machines to facilitate a Machine-to-Machine (M2M) communication system more efficiently, MPEG issued a Call for Proposal (CfP) for Coding for Machines in 2022.

[0003] Currently, M2M communications are typically managed by two distinct systems, namely, edge computing or local computing systems and remote computing or Video Coding for Machines (VCM) systems. In the context of edge computing systems, the complete Convolutional Neural Network (CNN) is typically executed on edge devices. However, end devices often lack the computational capacity to run extensive networks, requiring billions of operations for inference result calculations. In contrast, in Video Coding for Machine systems, the video is typically compressed initially and transmitted to a base server or cloud server with more computing resources to execute the full CNN. The calculated results are then subsequently relayed back to the edge devices. However, Video Coding for machine systemsmay not be able to support compute offload and may fail to fully leverage the potential of end devices, as they execute the complete CNN on the base server. To overcome these limitations for M2M communication and to enable collaborative intelligence and split computing for M2M connections, MPEG has issued a Call for Proposal (CfP) for Feature Coding for Machines (FCM) technology in 2023.

[0004] Figure 1 is a simplified block diagram providing a high-level illustration of a Feature Coding for Machine (FCM) system. A resource limited device receives input video 110 and uses a split neural network part 1115 (also referred to herein as NN front end or head) to extract features of the video which are encoded by FCM encoder 120. The encoded features are transmitted over a channel to a base server which decodes the features. The base server includes an FCM decoder 130 which provides decoded feature maps 135 to a neural network part 2140 (also referred to herein as NN back end or tail) which provides output to inference results 145.

[0005] Referring to Figure 1, in Feature Coding for Machine (FCM) systems, a Neural Network is first split into two parts, i.e., NN Part 1 (Split CNN - Head) 115 and NN Part 2 (Split CNN - Tail) 140. Instead of running the full CNN on the end devices, only the NN Part 1115 is executed on the resource-limited devices, and the NN Part 2140 is preferably executed on the base server which is expected to have more computing resources. Since the NN Part 1 115 is generally executed on the edge devices, instead of transmitting videos to the base server (like in Video Coding for Machine), the intermediate feature data extracted from NN Part 1 is sent to the base server. However, the volume of intermediate feature data is usually much larger than the video itself and needs efficient compression before sending it to the base server. An example of such a system is depicted in Error! Reference source not found..

[0006] Referring to Fig. 2, which is a block diagram further illustrating an exemplary FCM system, an FCM Encoder 205 which receives feature maps 210 from an NN part 1115 consistsof three main modules, e.g., feature reduction 220, feature conversion 230, and inner codec 240 for encoding sequences.

[0007] The FCM architecture employs a split neural network, typically a CNN. In the context of CNNs, input data, output data and transformations with the neural network are generally represented using tensors. A tensor is the primary data structure used by a CNN and is a mathematical representation of a concept in the form of an n-dimensional array. For example, tensors can represent concepts such as scalar (0 dimensional tensor), vector (1-dimensional tensor), matrix (2-dimensional tensor), and an nd-array (n-dimensional tensor). Typical tensors occurring in the CNNs are 3-dimensional, e.g. [c, w, h], where c represents the number of channels, and w and h represent the width and height of each channel.

[0008] As noted in Figure 1, the FCM encoder provides an encoded bitstream over a communication link (e.g., wired or wireless communication link) to a base receiver including an FCM decoder. Referring to Fig. 2, FCM decoder 275 consists of three principal modules, e.g., an inner codec 260, inverse feature conversion 270, and feature restoration 280 for decoding the sequences with feature maps 290. The input of the FCM encoder 205 is intermediate feature data (^^௧) extracted from NN Part 1115 (Fig.1), and the output of the FCM decoder 255 is reconstructed intermediate feature data (^^௧ᇱ), as shown in Error! Reference source not found., which is applied to NN Part 2140 (Fig. 1). After reconstruction of the intermediate feature data (^^௧ᇱ), the NN Part 2 is executed to generate the inference results. Feature data usually consists of multiple layers. And each layer usually consists of multiple channels. For instance, in one example, the backbone of detectron2 (NN Part 1) may consist of 4 layers and each layer consists of 256 channels. The details of all the FCM Encoder and Decoder modules are described in more detail below.

[0009] The feature reduction module 220 of the FCM encoder 205 takes original feature maps 210 and / or layers as input and outputs a reduced number of feature maps and / or layersrepresenting the original data. Essentially feature reduction module 220 takes the data from many layers and conveys it as a single layer. The feature reduction module 220 can be a neural network, auto-encoder, or just classical statistical approaches like PCA or clustering methods to reduce the number of feature maps and / or layers.

[0010] The feature conversion module 230 of the FCM encoder 205 receives reduced feature maps and / or layers as input and converts the feature tensor data from floating point to unsigned integers, such as 8bit or 10bit integers. In some cases, depending on the data, it may also pack the feature maps into one frame. Then, the packed frame or converted feature maps and / or layers are converted by the feature conversion module to a suitable video format, such as YUV 4:0:0, before sending it to the inner codec 240.

[0011] The Inner Codec (Encoder) module 240 takes the video, such as YUV 4:0:0, representing the feature maps as input and outputs a bitstream to transmit over a communication channel 250 to the receiver 255. The Inner Codec (Encoder) 240 could be a Neural Network encoder or any traditional encoder like HEVC, VVC, VVenC, or a combination of both Neural Network encoder and traditional encoder.

[0012] Turning to the FCM decoder 255, the Inner Codec (Decoder) module 260 receives the encoded bitstream as input and decompresses the bitstream file to a suitable format, such as YUV 4:0:0. Like the inner codec (encoder) module 240, the inner codec (decoder) 260 can be implemented by a Neural Network based decoder or any traditional decoder like HEVC, VVC, VVdeC or the combination of both Neural Network based decoder and traditional decoder.

[0013] The inverse feature conversion module 270 takes the decoded YUV 4:0:0 from Inner Codec 260 as input and unpacks (if necessary) and converts the unsigned integers to floating point values.

[0014] The feature restoration module 280 receives feature tensors from the inverse feature conversion module 270 as input and restores the feature tensors from reduced size to theoriginal size. Like the feature reduction module 220, the feature restoration module 280 can also be a Neural Network, Auto-Decoder, or just classical statistical approaches like Inverse PCA or de-clustering to restore the original size of feature maps and layers.

[0015] Documents which may provide relevant background information for the present disclosure include: [1] Yuan Zhang, Manouchehr Rafie, Shan Liu, Christopher Hollmann, “[N00164] Call for Proposals on Video Coding for Machines.” ISO / IEC JTC 1 / SC 29 / WG 2, Jan. 2022; [2] C. Rosewarne (Canon) and Y. Zhang (China Telecom), “[N00282] Call for Proposals on Feature Compression for Video Coding for Machines.” ISO / IEC JTC 1 / SC 29 / WG 2, Jan. 2023; [3] WG 04 MPEG Video coding, “[N00460] Algorithm description of FCTM.” Jan. 2024; and [4] Y. Wu, A. Kirillov, F. Massa, W. Y. Lo, and R. Girshick, “Detectron2,” 2019, Feb.2024 (Online). SUMMARY OF THE DISCLOSURE

[0016] Systems and methods for improved FCM encoding and decoding are provided. Examples of embodiments, features and advantages of the presently disclosed encoder and decoder embodiments include, but are not limited to, the following summary.

[0017] In one exemplary embodiment, an encoder for feature coding for machines (FCM) is disclosed which includes a split neural network part 1 configured to receive an input video signal and generate at least one feature map representing the video signal. A feature reduction module receives the at least one feature map and generates at least one reduced feature map representing the video signal. A feature conversion module receives the reduced feature maps and converts tensors of the reduced feature maps from a floating point representation to a fixed integer video format. A right bit shift module coupled to the feature conversion module and performing a right shift of the converted feature tensors by a shift value, n. A video encoder is coupled to the right bit shift module, the video encoder receives the shifted feature tensors and generates an encoded bitstream.

[0018] In some embodiments, the value of n is a user provided value. In certain embodiments, the encoder further includes a rate distortion optimization loop which tests multiple values of n and selects the value of n providing optimal rate-distortion performance.

[0019] In some embodiments, the encoder may include a calculate distribution properties module receiving the at least one feature map output from a split neural network front end, and at least one feature map having at least one feature tensor. The calculate distribution properties module receives at least one feature tensor and determines at least one probability distribution property therefrom, the at least one probability distribution property being provided to the video encoder to be signaled in the encoded bitstream.

[0020] In certain embodiments, the calculate distribution properties module may further receive the reduced feature map from the feature reduction module and determine at least one probability distribution property of feature tensors of the reduced feature map and provide the probability distribution property of the reduced feature map to the video encoder to be signaled in the encoded bitstream.

[0021] In certain embodiments, the calculate distribution properties module may receive the reduced feature map from the feature reduction module and determine at least one probability distribution property of feature tensors of the reduced feature map and provide the probability distribution property of the reduced feature map to the video encoder to be signaled in the encoded bitstream.

[0022] In the various encoder embodiments of the present disclosure, the probability distribution property can be a mean value of feature tensors in a feature map.

[0023] In the various encoder embodiments of the present disclosure, the probability distribution property can include a standard deviation of feature tensors in a feature map.

[0024] In the various encoder embodiments of the present disclosure, the probability distribution properties can be calculated on a per layer basis. Alternatively, the probability distribution properties can be calculated on a per channel basis.

[0025] In some embodiments, the probability distribution property includes a mean value and a standard deviation of feature tensors in the reduced feature map. The probability distribution properties can be calculated on one of a per-layer basis, a per-channel basis, or both a per-layer basis and per-channel basis.

[0026] In a further embodiment of the present disclosure a decoder includes an inner video decoder receiving a coded bitstream which includes at least one reduced feature map representing feature maps output from a neural network front end at an encoder. The video decoder providing a decoded video signal with tensors in integer format representing the at least one reduced feature map. A left shift module is coupled to the decoder and receives the integer format tensors and a shift value n signaled in the encoded bitstream. The left shift module performs a left shift operation of n bits on each tensor. An inverse feature conversion module receives the left shifted feature tensors of the decoded video signal and converts the left shifted feature tensors to a floating point format of at least one feature map. A feature restoration module receives the converted floating point tensors of the reduced feature map and reconstructs tensors for a plurality of feature maps representing a signal processed by a neural network front end used an encoder site from the reduced feature map. A NN part 2 of a split neural network receives the reconstructed feature maps and performs an inference operation.

[0027] In certain embodiments the encoded bitstream includes distribution properties of tensors of a reduced feature map, and the decoder further comprises a first reconstruction refinements module interposed between the inverse feature conversion module and the feature restoration module, the first reconstruction refinements module aligning a probabilitydistribution property of the decoded tensors of the reduced feature maps with the probability distribution property encoded in the bitstream.

[0028] In certain embodiments, the encoded bitstream includes distribution properties of tensors of at least one original feature map, and the decoder further comprises a second reconstruction refinements module after the feature restoration module, the first reconstruction refinements module the aligning a probability distribution property of the reconstructed tensors with the probability distribution property encoded in the bitstream.

[0029] In the various decoder embodiments, the probability distribution property includes a mean value of feature tensors in the reduced feature map. The probability distribution property may also include a standard deviation of feature tensors in the reduced feature map.

[0030] In the various decoder embodiments disclosed herein, the probability distribution properties can be calculated on a per layer basis. Alternatively, the probability distribution properties can be calculated on a per channel basis.

[0031] In the various decoder embodiments disclosed herein, reconstruction refinement is performed on a periodic basis. Alternatively, reconstruction refinement can be performed on every frame in the bitstream. BRIEF DESCRIPTION OF THE FIGURES

[0032] For the purpose of illustrating the invention, the drawings show aspects of one or more embodiments of the invention. However, it should be understood that the present invention is not limited to the precise arrangements and instrumentalities shown in the drawings, wherein:

[0033] Figure 1 is a simplified block diagram of a system for Feature Coding for Machines (FCM);

[0034] Figure 2 is block diagram of a system for Feature Coding for Machines (FCM) of Fig. 1 and further illustrating the components of the encoder and decoder systems;

[0035] Figure 3 is block diagram of a first embodiment of a system for Feature Coding for Machines (FCM) in accordance with the present disclosure;

[0036] Figure 4 is block diagram of a second embodiment of a system for Feature Coding for Machines (FCM) in accordance with the present disclosure;

[0037] Figure 5 is a block diagram of a computing system that can be used to implement any one or more of the methodologies disclosed herein and any one or more portions thereof.

[0038] The drawings are not necessarily to scale and may be illustrated by phantom lines, diagrammatic representations and fragmentary views. In certain instances, details that are not necessary for an understanding of the embodiments or that render other details difficult to perceive may have been omitted. DETAILED DESCRIPTION OF DISCLOSED EMBODIMENTS

[0039] The present systems and methods provide improvements over known systems of feature coding for machine to machine based applications (FCM). Exemplary embodiments of the presently disclosed FCM Codec include a calculate distribution properties module / process on the encoder side or edge device and a reconstruction refinement module / process on decoder side, typically at the base receiver. The empbodiments disclosed herein preferably include a right shift process at the encoder and a similar right shift process at the decoder to reduce bandwith of the coded bitstream without substantially degrading machine performance at the decoder side.

[0040] Figure 3 is a simplified block diagram illustrating a first exemplary embodiment of an FCM system in accordance with the present disclosure. Referring to Fig.3, the FCM encoder includes a NN Part 1305 generating feature maps 310, as discussed in connection with Figs.1 and 2. The FCM encoder further includes a feature reduction module 320, a feature conversion module 330, and an inner codec 340, which are substantially the same as described inconnection with Fig.2. The FCM encoder further includes a right bit shift module 335 which is preferably interposed between the feature conversion module 330 and the inner codec 340.

[0041] The right shift module 335 receives converted feature tensors from the feature conversion module 330 and a bit-shift value n as inputs. The right shift module shifts to converted feature tensors values (such as in YUV 4:0:0 format) using a binary shift to the right by the value of n. This operation has the effect of removing the n LSBs and replacing the n MSB’s with zeroes and effectively dividing each value by 2^n such that the resulting bitstream can be significantly reduced for certain machine tasks where the LSBs are insignificant to the process and, therefore, are not sensitive to the right shift operation.

[0042] The value n is an encoder parameter that is set by the user and will be determined for a particular machine task. This may include iteratively running tests on sample data with varying values of n or determining the value of n using a rate-distortion optimization loop in which several values of n are probed and the value of n resulting in the best rate-distortion performance is selected.

[0043] The bit-shift operation may be applied to all the layers of the Neural Network or on different subsets of the layers of the Neural Network. It should also be noted that the bit-shift operation can also be applied to different subsets of the channels from a selected layer of the Neural Network. The bit-shifted feature tensors are provided to the inner codec 340 and provided to the decoder via an encoded bitstream.

[0044] The FCM decoder of Fig. 3 includes an inner codec (decoder) 360, reverse feature conversion module 370, and feature restoration module 380, which provides reconstructed feature maps 390 for the NN Part 2395, substantially as described in like-numbered elements in connection with Fig.2. The FCM decoder further includes a left bit shift module 365 which is preferably interposed between the inner codec 360 and inverse feature conversion module 370. The left shift module receives the bit shifted tensors and the shift value n from the decodedbitstream and left shifts the bits of the tensors by the shift value, n, thereby restoring the original MSBs of the feature tensors (while replacing the n MSBs with zeroes) output from feature conversion module 330 and the encoder.

[0045] Figure 4 is a block diagram illustrating a second embodiment of the presently disclosed FCM system which includes a bit shift operation and reconstruction refinement.

[0046] At the FCM encoder, the system includes feature maps 410 output from a NN Part 1 (not shown in Fig.4) which are applied to a feature reduction module 420, feature conversion module 430, right bit shift module 435 receiving a shift value n 433, and inner codec 440, substantially described in connection with Figure 3.

[0047] The FCM encoder further includes a calculate distribution properties module 425 which receives the original feature tensors of feature maps 410 as input via optional module 415 and the reduced feature tensors from feature reduction module 420. The calculate distribution properties module 425 calculates properties of the probability distribution of features such as, but not limited to, mean (mu) and standard deviation (sigma), and sends those distribution properties in a bitstream to the decoder. It will be appreciated that other statistical distributions, such as a root mean square (RMS), may also be used. The properties may be calculated for the reduced feature set, “reduced_ft” and / or calculated for the original features, “orig_ft”. Properties of the probability distribution can be calculated for the features from one or more layers of the NN part 1, and preferably, are calculated for the last layer of the NN part 1 of a split neural network. Properties such as mu and sigma can be calculated on a per-channel or per-layer basis. Calculating properties on a per-layer basis may be advantageous in that only a single value or pair of values needs to be signaled for the entire layer.

[0048] The probability distribution properties determined by the calculate distribution properties module 425 are preferably sent to the decoder 460 as part of the bitstream, in either a lossless or lossy form, depending on the settings and system preferences. The probabilitydistribution properties of channels or layers can also be derived using the reference frame’s distribution properties. In the case of channel level refinement, a subset of the channel’s distribution properties could be sent to the decoder, and the other distribution properties may be derived by the FCM decoder.

[0049] The FCM decoder of Fig. 4 includes an inner codec 460, left bit shift module 465 receiving shift value n 463 from the bitstream, inverse feature conversion module 470, and a feature restoration module 480, substantially as described above in like-numbered elements in connection with Fig. 3. The FCM decoder of Fig. 4 further includes a reconstruction refinements module 475 interposed between the inverse feature conversion module 470 and feature restoration module 480 and an optional second reconstruction refinements module 485 coupled to the output of feature restoration module 480 and providing the reconstructed feature maps 490 to be applied to a NN part 2 for inferencing.

[0050] The reconstruction refinement module 475 uses the signaled probability distribution properties from bitstream to refine the reconstructed reduced feature tensors. Using the distribution properties for the reduced feature maps, “feature_ft” calculated by the calculated distribution properties module 425 and provided in the bitstream, the reconstruction refinement module 475 will operate to align the reconstructed tensor’s distribution at the decoder to match the reduced tensor’s distribution at the encoder. This alignment can be performed on different levels, e.g., layer level or channel level.

[0051] Reconstruction refinements module 485 optionally uses the signaled probability distribution properties of the original feature maps from the bitstream to refine the reconstructed feature tensors. Using the distribution properties for the original feature maps, “org_ft”, calculated by the calculated distribution properties module 425 and signaled in the bitstream, the reconstruction refinement module 475 operates to align the reconstructedtensor’s distribution as close to the original tensor’s distribution as possible. This alignment can be performed on different levels, e.g., layer level or channel level.

[0052] It should also be noted that the refinement could be done for every frame or periodically, depending on the system requirements. In the case of periodic refinement, the distribution properties from reference frames or derived distribution properties using reference frames could be used for distribution alignment of the current frame.

[0053] Reconstruction refinement may be performed on the decoder system using the following two exemplary methods described below. Typically, the input and output of all these methods is usually a tensor (or N-dimensional matrix). Method.1. Using only ^^ of the original distribution: ^^^^^^^^^^ௗ ൌ ^^^^^^^^^^ ൈఙ^^^ఙ^^^^^^ Eq.1. ^^^^^^^^^^ௗ= refined layer^^^^^^^^^= reconstructed layer or channel at the decoder side [input] ^^^^^= sigma of the original channel or layer distribution ^^^^^^^= sigma of the reconstructed channel or layer distributionMethod.2. Using only (^^,^^) of the original distribution: A) ^^^^^^^^^^ௗ ൌ ^^^^^^^^^^ ൈఙ^^^ఙ^^^^^^ Eq.2.^^^^^^^^^^ௗ= refined layer or channel [output] ^^^^^^^^^= reconstructed layer or channel at the decoder side [input] ^^^^^= sigma of the original channel or layer distribution ^^^^^^^= sigma of the reconstructed channel or layer distribution ^ ^^^^^^^ ^^௧= mu of reconstructed channel or layer distribution after applying Eq.2.^^^^^= mu of original channel or layer distributionB) Here, ^^^^^^^^^^^^^^^^௬^^, ^^^^^^^^^^^^^^^^^^^^_^^^^^^^^^^^^௬^^ , ^^^^^^^^^^^^^^_^^^^^^^^^^^^௬^^ are tensors or (N-dimensional matrix).where: ^^^^^^^^^^ ^^^^^^^^^^^^ : Mean of the layer during reconstruction.^^^^^^^^^^ ^^^^^^^^^^^^ : Standard deviation of the layer during reconstruction.ftensorlayer: The feature tensor corresponding to the layer ^^^^^^ ^^^^^^^^^^^^: Original mean of the layer.^^^^^^ ^^^^^^^^^^^^: Original standard deviation of the layer.normalized_layerlayer : The normalized layer obtained by subtracting the mean and dividing by the standard deviation during reconstruction. refined_layerslayer: The final refined layer obtained by multiplying the normalized layer by the original standard deviation and adding the original mean.

[0054] Both Method 1 and Method.2 can be applied at either the channel level or layer level. Further, Eq.2 and Eq.3 set forth above can be applied in any order and in any combination.

[0055] Embodiments of encoders in accordance with the present disclosure may send layers or channel distribution properties to the decoder and the decoder will use those properties to align the reconstructed layers' or channel distributions' properties close to the original tensor's distribution property.

[0056] The present systems and methods of encoding and decoding FCM transmit an encoded bitstream from an encoder site, typically an edge device, and a decoder site, e.g., typically abase receiver. The present disclosure further provides exemplary bitstream syntax for implementing the present systems and methods.

[0057] The following illustrates a first exemplary bitstream syntax suitable for implementing the bit-shift processing of the current embodiments. (Bitstream Syntax 1): Examplary Bitstream Syntax 1 Descriptor (in bits)

[0058] A description of the syntax elements used above include: ^ is_apply_bitshift (unsigned integer: 1 bit) = flag to signal whether the bitshift is operation is applied ^ n_bitshift (unsigned integer: 2 bit) = how many bits is right shifted by the encoder (i.e., the value of n) Descriptor} isapplyonorgft u(1)Descriptor} }^ is_apply_bitshift (unsigned integer: 1 bit): flag to signal whether the bitshift is applied or not ^ n_bitshift (unsigned integer: 2 bit): how many bit is right shifted by the encoder ^ is_apply_on_org_ft: flag to signal whether the reconstruction refinement will be applied to the original feature tensor or not (unsigned integer: 1 bit) ^ refinement_period_for_org_ft: how often reconstruction refinement will be applied to the original feature tensor (unsigned integer: 8 bits) ^ is_apply_on_reduced_ft: flag to signal whether the reconstruction refinement will be applied to the reduced feature tensor or not (unsigned integer: 1 bit) ^ refinement_period_for_reduced_ft: how often reconstruction refinement will be applied to the reduced feature tensor (unsigned integer: 8 bits) ^ poc : picture of coding ^ layer_sigma[i]: sigma of the ith layer’s distribution (floating point: 32 bits) ^ layer_mu[i] : mu of the ith layer’s distribution (floating point: 32 bits)

[0060] Some embodiments of the present disclosure may include / and or be embodied by non- transitory computer program products (i.e., physically embodied computer program products) that store instructions, which when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform operations herein. Such non- transitory computer program products (i.e., physically embodied computer program products) may store instructions, which when executed by one or more data processors of one or morecomputing systems, causes at least one data processor to perform operations, and / or steps thereof described in this disclosure, including without limitation any operations described above and / or any operations decoder and / or encoder may be configured to perform. Similarly, computer systems are also described that may include one or more data processors and memory coupled to the one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. In addition, methods can be implemented by one or more data processors either within a single computing system or distributed among two or more computing systems. Such computing systems can be connected and can exchange data and / or commands or other instructions or the like via one or more connections, including a connection over a network (e.g. the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, or the like), via a direct connection between one or more of the multiple computing systems, or the like.

[0061] It is to be noted that any one or more of the aspects and embodiments described herein may be conveniently implemented using one or more machines (e.g., one or more computing devices that are utilized as a user computing device for an electronic document, one or more server devices, such as a document server, etc.) programmed according to the teachings of the present specification, as will be apparent to those of ordinary skill in the computer art. Appropriate software coding can readily be prepared by skilled programmers based on the teachings of the present disclosure, as will be apparent to those of ordinary skill in the software art. Aspects and implementations discussed above employing software and / or software modules may also include appropriate hardware for assisting in the implementation of the machine executable instructions of the software and / or software module.

[0062] Such software may be a computer program product that employs a machine-readable storage medium. A machine-readable storage medium may be any medium that is capable ofstoring and / or encoding a sequence of instructions for execution by a machine (e.g., a computing device) and that causes the machine to perform any one of the methodologies and / or embodiments described herein. Examples of a machine-readable storage medium include, but are not limited to, a magnetic disk, an optical disc (e.g., CD, CD-R, DVD, DVD-R, etc.), a magneto-optical disk, a read-only memory “ROM” device, a random-access memory “RAM” device, a magnetic card, an optical card, a solid-state memory device, an EPROM, an EEPROM, and any combinations thereof. A machine- readable medium, as used herein, is intended to include a single medium as well as a collection of physically separate media, such as, for example, a collection of compact discs or one or more hard disk drives in combination with a computer memory. As used herein, a machine-readable storage medium does not include transitory forms of signal transmission.

[0063] Such software may also include information (e.g., data) carried as a data signal on a data carrier, such as a carrier wave. For example, machine-executable information may be included as a data-carrying signal embodied in a data carrier in which the signal encodes a sequence of instruction, or portion thereof, for execution by a machine (e.g., a computing device) and any related information (e.g., data structures and data) that causes the machine to perform any one of the methodologies and / or embodiments described herein.

[0064] Examples of a computing device include, but are not limited to, an electronic book reading device, a computer workstation, a terminal computer, a server computer, a handheld device (e.g., a tablet computer, a smartphone, etc.), a web appliance, a network router, a network switch, a network bridge, any machine capable of executing a sequence of instructions that specify an action to be taken by that machine, and any combinations thereof. In one example, a computing device may include and / or be included in a kiosk.

[0065] FIG.5 shows a diagrammatic representation of one embodiment of a computing device in the exemplary form of a computer system 500 within which a set of instructions for causinga control system to perform any one or more of the aspects and / or methodologies of the present disclosure may be executed. It is also contemplated that multiple computing devices may be utilized to implement a specially configured set of instructions for causing one or more of the devices to perform any one or more of the aspects and / or methodologies of the present disclosure. Computer system 500 includes a processor 504 and a memory 508 that communicate with each other, and with other components, via a bus 512. Bus 512 may include any of several types of bus structures including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combinations thereof, using any of a variety of bus architectures.

[0066] Processor 504 may include any suitable processor, such as without limitation a processor incorporating logical circuitry for performing arithmetic and logical operations, such as an arithmetic and logic unit (ALU), which may be regulated with a state machine and directed by operational inputs from memory and / or sensors; processor 504 may be organized according to Von Neumann and / or Harvard architecture as a non-limiting example. Processor 504 may include, incorporate, and / or be incorporated in, without limitation, a microcontroller, microprocessor, digital signal processor (DSP), Field Programmable Gate Array (FPGA), Complex Programmable Logic Device (CPLD), Graphical Processing Unit (GPU), general purpose GPU, Tensor Processing Unit (TPU), analog or mixed signal processor, Trusted Platform Module (TPM), a floating-point unit (FPU), and / or system on a chip (SoC).

[0067] Memory 508 may include various components (e.g., machine-readable media) including, but not limited to, a random-access memory component, a read only component, and any combinations thereof. In one example, a basic input / output system 516 (BIOS), including basic routines that help to transfer information between elements within computer system 500, such as during start-up, may be stored in memory 508. Memory 508 may also include (e.g., stored on one or more machine-readable media) instructions (e.g., software) 520 embodyingany one or more of the aspects and / or methodologies of the present disclosure. In another example, memory 508 may further include any number of program modules including, but not limited to, an operating system, one or more application programs, other program modules, program data, and any combinations thereof.

[0068] Computer system 500 may also include a storage device 524. Examples of a storage device (e.g., storage device 524) include, but are not limited to, a hard disk drive, a magnetic disk drive, an optical disc drive in combination with an optical medium, a solid-state memory device, and any combinations thereof. Storage device 524 may be connected to bus 512 by an appropriate interface (not shown). Example interfaces include, but are not limited to, SCSI, advanced technology attachment (ATA), serial ATA, universal serial bus (USB), IEEE 1394 (FIREWIRE), and any combinations thereof. In one example, storage device 524 (or one or more components thereof) may be removably interfaced with computer system 500 (e.g., via an external port connector (not shown)). Particularly, storage device 524 and an associated machine-readable medium 528 may provide nonvolatile and / or volatile storage of machine- readable instructions, data structures, program modules, and / or other data for computer system 500. In one example, software 520 may reside, completely or partially, within machine- readable medium 528. In another example, software 520 may reside, completely or partially, within processor 504.

[0069] Computer system 500 may also include an input device 532. In one example, a user of computer system 500 may enter commands and / or other information into computer system 500 via input device 532. Examples of an input device 532 include, but are not limited to, an alpha- numeric input device (e.g., a keyboard), a pointing device, a joystick, a gamepad, an audio input device (e.g., a microphone, a voice response system, etc.), a cursor control device (e.g., a mouse), a touchpad, an optical scanner, a video capture device (e.g., a still camera, a video camera), a touchscreen, and any combinations thereof. Input device 532 may be interfaced tobus 512 via any of a variety of interfaces (not shown) including, but not limited to, a serial interface, a parallel interface, a game port, a USB interface, a FIREWIRE interface, a direct interface to bus 512, and any combinations thereof. Input device 532 may include a touch screen interface that may be a part of or separate from display 536, discussed further below. Input device 532 may be utilized as a user selection device for selecting one or more graphical representations in a graphical interface as described above.

[0070] A user may also input commands and / or other information to computer system 500 via storage device 524 (e.g., a removable disk drive, a flash drive, etc.) and / or network interface device 540. A network interface device, such as network interface device 540, may be utilized for connecting computer system 500 to one or more of a variety of networks, such as network 544, and one or more remote devices 548 connected thereto. Examples of a network interface device include, but are not limited to, a network interface card (e.g., a mobile network interface card, a LAN card), a modem, and any combination thereof. Examples of a network include, but are not limited to, a wide area network (e.g., the Internet, an enterprise network), a local area network (e.g., a network associated with an office, a building, a campus or other relatively small geographic space), a telephone network, a data network associated with a telephone / voice provider (e.g., a mobile communications provider data and / or voice network), a direct connection between two computing devices, and any combinations thereof. A network, such as network 544, may employ a wired and / or a wireless mode of communication. In general, any network topology may be used. Information (e.g., data, software 520, etc.) may be communicated to and / or from computer system 500 via network interface device 540.

[0071] Computer system 500 may further include a video display adapter 552 for communicating a displayable image to a display device, such as display device 536. Examples of a display device include, but are not limited to, a liquid crystal display (LCD), a cathode ray tube (CRT), a plasma display, a light emitting diode (LED) display, and any combinationsthereof. Display adapter 552 and display device 536 may be utilized in combination with processor 504 to provide graphical representations of aspects of the present disclosure. In addition to a display device, computer system 500 may include one or more other peripheral output devices including, but not limited to, an audio speaker, a printer, and any combinations thereof. Such peripheral output devices may be connected to bus 512 via a peripheral interface 556. Examples of a peripheral interface include, but are not limited to, a serial port, a USB connection, a FIREWIRE connection, a parallel connection, and any combinations thereof.

[0072] The foregoing has been a detailed description of illustrative embodiments of the invention. Various modifications and additions can be made without departing from the spirit and scope of this invention. Features of each of the various embodiments described above may be combined with features of other described embodiments as appropriate in order to provide a multiplicity of feature combinations in associated new embodiments. Furthermore, while the foregoing describes a number of separate embodiments, what has been described herein is merely illustrative of the application of the principles of the present invention. Additionally, although particular methods herein may be illustrated and / or described as being performed in a specific order, the ordering is highly variable within ordinary skill to achieve methods, systems, and software according to the present disclosure. Accordingly, this description is meant to be taken only by way of example, and not to otherwise limit the scope of this invention.

[0073] Exemplary embodiments have been disclosed above and illustrated in the accompanying drawings. It will be understood by those skilled in the art that various changes, omissions and additions may be made to that which is specifically disclosed herein without departing from the spirit and scope of the present invention.

Claims

What is Claimed is:

1. An encoder for feature coding for machines (FCM), comprising: a split neural network part 1 configured to receive an input video signal and generate at least one feature map representing the video signal; a feature reduction module receiving the at least one feature map and generating at least one reduced feature map representing the video signal; a feature conversion module receiving the reduced feature maps and converting tensors of the reduced feature maps from a floating point representation to a fixed integer video format; a right bit shift module coupled to the feature conversion module and performing a right shift of the converted feature tensors by a shift value, n; and a video encoder coupled to the right bit shift module, the video encoder receiving the shifted feature tensors and generating an encoded bitstream.

2. The encoder of claim 1, wherein the value of n is a user provided value.

3. The encoder of claim 1, further comprising a rate distortion optimization loop which tests multiple values of n and selects the value of n providing optimal rate-distortion performance.

4. The encoder of claim 1, further comprising a calculate distribution properties module receiving the at least one feature map output from a split neural network front end, at least one feature map having at least one feature tensor, the calculate distribution properties module receiving at least one feature tensor and determining at least one probability distribution property therefrom, the at least one probability distribution property being provided to the video encoder to be signaled in the encoded bitstream.

5. The encoder of claim 4, wherein the calculate distribution properties module further receives the reduced feature map from the feature reduction module and determines at least one probability distribution property of feature tensors of the reduced feature map and provides the probability distribution property of the reduced feature map to the video encoder to be signaled in the encoded bitstream.

6. The encoder of claim 1, further comprising a calculate distribution properties module receiving the reduced feature map from the feature reduction module and determining at least one probability distribution property of feature tensors of the reduced feature map and providing the probability distribution property of the reduced feature map to the video encoder to be signaled in the encoded bitstream.

7. The encoder of any of claims 4-6, wherein the probability distribution property is a mean value of feature tensors in a feature map.

8. The encoder of any of claims 4-6, wherein the probability distribution property is a standard deviation of feature tensors in a feature map.

9. The encoder of any of claims 4-8, wherein the probability distribution properties are calculated on a per layer basis.

10. The encoder of any of claims 4-8, wherein the probability distribution properties are calculated on a per channel basis.

11. The encoder of claim 6, wherein: the probability distribution property includes a mean value and a standard deviation of feature tensors in the reduced feature map; and the probability distribution properties are calculated on one of a per layer basis, a per channel basis, or both a per layer basis and per channel basis.

12. A decoder comprising: an inner video decoder receiving a coded bitstream including at least one reduced feature map representing feature maps output from a neural network front end at an encoder providing a decoded video signal with tensors in integer format representing the at least one reduced feature map; a left shift module coupled to the decoder, the left shift module receiving the integer format tensors and a shift value n signaled in the encoded bitstream, the left shift module performing a left shift operation of n bits on each tensor;an inverse feature conversion module receiving the left shifted feature tensors of the decoded video signal and converting the left shifted feature tensors to a floating point format of at least one feature map; a feature restoration module receiving the converted floating point tensors of the reduced feature map and reconstructing tensors for a plurality of feature maps representing a signal processed by a neural network front end used an encoder site from the reduced feature map; and a NN part 2 receiving the reconstructed feature maps and performing an inference operation.

13. The decoder of claim 12, wherein the encoded bitstream includes distribution properties of tensors of a reduced feature map, and the decoder further comprises a first reconstruction refinements module interposed between the inverse feature conversion module and the feature restoration module, the first reconstruction refinements module aligning a probability distribution property of the decoded tensors of the reduced feature maps with the probability distribution property encoded in the bitstream.

14. The decoder of claim 13, wherein the encoded bitstream includes distribution properties of tensors of at least one original feature map, and the decoder further comprises a second reconstruction refinements module after the feature restoration module, the first reconstruction refinements module the aligning a probability distribution property of the reconstructed tensors with the probability distribution property encoded in the bitstream.

15. The decoder of any of claims 13-14, wherein the probability distribution property is a mean value of feature tensors in the reduced feature map.

16. The decoder of any of claims 13-15, wherein the probability distribution property is a standard deviation of feature tensors in the reduced feature map.

17. The decoder of any of claims 15 or 16, wherein the probability distribution properties are calculated on a per layer basis.

18. The decoder of any of claims 15 or 16, wherein the probability distribution properties are calculated on a per channel basis.

19. The decoder of any of claims 17 or 18, wherein reconstruction refinement is performed on a periodic basis.

20. The decoder of any of claims 17 or 18, wherein reconstruction refinement is performed on every frame in the bitstream.

Images