Systems, methods and bitstreams for removing non-essential feature map information in machine-based applications using regions of interest

Abstract

Systems and methods for codecs for feature coding for machines with region of interest processing are provided. At an FCM encoder, a split neural network part 1 is configured to receive a video signal and generate a plurality of feature maps representing the video signal. A feature reduction module coupled to the NN Part 1 receives the plurality of feature maps and generating at least one reduced feature map. An image ROI extractor is configured to receive the video signal and perform detection methods to identify regions of interest in the video signal. A feature ROI extractor receives the feature maps and identifies regions of interest in the feature maps. A region filter removes content of the feature maps outside of the detected regions of interest. The remaining feature information for the regions of interest is provided in an encoded bitstream for transmission to an FCM decoder.

Description

SYSTEMS, METHODSAND BITSTREAMS FOR REMOVING NON- ESSENTIAL FEATURE MAP INFORMATION IN MACHINE-BASED APPLICATIONS USING REGIONS OF INTERESTCROSS REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims the benefit of priority to U.S. Provisional Application serial no. 63 / 634,416 filed on April 15, 2024, and entitled Systems and Methods of Non- Essential Feature Map Information Removal through Activity Regions, the disclosure of which is hereby incorporated by reference in its entirety.BACKGROUND OF THE DISCLOSURE

[0002] As the number and scale of deployed video sensors / devices increases, an increasing amount of video is expected to be processed by machines and for machine use rather than human viewing. Indeed, an exemplary system or solution employing thousands of cameras would produce significant amounts of video that cannot be monitored by humans in a cost- effective manner. Because of the significant volume of the data, an efficient compression system is desirable for Machine-to-Machine (M2M) communication. To standardize the coding for machines to facilitate a Machine-to-Machine (M2M) communication system more efficiently, MPEG issued a Call for Proposal (CfP) for Coding for Machines in 2022.

[0003] Currently, M2M communications are typically managed by two distinct systems, namely, edge computing or local computing systems and remote computing or Video Coding for Machines (VCM) systems. In the context of edge computing systems, the complete Convolutional Neural Network (CNN) is typically executed on edge devices. However, end devices often lack the computational capacity to run extensive networks, requiring billions of operations for inference result calculations. In contrast, in Video Coding for Machine systems, the video is typically compressed initially and transmitted to a base server or cloud server with more computing resources to execute the full CNN. The calculated results are then subsequently relayed back to the edge devices. However. Video Coding for machine systems may not be able to support compute offload and may fail to fully leverage the potential of end devices, as they execute the complete CNN on the base server. To overcome these limitations for M2M communication and to enable collaborative intelligence and split computing for M2M connections, MPEG has issued a Call for Proposal (CfP) for Feature Coding for Machines (FCM) technology in 2023.

[0004] Figure 1 is a simplified block diagram providing a high-level illustration of a Feature Coding for Machine (FCM) system. A resource limited device receives input video 110 and uses a split neural network part 1 115 (also referred to herein as NN front end or NN head) to extract features of the video which are encoded by FCM encoder 120. The encoded features are transmitted over a channel to a base server which decodes the features. The base server includes an FCM decoder 130 which provides decoded feature maps 135 to a neural network part 2 140 (also referred to herein as NN back end or NN tail) which provides output to inference results 145.

[0005] Referring to Figure 1, in Feature Coding for Machine (FCM) systems, a Neural Network is first split into two parts, i.e., NN Part 1 (Split CNN - Head) 115 and NN Part 2 (Split CNN - Tail) 140. Instead of running the full CNN on the end devices, only the NN Part 1 115 is executed on the resource-limited devices, and the NN Part 2 140 is preferably executed on the base server which is expected to have more computing resources. Since the NN Part 1 115 is generally executed on the edge devices, instead of transmitting videos to the base server (like in Video Coding for Machine), the intermediate feature data extracted from NN Part 1 is sent to the base server. However, the volume of intermediate feature data is usually much larger than the video itself and needs efficient compression before sending it to the base server. An example of such a system is depicted in Error! Reference source not found..

[0006] Referring to Fig. 2, which is a block diagram further illustrating an exemplary FCM system, an FCM Encoder 205 which receives feature maps 210 from an NN part 1 115 consists of three main modules, e.g., a feature reduction module 215, a feature conversion module 220, and an inner codec 225 for encoding sequences.

[0007] The FCM architecture employs a split neural network, typically a CNN. In the context of CNNs, input data, output data and transformations with the neural network are generally represented using tensors. A tensor is the primary' data structure used by a CNN and is a mathematical representation of a concept in the form of an n-dimensional array. For example, tensors can represent concepts such as scalar (0-dimensional tensor), vector (1 -dimensional tensor), matrix (2- dimensional tensor), and an nd-array (n-dimensional tensor). Typical tensors occurring in the CNNs are 3-dimensional, e.g. [c, w, h], where c represents the number of channels, and w and h represent the width and height of each channel.

[0008] As noted in Figure 1, the FCM encoder provides an encoded bitstream over a communication link (e.g., wired or wireless communication link) to a base receiver includingan FCM decoder. Referring to Fig. 2, FCM decoder 235 consists of three principal modules, e.g., an inner codec 240, inverse feature conversion module 245. and feature restoration module 250 for decoding the sequences with feature maps 255. The input of the FCM encoder 205 is intermediate feature data (Xt) extracted from NN Part 1 115 (Fig. 1), and the output of the FCM decoder is reconstructed intermediate feature data (X(). as shown in Error! Reference source not found., which is applied to NN Part 2 140 (Fig. 1). After reconstruction of the intermediate feature data (X(). the NN Part 2 is executed to generate the inference results. Feature data usually consists of multiple layers. And each layer usually consists of multiple channels. For instance, in one example, the backbone of detectron2 (NN Part 1) may consist of 4 layers and each layer consists of 256 channels. The details of all the FCM Encoder and Decoder modules are described in more detail below.

[0009] The feature reduction module 215 of the FCM encoder 205 takes original feature maps 210 and / or layers as input and outputs a reduced number of feature maps and / or layers representing the original data. Essentially, feature reduction module 215 takes the data from many layers of the feature maps 210 and condenses that information to a single layer. The feature reduction module 215 can be a neural network, auto-encoder, or just classical statistical approaches like PCA or clustering methods to reduce the number of feature maps and / or layers.

[0010] The feature conversion module 220 of the FCM encoder 205 receives reduced feature maps and / or layers as input and converts the feature tensor data from floating point to unsigned integers, such as 8 bit or lObit integers. In some cases, depending on the data, it may also pack the feature maps into one frame. Then, the packed frame or converted feature maps and / or layers are converted by the feature conversion module to a suitable video format, such as YUV 4:0:0, before sending it to the inner codec 225.

[0011] The inner codec (encoder) module 225 takes the video, such as YUV 4:0:0 format, representing the feature maps as input and outputs a bitstream to transmit over a communication channel 230 to the receiver 235. The inner codec (encoder) 225 can take the form of a Neural Network encoder or any traditional video encoder, such as known encoders for use with any known standard, such as HEVC, VVC, VVenC, AVI, or may be implemented as a combination of both Neural Network encoder and traditional encoder.

[0012] Turning to the FCM decoder 235. the inner codec (decoder) module 240 receives the encoded bitstream as input and decompresses the bitstream file to a suitable format, such as YUV 4:0:0. Like the inner codec (encoder) module 225, the inner codec (decoder) 240 can beimplemented by a Neural Network based decoder or any traditional decoder like HEVC, VVC, VVdeC or the combination of both Neural Network based decoder and traditional decoder.

[0013] The inverse feature conversion module 245 takes the decoded YUV 4:0:0 from Inner Codec 240 as input and unpacks (if necessary ) and converts the unsigned integers to floating point values.

[0014] The feature restoration module 250 receives feature tensors from the inverse feature conversion module 245 as input and restores the feature tensors from reduced size to the original size. Like the feature reduction module 215. the feature restoration module 250 can also be a Neural Network, Auto-Decoder, or implemented with classical statistical approaches like Inverse PCA or de-clustering to restore the original size of the feature maps and layers.

[0015] Documents which may provide relevant background information for the present disclosure include: [1] Yuan Zhang, Manouchehr Rafie, Shan Liu. Christopher Hollmann, "|NOO I 64| Call for Proposals on Video Coding for Machines.” ISO / IEC JTC 1 / SC 28 / WG 2. Jan. 2022; [2] C. Rosewame (Canon) and Y. Zhang (China Telecom), “[N00282] Call for Proposals on Feature Compression for Video Coding for Machines.” ISO / IEC JTC 1 / SC 29 / WG 2, Jan. 2023; [3] WG 04 MPEG Video coding, “[N00460] Algorithm description of FCTM.” Jan. 2024; and [4] Y. Wu. A. Kirillov. F. Massa, W. Y. Lo. and R. Girshick, “Detectron2,” 2019, Feb. 2024 (Online).SUMMARY OF THE DISCLOSURE

[0016] Systems and methods for improved FCM encoding with region of interest processing, decoding methods, and bitstream syntax are provided. Examples of embodiments, features and advantages of the presently disclosed encoder and decoder embodiments include, but are not limited to, the following summary.

[0017] In a first embodiment, an encoder for feature coding for machines (FCM) with region of interest processing includes a split neural network part 1 which receives an input video signal and generates a plurality of feature maps representing the video signal. A feature reduction module receives the plurality of feature maps and generates at least one reduced feature map representing the original plurality of feature maps. An image ROI extractor is provided which receives the video signal and performs object detection to identify regions of interest in the video signal. A feature ROI extractor configured to receive the feature maps and identify regions of interest in the feature maps is preferably included. A region filter coupled to the image ROI extractor, feature ROI extractor, and NN Part 1 , removes content of the featuremaps located outside of the detected regions of interest. A quantization and packing module coupled to the region filter, receives the feature information for the regions of interest, and packs and quantizes the regions of interest into a format suitable for video compression. A video encoder receiving the packed and quantized ROI feature information and generating an encoded bitstream.

[0018] In another embodiment, an encoder for feature coding for machines (FCM) includes a split neural network part 1 configured to receive a video signal having a plurality of frames, including a current frame and a previous frame, and generates a plurality of feature maps representing the video signal. An intra frame detector indicating when intra frame encoding period is enabled. An image object extractor receives the video signal and a signal from the intra frame detector. The image object extractor performs object detection and identifying regions of interest based on the detected objects once per intra period. An image motion extractor module receives the previous frame and current frame of the video signal and detects regions of interest based on motion detected in the video signal following an intra frame. A region filter is coupled to the image ROI extractor, image motion extractor, and NN Part 1 , and removes content of the feature maps located outside of the detected regions of interest. A quantization and packing module coupled to the region filter, receives the feature information for the regions of interest, and packs and quantizing the regions of interest into frames suitable for video compression. A video encoder receives the packed and quantized ROI feature information and generates an encoded bitstream.

[0019] In some embodiments, a feature reduction module may be coupled to the NN Part 1, receive the plurality of feature maps, and generates at least one reduced feature map representing the original plurality of feature maps. A feature ROI extractor can be provided and configured to receive the reduced feature maps and identify regions of interest in the reduced feature maps. In such embodiments, the region filter is preferably further coupled to the feature ROI extractor to receive parameters of regions of interest detected in the reduced feature maps.

[0020] In some embodiments the feature ROI extractor can extracts regions of interest from the feature maps using statistical methods. In other embodiments, machine learning methods may be used.

[0021] In some embodiments, the ROI detectors can generate regions of interest on a per frame basis or signal areas of activity with feature maps and be run within a subset of total frames such as during intra frames.

[0022] In some embodiments, regions of interest can be identified with one or more ROI bounding boxes. Alternatively, a mask of identified ROIs can be used.

[0023] In an embodiment of the present disclosure, a decoder for FCM with ROI processing includes a video decoder receiving a bitstream, the bitstream including coded feature information of regions of interest of a video signal and signaling information with parameters of the detected regions of interest. A dequantization and depacking module coupled to the video decoder receives the decoded video frames including the feature maps of the regions of interest. An intra detection module is preferably coupled to the video decoder and identifies boundaries of detected regions of interest once for each intra period. A stationary tensors module is coupled to the intra detection module and extracts stationary7tensors of regions of interest without detected motion once per intra period. The stationary tensors are incorporated into the feature maps of the regions of interest. A NN part 2 receives the feature maps of the regions of interest from the stationary7tensors module and the dequantization and depacking module and performs an inferencing operation for a machine task.BRIEF DESCRIPTION OF THE FIGURES

[0024] For the purpose of illustrating the invention, the drawings show aspects of one or more embodiments of the invention. However, it should be understood that the present invention is not limited to the precise arrangements and instrumentalities shown in the drawings, wherein:

[0025] Figure 1 is a simplified block diagram of a system for Feature Coding for Machines (FCM);

[0026] Figure 2 is block diagram of a system for Feature Coding for Machines (FCM) of Fig. 1 and further illustrating the components of the encoder and decoder systems;

[0027] Figure 3 is block diagram of a first embodiment of a system for Feature Coding for Machines (FCM) in accordance with the present disclosure;

[0028] Figure 4 is block diagram of a further embodiment of a system for Feature Coding for Machines (FCM) with ROI extraction in accordance with the present disclosure;

[0029] Figure 5 A is an image of a sample frame of input video;

[0030] Figure 5B is an image showing the processed frame of Fig. 5A showing a region of interest processed feature map;

[0031] Figure 5C is an image showing the feature map ROI sample of Figure 5B;

[0032] Figure 5D is the image of Fig. 5A showing bounding boxes identifying the detected region of interest;

[0033] Figure 6 is a graph illustrating mAp versus bitrate (Kbps) for a test video, “Park Scene7’;

[0034] Figure 7 is a block diagram of another embodiment of a system for Feature Coding for Machines (FCM) with ROI extraction in accordance with the present disclosure;

[0035] Figure 8 is a block diagram of a computing system that can be used to implement any one or more of the methodologies disclosed herein and any one or more portions thereof.

[0036] The drawings are not necessarily to scale and may be illustrated by phantom lines, diagrammatic representations and fragmentary views. In certain instances, details that are not necessary for an understanding of the embodiments or that render other details difficult to perceive may have been omitted.DETAILED DESCRIPTION OF DISCLOSED EMBODIMENTS

[0037] The present systems and methods provide improvements over known FCM codec systems. Exemplary embodiments of the presently disclosed FCM codecs include a calculate distribution properties module / process on the encoder side or edge device and a reconstruction refinement module / process on decoder side, typically at the base receiver. Further, region extraction is preferably employed to identify regions of interest in the feature maps.

[0038] Figure 3 is block diagram of a first embodiment of a system for Feature Coding for Machines (FCM) with ROI extraction in accordance with the present disclosure. At the encoder side, the FCM encoder 305 includes a NN Part 1 310, a quantization and packing module 315, and an encoder or inner codec 320, similar to known FCM encoders, such as described in connection with Fig. 2. In the NN part 1 310 features of an input image are extracted, preferably using the Neural Network such as the Res-Net. These features are composed of multiple layers each with n number of channels. Each channel in each layer represents a convolution with a kernel and represents input image features. Typically, all channels of a given layer have the same dimensions.

[0039] In the quantization & packing module 315 feature information is packed and quantized into a format that can be passed to and image / video encoder 320. The encoder 320 compresses the packed features using image and video compression systems such as the versatile videocoding (VVC), HEVC, AVI or any other suitable image or video compression. The compressed data is represented in a bitstream that is transmitted to an FCM decoder receiver over communication networks.

[0040] In the embodiment of Fig. 3, methods of non-essential feature map information removal through activity^ regions can be performed. Areas of activity7or object presence within feature maps can be attained through methods such as object detection, statistical inference of feature maps, or motion activity calculation. This will generally represent areas within the feature maps that are necessary for the proper functioning of the system. Once these areas representing regions of interest (ROI) in the original input image(s), are identified, information outside of the identified areas can be removed from the feature maps (for example, by setting the values to a constant value) to reduce the size and complexity of the content to be encoded.

[0041] The system of Fig. 3 includes an Image ROI extractor module 325. The ROI extractor module 325 receives the input image data and detects regions of interest using traditional methods, such as object detection network, applied to the original input image. Extraction of region(s) of interest from the input image can be performed, for example, using an encoder side object detection network. The network can generate regions of interest on a per frame basis or signal areas of activity with feature maps and be run within a subset of total frames such as during intra frames.

[0042] In addition, the system of Figure 3 includes feature ROI extractor 330 which is coupled to the output of the NN Part 1 and extracts regions of interest from the feature maps using statistical and / or learned methods. This method will extract regions of interest from the feature information already available at the encoder side. Regions of interest can be generated using statistical methods, for example such as a gradient-weighted class activation method (“GradCam”) method, or by using learned network architectures such as region of interest (ROI) pooling.

[0043] Both the Image ROI extractor 325 and Feature ROI extractor 330 can be used individually or together depending on the requirements of a particular task.

[0044] Regions of interest can be specified, for example, as a mask or defined bounding boxes.

[0045] A region filter 335 is coupled to the outputs of the NN Part 1 310, image ROI extractor 325, and feature ROI extractor 330 and provides an output coupled to the quantization and packing module 315. The region filter 335 applies reduction / truncation to feature regions thatlie outside of the regions of interest as defined by one or both of the image ROI and / or feature ROI. The processing of the region filter 335 can be applied to either the full or reduced feature maps depending on the pipeline and machine task.

[0046] The FCM receiver (decoder side) of Fig 3 includes a conventional decoder / inner codec 345 in which the compressed channel image data is decompressed using an appropriate decompression system to produce a decoded tiled image. If lossy compression is used, the decoded image may have distortion caused by the lossy compression.

[0047] A dequantization & unpacking module 350 follows the decoder 345 and processes the decoded bitstream such that image representation of features is converted back into feature information formatted for the neural network part 2 355 or feature restoration where applicable.

[0048] It will be appreciated that since the encoder of Fig. 3 only encodes the features of the identified ROIs and the region filter 335 discards information outside the ROIs as unimportant, the resulting bitstream is less complex and the decoding process may be is simplified and / or more efficient.

[0049] Figure 4 is an alternate embodiment of an FCM system with ROI extraction in accordance with the present disclosure. The encoder of Fig. 4 includes the elements of Fig. 3, including NN part 1 410, image ROI extractor 425, feature ROI extractor 430, region filter 435, quantization and packing module 415, and encoder 420. The encoder in the system of Fig. 4 further includes a feature reduction module 433 interposed between NN part 1 410 and region filter 435. In the feature reduction module 433, feature information from NN Part 1 is reduced into a smaller format, such as reducing many feature layers into a single layer. This can be performed by machine learning based reduction or other processing methods.

[0050] On the decoder side, the embodiment of Figure 4 includes a decoder 445, dequantization and unpacking module 450, and NN Part 2 455, as described in corresponding components in connection with Fig. 3. The decoder in the embodiment of Fig. 4 further includes a feature restoration module 453 in which reduced feature information from feature reduction module 433 is restored to a format that is accepted by part 2 of the neural network. This can be from a machine learning based restoration process or other conventional processing.

[0051] Figure 5A is an image of a sample frame of input video. Figure 5B is an image illustrating the processed frame of Fig. 5A, including showing a region of interest processedfeature map. Figure 5C is an image showing an enlarged view of the feature map sample in the ROI of Figure 5B. Figure 5D is the image of Fig. 5A showing the detected region of interest identified by bounding boxes.

[0052] Figure 6 is a graph illustrating exemplary detection performance (mAP) versus bitrate for an exemplary' test video sequence, “Park Scene.” The graph shows that for a certain mAP (detection performance) applying the present methods results in a significant reduction in bitrate (kbps) required for obtaining the same or higher precision (mAP). As a result, the present methods can provide for achieving comparable mAP with a significantly lower bitrate, or can be used to achieve significantly higher precision at a comparable bitrate.

[0053] Figure 7 is a block diagram of an alternate embodiment of the present FCM System which includes feature motion-based ROI processing. Similar to the embodiment in Fig. 3, the encoder side of the FCM system of Fig. 7 includes NN part 1 710, region filter 735, quantization and packing module 715, and encoder 720. Rather than the image ROI extractor (detector) and feature ROI extractor modules of Fig. 3, the embodiment of Fig. 7 employs an image object extractor module 725 couple to an intra frame detector 723, which signals an intra frame. The image object extractor 725 uses a detection network to identify regions of interest and outputs bounding boxes of the detected ROIs to the region filter 735. In this embodiment, to reduce processing complexity, the image object extractor 725 is preferably run only once per intra frame, as signaled by the intra frame detector 723.

[0054] The image motion extractor 730 receives the input images and processes a current frame 707b and a previous frame 707a to detect regions in the input video with motion. Based on the assumption that detectable objects or objects of interest would create tensor motion, these regions are treated as essential. Frames following the intra frame will preferably identify essential tensor areas based on motion, based on proposition that detectable objects will cause motion.

[0055] In some embodiments, the detected object boundaries and object motion can be represented as a mask and that mask can be provided to the encoder 720 once per frame. Alternatively, object boundaries can be represented as one or more bounding boxes.

[0056] At the decoder side of the FCM system of Fig. 7, the system includes a decoder 745, dequantization & unpacking module 750, and NN part 2 755, as described above. The decoder 745 provides the decoded bitstream to an intra detection box bounds module 747 to extract the detected box bounds identified by the image object extraction module 735 at the encoder.These box bounds are preferably saved once per intra period to capture essential regions not caught by the image motion extractor 730.

[0057] A stationary tensor module 749 is coupled to the intra detection box bounds module and the dequantization & unpacking module 750 and operates to extract stationary' tensors once per intra frame. The stationary' tensor values are provided by the region filter 735 and are used to identify regions of the input video that do not change over the period of a group of pictures from one intra frame to a subsequent intra frame. The encoder signals that the decoder, via the stationary tensor module 749, can reuse the stationary tensors from an intra frame and maintain stationary essential areas which may not be captured by' the image motion extractor.

[0058] The following is an example of a suitable bitstream syntax for use with the current embodiments of FCM codecs:

[0059] Example Bitstream Object

[0060]

[0083]

[0084] Example Bitstream Mask

[0085]

[0100]

[0101] An interest area mask can also be represented implicitly such as a '‘green screen” approach where the mask is inlayed within the bitstream using a unique value.

[0102] Some embodiments of the present disclosure may include / and or be embodied by non- transitory computer program products (i.e., physically embodied computer program products) that store instructions, which when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform operations herein. Such non- transitory computer program products (i.e., physically embodied computer program products) may store instructions, which when executed by one or more data processors of one or more computing systems, causes at least one data processor to perform operations, and / or steps thereof described in this disclosure, including without limitation any operations described above and / or any operations of the FCM decoder or FCM encoder may be configured to perform. Similarly, computer systems are also described that may include one or more data processors and memory coupled to the one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. In addition, methods can be implemented by oneor more data processors either within a single computing system or distributed among two or more computing systems. Such computing systems can be connected and can exchange data and / or commands or other instructions or the like via one or more connections, including a connection over a network (e.g. the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, or the like), via a direct connection betw een one or more of the multiple computing systems, or the like.

[0103] It is to be noted that any one or more of the aspects and embodiments described herein may be conveniently implemented using one or more machines (e.g., one or more computing devices that are utilized as a user computing device for an electronic document, one or more server devices, such as a document server, etc.) programmed according to the teachings of the present specification, as will be apparent to those of ordinary’ skill in the computer art. Appropriate software coding can readily be prepared by skilled programmers based on the teachings of the present disclosure, as will be apparent to those of ordinary skill in the softw are art. Aspects and implementations discussed above employing software and / or software modules may also include appropriate hardware for assisting in the implementation of the machine executable instructions of the software and / or software module.

[0104] Such software may be a computer program product that employs a machine-readable storage medium. A machine-readable storage medium may be any medium that is capable of storing and / or encoding a sequence of instructions for execution by a machine (e.g., a computing device) and that causes the machine to perform any one of the methodologies and / or embodiments described herein. Examples of a machine-readable storage medium include, but are not limited to, a magnetic disk, an optical disc (e.g., CD, CD-R, DVD, DVD-R, etc.), a magneto-optical disk, a read-only memory' ‘'ROM” device, a random-access memory “RAM” device, a magnetic card, an optical card, a solid-state memory device, an EPROM, an EEPROM, and any combinations thereof. A machine- readable medium, as used herein, is intended to include a single medium as well as a collection of physically separate media, such as, for example, a collection of compact discs or one or more hard disk drives in combination with a computer memory’. As used herein, a machine-readable storage medium does not include transitory forms of signal transmission.

[0105] Such software may also include information (e.g., data) carried as a data signal on a data carrier, such as a carrier wave. For example, machine-executable information may be included as a data-carrying signal embodied in a data carrier in which the signal encodes asequence of instruction, or portion thereof, for execution by a machine (e.g., a computing device) and any related information (e.g., data structures and data) that causes the machine to perform any one of the methodologies and / or embodiments described herein.

[0106] Examples of a computing device include, but are not limited to, an electronic book reading device, a computer workstation, a terminal computer, a server computer, a handheld device (e.g., a tablet computer, a smartphone, etc.), a web appliance, a network router, a network switch, a network bridge, any machine capable of executing a sequence of instructions that specify an action to be taken by that machine, and any combinations thereof. In one example, a computing device may include and / or be included in a kiosk.

[0107] FIG. 8 shows a diagrammatic representation of one embodiment of a computing device in the exemplary form of a computer system 800 within which a set of instructions for causing a control system to perform any one or more of the aspects and / or methodologies of the present disclosure may be executed. It is also contemplated that multiple computing devices may be utilized to implement a specially configured set of instructions for causing one or more of the devices to perform any one or more of the aspects and / or methodologies of the present disclosure. Computer system 800 includes a processor 804 and a memory 808 that communicate with each other, and with other components, via a bus 812. Bus 812 may include any of several types of bus structures including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combinations thereof, using any of a variety of bus architectures.

[0108] Processor 804 may include any suitable processor, such as without limitation a processor incorporating logical circuitry for performing arithmetic and logical operations, such as an arithmetic and logic unit (ALU), which may be regulated with a state machine and directed by operational inputs from memory and / or sensors; processor 804 may be organized according to Von Neumann and / or Harvard architecture as a non-limiting example. Processor 804 may include, incorporate, and / or be incorporated in, without limitation, a microcontroller, microprocessor, digital signal processor (DSP), Field Programmable Gate Array (FPGA), Complex Programmable Logic Device (CPLD), Graphical Processing Unit (GPU), general purpose GPU, Tensor Processing Unit (TPU), analog or mixed signal processor, Trusted Platform Module (TPM), a floating-point unit (FPU), and / or system on a chip (SoC).

[0109] Memory 808 may include various components (e.g., machine-readable media) including, but not limited to, a random-access memory component, a read only component, andany combinations thereof. In one example, a basic input / output system 816 (BIOS), including basic routines that help to transfer information between elements within computer system 800, such as during start-up, may be stored in memory 808. Memory 808 may also include (e.g., stored on one or more machine-readable media) instructions (e.g., software) 820 embodying any one or more of the aspects and / or methodologies of the present disclosure. In another example, memory 808 may further include any number of program modules including, but not limited to, an operating system, one or more application programs, other program modules, program data, and any combinations thereof.

[0110] Computer system 800 may also include a storage device 824. Examples of a storage device (e.g., storage device 824) include, but are not limited to, a hard disk drive, a magnetic disk drive, an optical disc drive in combination with an optical medium, a solid-state memory device, and any combinations thereof. Storage device 824 may be connected to bus 812 by an appropriate interface (not shown). Example interfaces include, but are not limited to, SCSI, advanced technology attachment (ATA), serial ATA, universal serial bus (USB), IEEE 1394 (FIREWIRE), and any combinations thereof. In one example, storage device 824 (or one or more components thereof) may be removably interfaced with computer system 800 (e.g., via an external port connector (not shown)). Particularly, storage device 824 and an associated machine-readable medium 828 may provide nonvolatile and / or volatile storage of machine- readable instructions, data structures, program modules, and / or other data for computer system 800. In one example, software 820 may reside, completely or partially, within machine- readable medium 828. In another example, software 820 may reside, completely or partially, within processor 804.

[0111] Computer system 800 may also include an input device 832. In one example, a user of computer system 800 may enter commands and / or other information into computer system 800 via input device 832. Examples of an input device 832 include, but are not limited to, an alphanumeric input device (e.g., a keyboard), a pointing device, a joystick, a gamepad, an audio input device (e.g., a microphone, a voice response system, etc.), a cursor control device (e.g., a mouse), a touchpad, an optical scanner, a video capture device (e.g., a still camera, a video camera), a touchscreen, and any combinations thereof. Input device 832 may be interfaced to bus 812 via any of a variety of interfaces (not shown) including, but not limited to, a serial interface, a parallel interface, a game port, a USB interface, a FIREWIRE interface, a direct interface to bus 812, and any combinations thereof. Input device 832 may include a touch screen interface that may be a part of or separate from display 836, discussed further below.Input device 832 may be utilized as a user selection device for selecting one or more graphical representations in a graphical interface as described above.

[0112] A user may also input commands and / or other information to computer system 800 via storage device 824 (e.g., a removable disk drive, a flash drive, etc.) and / or network interface device 840. A network interface device, such as network interface device 840, may be utilized for connecting computer system 800 to one or more of a variety of networks, such as network 844, and one or more remote devices 848 connected thereto. Examples of a network interface device include, but are not limited to, anetwork interface card (e.g., a mobile network interface card, a LAN card), a modem, and any combination thereof. Examples of a network include, but are not limited to, a wide area network (e.g., the Internet, an enterprise network), a local area network (e.g., a network associated with an office, a building, a campus or other relatively small geographic space), a telephone network, a data network associated with a telephone / voice provider (e g., a mobile communications provider data and / or voice network), a direct connection between two computing devices, and any combinations thereof. A network, such as network 844. may employ a wired and / or a wireless mode of communication. In general, any network topology may be used. Information (e.g.. data, software 820, etc.) may be communicated to and / or from computer system 800 via network interface device 840.

[0113] Computer system 800 may further include a video display adapter 852 for communicating a displayable image to a display device, such as display device 836. Examples of a display device include, but are not limited to, a liquid crystal display (LCD), a cathode ray tube (CRT), a plasma display, a light emitting diode (LED) display, and any combinations thereof. Display adapter 852 and display device 836 may be utilized in combination with processor 804 to provide graphical representations of aspects of the present disclosure. In addition to a display device, computer system 800 may include one or more other peripheral output devices including, but not limited to, an audio speaker, a printer, and any combinations thereof. Such peripheral output devices may be connected to bus 812 via a peripheral interface 856. Examples of a peripheral interface include, but are not limited to, a serial port, a USB connection, a FIREWIRE connection, a parallel connection, and any combinations thereof.

[0114] The foregoing has been a detailed description of illustrative embodiments of the invention. Various modifications and additions can be made without departing from the spirit and scope of this invention. Features of each of the various embodiments described above may be combined with features of other described embodiments as appropriate in order to providea multiplicity of feature combinations in associated new embodiments. Furthermore, while the foregoing describes a number of separate embodiments, what has been described herein is merely illustrative of the application of the principles of the present invention. Additionally, although particular methods herein may be illustrated and / or described as being performed in a specific order, the ordering is highly variable within ordinary7skill to achieve methods, systems, and software according to the present disclosure. Accordingly, this description is meant to be taken only by way of example, and not to otherwise limit the scope of this invention.

[0115] Exemplary embodiments have been disclosed above and illustrated in the accompanying drawings. It will be understood by those skilled in the art that various changes, omissions and additions may be made to that which is specifically disclosed herein without departing from the spirit and scope of the present invention.Y1

Claims

What is Claimed is:

1. An encoder for feature coding for machines (FCM), comprising: a split neural network part 1 configured to receive a video signal and generate a plurality of feature maps representing the video signal; a feature reduction module coupled to the NN Part 1, receiving the plurality of feature maps and generating at least one reduced feature map representing the original plurality' of feature maps; an image ROI extractor configured to receive the video signal, the image ROI extractor performing ROI detection and identifying regions of interest in the video signal based on the detected objects; a feature ROI extractor configured to receive the feature maps and identify regions of interest in the feature maps; a region filter coupled to the image ROI extractor, feature ROI extractor, and NN Part 1, the region filter removing content of the feature maps outside of the detected regions of interest; a quantization and packing module coupled to the region filter, receiving the feature information for the regions of interest, and packing and quantizing the regions of interest into a format for video compression; and a video encoder receiving the packed and quantized ROI feature information and generating an encoded bitstream.

2. The encoder of claim 1, wherein ROI detection includes at least one method selected from the group including object detection, statistical inference of feature maps, or motion activity calculation.

3. An encoder for feature coding for machines (FCM), comprising: a split neural network part 1 configured to receive a video signal having a plurality of frames including a current frame and a previous frame and generate a plurality of feature maps representing the video signal; an intra frame detector indicating when intra frame encoding is enabled; an image object extractor configured to receive the video signal and receive a signal from the intra frame detector, the image object extractor performing ROI detection and identify ing regions of interest based on the detected objects,an image motion extractor module receiving the previous frame and current frame of the video signal and detecting regions of interest based on motion detected in the video signal following an intra frame; a region fdter coupled to the image ROI extractor, feature ROI extractor, and NN Part 1, the region filter removing content of the feature maps outside of the detected regions of interest; a quantization and packing module coupled to the region filter, receiving the feature information for the regions of interest, and packing and quantizing the regions of interest into a format suitable for video compression; and a video encoder receiving the packed and quantized ROI feature information and generating an encoded bitstream.

4. The encoder of claim 3, wherein ROI detection includes at least one method selected from the group including object detection, statistical inference of feature maps, or motion activity calculation.

5. The encoder of claim 3, further comprising: a feature reduction module coupled to the NN Part 1 and receiving the lurality of feature maps and generating at least one reduced feature map representing the original plurality of feature maps; and a feature ROI extractor configured to receive the feature maps and identify regions of interest in the feature maps, wherein the region filter is further coupled to the feature ROI extractor to receive parameters of regions of interest detected in the feature maps.

6. The encoder of claim 3 further comprising a feature reduction module coupled to the NN Part 1. receiving the plurality of feature maps and generating at least one reduced feature map representing the original plurality of feature maps, the at least one reduced feature map being provided to the region filter, wherein the region filter reduction module removing content of the reduced feature maps outside of the detected regions of interest.

7. A decoder for FCM comprising:a video decoder receiving a bitstream including coded feature information of regions of interest of a video signal and signaling information with parameters of the detected regions of interest; a dequantization and depacking module coupled to the video decoder and receiving decoded video frames including the feature maps of the regions of interest; an intra detection module coupled to the video decoder and identifying boundaries of detected regions of interest once for each intra period; a stationary tensors module coupled to the intra detection module and configured to extract stationary tensors of regions of interest without detected motion once per intra period and incorporating the stationary tensors in the feature maps of the regions of interest; a NN part 2 receiving the feature maps from the stationary tensors module and dequantization and depacking module and performing inferencing for a machine task.

Images