Method and apparatus for decoding data

Abstract

To provide a system and method for decoding information for data generated by a first portion of a neural network.SOLUTION: The method comprises decoding information for determining at least a starting layer of a second portion of a convolutional neural network (CNN), the CNN including at least a first portion and a second portion, the second portion being different from the first portion, and determining the starting layer of the second portion of the CNN based on the decoded information.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] Reference to related applications This application claims the benefit under 35 U.S.C. § 119 of the filing date of Australian Patent Application No. 2021232739, filed September 15, 2021, the entirety of which is incorporated herein by reference as if fully set forth herein.

[0002] Technical Field The present invention relates to a method and apparatus for decoding data from a bitstream. [Background technology]

[0003] Video compression is a ubiquitous technology used to support many applications, including those for the transmission and storage of video data. The key to the ubiquity of video compression technology is the adoption of video coding standards that enable interoperability between applications and devices produced by many commercial entities. Video coding standards themselves are developed by standards-setting organizations (SSOs) such as the International Telecommunication Union (ITU) Telecommunication Standardization Sector (ITU-T) Study Group 16, Question 6 (SG16 / Q6), also known as the "Video Coding Experts Group" (VCEG), and the International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Group 11 (ISO / IEC JTC1 / SC29 / WG11), also known as the "Moving Picture Experts Group" (MPEG).

[0004] Convolutional neural networks (CNNs) are an emerging technology that addresses use cases involving machine vision, such as object recognition, object tracking, human pose estimation, and action recognition, among others. With the increasing use of machine vision in automated processes, MPEG has formed an exploratory ad hoc group to investigate techniques that can support video compression standards where the consumer of the video is a machine rather than a human.

[0005] CNNs typically contain many layers, such as convolutional and fully connected layers, and data is passed from one layer to the next in the form of "tensors." The parameters of a layer in a CNN are commonly called "weights," and the layer's output tensor is calculated from the input tensors to the layer and the layer's weights. Layer weights are determined by training the CNN. Typically, the CNN is trained using at least some labeled data. Training with labeled data is also known as supervised learning, while the labels are sometimes called "ground truth." Prior to training, the initial values ​​of the weights can be selected randomly or copied from a pre-trained network whose weights have been optimized for the relevant task. To achieve good performance, CNNs are trained on very large amounts of training data. Training on very large amounts of data is made tractable by iterating over the training data in batches. At each iteration, an error function calculated from the output and ground truth is used to optimize the weights in a process called backpropagation. The exact optimization method may be stochastic gradient descent or another variant, such as momentum. Once training is complete, the weights are fixed. Running a trained CNN on an input to produce an output is commonly called "inference" or "inference".

[0006] Generally, tensors have four dimensions: batch size, channels, height, and width. The data represented in a tensor is sometimes called a "feature." When inferring video data, the batch size is 1 if the video is processed frame by frame. The number of channels generally corresponds to the number of features that can be represented by the CNN at that layer. In earlier layers of a CNN, features tend to capture low-level visual characteristics such as edges and textures, while in later layers of a CNN, features tend to capture higher-level semantics such as object class. A tensor is sometimes called a set of "feature maps," where the number of feature maps is equal to the channel dimension, and the height and width of the tensor are the spatial resolution of each feature map.

[0007] If a convolutional layer has a "stride" greater than 1, the output tensor from the convolution has lower spatial resolution than the input tensor. Operations such as "max pooling" also reduce the spatial size of the output tensor compared to the input tensor. Max pooling generates an output tensor by dividing the input tensor into groups, such as 2x2, and selecting the maximum value from each group as the output for the corresponding value in the output tensor. As data progresses through the CNN, the tensor typically decreases in spatial resolution but may increase in channel dimension.

[0008] In one possible pipeline for video compression for machines, intermediate CNN features can be compressed rather than the original video data, which can be called "feature coding." The feasibility of feature coding, and in particular, the competitiveness of feature coding relative to video coding, depends on two main factors: the size of the features relative to the size of the original video data, and the ability of the feature coder to find and exploit redundancies within the features. Summary of the Invention

[0009] It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

[0010] One aspect of the present disclosure provides a method for decoding information of data generated by a first portion of a neural network, the method including: decoding information for at least determining a starting layer of the second portion of the neural network, the second portion including at least the first portion and a second portion, the second portion being different from the first portion; and determining a starting layer of the second portion of the neural network based on the decoded information.

[0011] According to another aspect, the method further includes decoding data generated by the first portion from the bitstream and processing the decoded data using a second portion of the neural network according to the determined starting layer.

[0012] According to another aspect, the method further includes transmitting the decoded information indicative of the starting layer to an external processing device; decoding, at the external processing device, data generated by the first portion of the neural network from the bitstream; and processing the decoded data using a second portion of the neural network in accordance with the determined starting layer.

[0013] According to another aspect, the neural network includes layers of a first type that are additive layers, and starting layers of a second portion of the neural network are limited to layers that are not of the first type.

[0014] According to another aspect, the neural network includes a layer of a second type that is a convolutional layer, and the starting layer of the second portion of the neural network is limited to the layer immediately following the layer of the second type.

[0015] According to another aspect, the neural network includes a layer of a third type that is an output layer, and the starting layer of the second portion of the neural network is limited to a layer included in a set of layers, in a processing order of the neural network, from the predetermined layer to the third layer, and the set of layers does not include a layer of the first type.

[0016] According to another aspect, the information indicates the difference between the given layer and the starting layer.

[0017] According to another aspect, the neural network includes a layer of a second type that is a convolutional layer, and the predetermined layer is the layer that immediately follows the layer of the second type in the processing order.

[0018] According to another aspect, the neural network includes a third type of layer that is an output layer, and the predetermined layer is a layer of the third type.

[0019] According to another aspect, the predetermined layer is determined based on information indicative of the neural network to be used.

[0020] According to another aspect, the information identifies the neural network to be used, and a starting layer of the second portion of the neural network is determined from the information associating the starting layer with the neural network.

[0021] According to another aspect, the information identifies (i) a neural network from the plurality of neural networks and (ii) split points for first and second portions of the neural network, and a starting layer of the second portion of the neural network is determined from the split point.

[0022] According to another aspect, a plurality of starting layers of a second portion of the neural network are determined according to the information, and the data is processed using the second portion of the neural network according to the determination of the plurality of starting layers.

[0023] Another aspect of the present disclosure provides a method for encoding information for data processed using a first portion of a neural network, the second portion including at least the first portion and a second portion, the second portion being different from the first portion, the method including: determining a starting layer for the second portion of the neural network; and encoding information used to determine at least the starting layer for the second portion of the neural network.

[0024] According to another aspect, the method further includes generating data according to the determination of the starting layer and encoding the processed data using the first portion.

[0025] According to another aspect, the neural network includes layers of a first type that are additive layers, and the starting layers of the second portion are limited to layers that are not of the first type.

[0026] According to another aspect, the neural network includes a layer of a second type that is a convolutional layer, and the starting layer of the second portion is limited to the layer immediately following the layer of the second type.

[0027] According to another aspect, the neural network includes a layer of a third type that is an output layer, and the starting layer of the second portion is limited to a layer included in a set of layers from the predetermined layer to the third layer in the processing order of the neural network, the set of layers not including a layer of the first type.

[0028] According to another aspect, the encoded information indicates the difference between the given layer and the starting layer.

[0029] According to another aspect, the neural network includes a layer of a second type that is a convolutional layer, and the predetermined layer is the layer that immediately follows the layer of the second type in the processing order.

[0030] According to another aspect, the neural network includes a third type of layer that is an output layer, and the predetermined layer is a layer of the third type.

[0031] According to another aspect, the predetermined layer is determined based on information indicative of the neural network to be used.

[0032] According to another aspect, the encoded information identifies the neural network to be used, and a starting layer of the second portion of the neural network is determined from information associating the starting layer with the neural network.

[0033] According to another embodiment, processing with the first portion of the neural network ends at the layer immediately preceding the starting layer.

[0034] According to another aspect, a plurality of starting layers of the second portion are determined, and the data is processed using the second portion of the neural network according to the determination of the plurality of starting layers.

[0035] According to another aspect, the second portion is used to process data decoded from the bitstream.

[0036] According to another aspect, the encoded information identifies (i) a neural network from the plurality of neural networks and (ii) a split point between a first and second portion of the neural network.

[0037] Another aspect of the present disclosure provides a decoder for decoding information of data generated by a first portion of a neural network, the decoder including: a decoding unit configured to decode information for determining at least a starting layer of the second portion of the neural network, the second portion including at least the first portion and a second portion, the second portion being different from the first portion; and a determination unit configured to determine a starting layer of the second portion of the neural network based on the decoded information.

[0038] Another aspect of the present disclosure provides an encoder for encoding information of data processed using a first portion of a neural network, the encoder including at least the first portion and a second portion, the first portion being different from the second portion; the encoder comprising: a determining unit configured to determine a starting layer of the second portion of the neural network; and an encoding unit configured to encode information used to determine at least the starting layer of the second portion of the neural network.

[0039] Another aspect of the present disclosure provides a non-transitory computer-readable storage medium storing a program for performing a method for decoding information of data generated by a first portion of a neural network, the method including: decoding information for at least determining a starting layer of the second portion of the neural network, the second portion including at least the first portion and a second portion, the second portion being different from the first portion; and determining a starting layer of the second portion of the neural network based on the decoded information.

[0040] Another aspect of the present disclosure provides a non-transitory computer-readable storage medium storing a program for performing a method for encoding information of data processed using a first portion of a neural network, the method including: determining a starting layer of the second portion of the neural network, the second portion including at least the first portion and a second portion, the first portion being different from the second portion; and encoding information used to determine at least the starting layer of the second portion of the neural network.

[0041] Other aspects are also disclosed. [Brief explanation of the drawings]

[0042] At least one embodiment of the present invention will now be described with reference to the following drawings: [Figure 1] FIG. 1 is a schematic block diagram illustrating a distributed machine task system. [Figure 2A] FIG. 2A forms a schematic block diagram of a general-purpose computer system on which the distributed machine task system of FIG. 1 may be implemented. [Figure 2B] FIG. 2B forms a schematic block diagram of a general-purpose computer system on which the distributed machine task system of FIG. 1 may be implemented. [Figure 3] FIG. 3 is a schematic block diagram illustrating an exemplary CNN architecture with a regular image classification CNN. [Figure 4]FIG. 4 is a schematic graph showing the backbone network for the ResNeXt 101 layer network. [Figure 5] Figure 5 is a schematic graph showing the YOLOv3 network. [Figure 6] Figure 6 is a schematic graph showing the YOLOv4 network. [Figure 7] FIG. 7 shows how neural network features and associated metadata are encoded into the bitstream. [Figure 8] FIG. 8 shows a method for decoding neural network features from a bitstream and performing a desired computer vision task. DETAILED DESCRIPTION OF THE INVENTION

[0043] Where reference is made in any one or more of the accompanying drawings to steps and / or features having the same reference numerals, those steps and / or features have the same function or operation for the purposes of this description, unless intended to the contrary.

[0044] As mentioned above, one potential pipeline for machine-readable video compression is feature coding. FIG. 1 is a schematic block diagram illustrating the functional modules of a distributed feature coding system 100. The system 100 includes a source device 110 and a destination device 140. A communication channel 130 is used to communicate encoded feature information from the source device 110 to the destination device 140. The source device 110 may include an edge device, such as a network camera, a smartphone, or a system of devices incorporating the functional modules included in the source device 110. The communication channel 130 may be a wired connection, such as Ethernet, or a wireless connection, such as WiFi or 5G. The destination device 140 may be a centralized device, such as a server farm-based (“cloud”) application or an automated surveillance system. Furthermore, edge device functionality may be embodied in a cloud server, and intermediate compressed data may be stored for later processing, potentially for multiple different tasks, as needed.

[0045] 1, source device 110 includes a video source 112, a frame pre-processing module 114, a CNN backbone 116, a feature encoder 120, and a transmitter 122. Video source 112 typically comprises a source of captured video frame data (shown as 113), such as an image capture sensor, a previously captured video sequence stored on a non-transitory recording medium, or a video feed from a remote image capture sensor.

[0046] The frame preprocessing module 114 may receive the video frame data 113, perform image preprocessing steps such as image alignment, white balancing, and image resizing, and output preprocessed frame data 115. Image preprocessing may be performed to improve the performance of the CNN backbone 116. For example, image resizing may be useful as a preprocessing step to control the computational requirements of the CNN backbone 116 and the feature encoder 120. The CNN backbone 116 may also be trained to perform computer vision tasks, also known as machine vision tasks, over an optimal range of image scales, and image resizing may be performed to generate preprocessed frame data 115 having spatial dimensions within the optimal range of image scales.

[0047] The CNN backbone 116 receives preprocessed frame data 115 and provides the frame data 115 for propagation through the initial layers of the overall CNN architecture. The initial layers of the overall CNN architecture are called the "backbone" or "backbone network" of the CNN and can also output a set of features. As mentioned above, the features generated by the initial layers of a CNN tend to capture low-level visual characteristics such as edges and texture. Low-level visual characteristics are fundamental to computer vision and are not specialized for specific machine tasks. The overall CNN architecture can support multiple different CNNs, each performing a different computer vision task but each sharing the same CNN backbone 116. For example, in one arrangement, the overall CNN architecture can be a Detectron2 architecture. In a configuration using the Detectron2 architecture, the CNN backbone 116 can be a ResNet backbone or a ResNeXt backbone. In another arrangement, the overall CNN architecture can be a YOLO architecture. In the case of a YOLO architecture, the CNN backbone 116 can be a Darknet53 backbone or a CSP-Darknet53 backbone.

[0048] The feature coding system 100 corresponds to a particular CNN selected to perform a desired computer vision task. The selected CNN is split into an initial layer, called a backbone network, and the remaining layers, called a "head" or "head network." The CNN backbone 116 generalizes across multiple computer vision tasks, while the head network is specialized for the desired computer vision task of the feature coding system 100. In one arrangement, the split between the backbone network and the head network occurs at a single point within the CNN. The starting layer of the head network is determined from the split point. When the split occurs at a single point, the CNN backbone 116 outputs a backbone feature 117 corresponding to a single tensor output from the last layer of the backbone network. In another arrangement, the split between the backbone network and the head network occurs at multiple points within the CNN. The multiple split points typically correspond to different spatial scales along the backbone 116. The backbone feature 117 represents the video data processed by at least a first portion of the CNN, the backbone network.

[0049] When multiple segmentation points are used, the backbone features 117 consist of multiple tensors corresponding to the output of the backbone network 116 at each of the segmentation points. In a multiple segmentation configuration, the segmentation points typically occur at different spatial scales of the backbone network, and the backbone features 117 are sometimes referred to as a "feature pyramid network" (FPN). FPN features are generally larger in size than single-layer features, but FPN features can capture information across a wider range of spatial scales. Computer vision tasks performed with FPN features can typically achieve higher performance. For example, object detection based on FPN features can typically achieve better accuracy and precision.

[0050] The feature encoder 120 receives and encodes the backbone features 117, thereby encoding the tensors generated by the CNN backbone module 116. In the example of FIG. 1, the feature encoder 120 encodes the backbone features 117 into a bitstream 121. In other implementations, the feature encoder can store the backbone features 117 in a different format or structure, such as a packed frame structure or another structured storage structure. In one configuration, the feature encoder 120 can reuse a conventional hybrid video encoder. For example, the feature encoder 120 may use a hybrid video encoder such as an encoder compatible with the High Efficiency Video Coding (HEVC) standard, the Versatile Video Coding (VVC) standard, or the AV1 standard. In a configuration using a conventional hybrid video encoder, the backbone features 117 are first processed into a format suitable for hybrid video coding. The backbone features 117 may be quantized from floating-point to integer representation. The tensors corresponding to the backbone features 117 are packed into frames for video coding. For example, a tensor corresponding to the FPN features of a single video frame 113 has dimensions determined by the number of channels and spatial size, with dimensions varying across the FPN scale. A packing algorithm may rearrange the tensor samples into a monochrome frame, a YCbCr frame, or a set of temporally contiguous frames.

[0051] In another arrangement, the feature encoder 120 may use the backbone features 117 directly without quantization or frame packing. In such a configuration, the feature encoder 120 may be implemented using a neural network encoder. In yet another arrangement, the feature encoder 120 may quantize the backbone features 117 to an integer representation but without performing any frame packing. In an arrangement with quantization but without frame packing, the feature encoder 120 may use an integerized neural network encoder, or a designed algorithm for feature coding, or a combination of both.

[0052] In addition to the backbone features 117, the feature encoder 120 also encodes metadata into the bitstream 121. The metadata identifies the CNN backbone 116 used to generate the backbone features 117. The backbone is uniquely identified by indicating both the overall CNN architecture and the split point or specific split points that separate the backbone network from the head network. The overall CNN architecture may be signaled in the metadata by the syntax element cnn_architecture, with semantics as shown in Table 1 below. The CNN architectures listed in Table 1 are exemplary only and not exhaustive. The split point or split points may be signaled in the metadata by the syntax element network_split_points. The signaling mechanism and semantics of network_split_points are described in the following configuration with reference to Figures 3, 4, 5, and 6, and Table 5.

[0053] [Table 1]

[0054] Bitstream 121 is transmitted by transmitter 122 as encoded feature data over communication channel 130. Bitstream 121 may in some implementations be stored in a non-transitory storage device 132, such as "flash" memory or a hard disk drive, until later transmitted over communication channel 130, or in lieu of transmission over communication channel 130. For example, the encoded feature data may be accessed on demand from storage for a video surveillance application.

[0055] Destination device 140 includes receiver 142, feature decoder 150, CNN head 154, task result buffer 156, and optionally display device 158. Receiver 142 receives encoded feature data from communication channel 130 and passes the received feature data as a bitstream to feature decoder 150 (indicated by arrow 143). Feature decoder 150 decodes backbone features 151 as well as metadata 152 from bitstream 143. In implementations in which the feature encoder encoded the feature data into a structure other than a bitstream, decoder 150 decodes backbone features 151 and metadata 152 based on a different structure. Similar to feature encoder 120, in one arrangement, feature decoder 150 may use a hybrid video decoder, such as a decoder compatible with the High Efficiency Video Coding (HEVC) standard, or the Versatile Video Coding (VVC) standard, or the AV1 standard. In a configuration using a conventional hybrid video decoder, decoded video frames are unpacked and dequantized into floating-point format tensors suitable for insertion into the head network. In another arrangement, the feature decoder 150 can directly decode the backbone features 151 as floating-point tensors. In a direct decoding arrangement, the feature decoder 150 can be implemented using a neural network decoder. In yet another arrangement, the feature decoder 150 can decode integer precision tensors using an integerized neural network decoder, an algorithm designed for feature decoding, or a combination of both. In such a configuration, the integer precision tensors can be dequantized to floating-point tensors or passed directly to the CNN head 154 as integer precision tensors.

[0056] The decoded metadata 152 identifies the CNN backbone 116 used to generate the decoded backbone features 151. The decoded metadata 152 may uniquely identify the CNN backbone 116 by a syntax element cnn_architecture, which indicates the overall CNN architecture, and a syntax element network_split_points, which identifies the split points or specific split points separating the backbone network from the head network. The decoded metadata 152 may be obtained from a supplementary enhancement information (SEI) message present in the bitstream 143. In one configuration, the decoded metadata 152 may be present and decoded from the bitstream every frame. In another arrangement, the decoded metadata 152 may be present and decoded less frequently than every frame. For example, the decoded metadata 155 may be decoded from a header of the bitstream 143, such as a sequence parameter set (SPS) or a video parameter set (VPS). When the decoded metadata 152 does not exist for a given frame, the most recent available metadata is used.

[0057] As described above, the CNN backbone 116 in the source device 110 generalizes across multiple computer vision tasks, while the head network is specialized for the desired computer vision task of the feature coding system 100. For example, the desired computer vision task may be, but is not limited to, any one of image classification, object detection, object segmentation, object tracking, pose estimation, video reconstruction, or action recognition. The CNN head module 154 first identifies the CNN backbone 116 used from the decoded metadata 152. A head network that is compatible with the CNN backbone 116 and specialized for the desired computer vision task is selected. The selected head network receives the decoded backbone features 151 and executes the remaining “head network” layers of the overall CNN architecture. In some arrangements, the output of the selected head network is the task result 155. In other arrangements, the output of the selected head network may be further processed by the CNN head module 154 to generate the task result 155. The CNN head module 154 performs the computer vision task in generating the task result 155. For example, for an object detection task, the selected head network may output an object detection proposal. The object detection proposals are filtered down to the object detection result by discarding object detection proposals with low confidence scores and discarding object detection proposals with significant overlapping areas. The task results 155 are stored in a task result buffer 156. The task results 155 stored in the task buffer can be used by the destination device 140 to complete an action associated with the machine vision task result. For example, the detected object can be output to a security application, displayed, or an alert can be issued, or the tracked object's path can be used for security purposes.

[0058] The task results 155 may also be displayed on an optional display device 158. For example, in the case of an object detection or object tracking task, bounding boxes associated with detected objects may be plotted on the display device 158. In another example, if the computer vision task is the reconstruction of original video frame data 113, the task results 155 are reconstructed video data that may be displayed on the display device 158. Examples of display devices 158 include cathode ray tubes, liquid crystal displays, light emitting diode (LED) displays, organic LED (OLED) displays, and quantum dot LED (QLED) displays.

[0059] In the above configuration, both the backbone features 117 and the metadata identifying the CNN backbone 116 are encoded into the same bitstream 121. At the destination device 140, both the backbone features 151 and the metadata 152 are decoded from the same bitstream 143. In another arrangement, separate bitstreams may be used to transmit the backbone features and the metadata. For example, the metadata identifying the CNN backbone 116 is encoded into the bitstream 121, while the backbone features 117 are encoded into a separate feature bitstream. At the destination device 140, the metadata 152 is decoded from the bitstream 143, and the backbone features 151 are decoded from the separate feature bitstream. One advantage of transmitting the metadata and backbone features in separate bitstreams is that the metadata can be transmitted in a separate “out-of-band” channel. Metadata transmitted in a separate channel can be transmitted in a channel with greater error resilience and lower bandwidth requirements and can be decoded independently from the processing of the separate feature bitstream.

[0060] Notwithstanding the exemplary devices described above, each of source device 110 and destination device 140 may generally be configured within a general-purpose computing system through a combination of hardware and software components. Figure 2A illustrates such a computer system 200, including a computer module 201, input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 (which may be configured as video source 112), a microphone 280, and output devices including a printer 215, a display device 214 (which may be configured as display device 158), and a loudspeaker 217. An external modulator-demodulator (modem) transceiver device 216 may be used by computer module 201 to communicate with a communications network 220 via connection 221. Communications network 220, which may represent communications channel 130, may be a WAN (wide area network), such as the Internet, a cellular telecommunications network, or a private WAN. If connection 221 is a telephone line, modem 216 may be a conventional "dial-up" modem. Alternatively, the modem 216 may be a broadband modem if the connection 221 is a high-capacity (e.g., cable or optical) connection. A wireless modem may also be used for wireless connection to the communication network 220. The transceiver device 216 may provide the functionality of the transmitter 116 and receiver 142, and the communication channel 130 may be embodied in the connection 221.

[0061] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 can include semiconductor random access memory (RAM) and semiconductor read-only memory (ROM). The computer module 201 also includes several input / output (I / O) interfaces, including an audio-video interface 207 that couples to a video display 214, a loudspeaker 217, and a microphone 280, an I / O interface 213 that couples to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and optionally a joystick or other human interface device (not shown), and an interface 208 for an external modem 216 and a printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is typically the output of a computer graphics card. In some implementations, the modem 216 can be integrated into the computer module 201, for example, into the interface 208. The computer module 201 also has a local network interface 211 that allows the computer system 200 to be coupled via a connection 223 to a local area communication network 222, known as a local area network (LAN). As shown in Figure 2A, the local communication network 222 can also be coupled via a connection 224 to a wide network 220, which typically includes a so-called "firewall" device or device of similar functionality. The local network interface 211 can be an Ethernet TM Circuit card, Bluetooth TM The local network interface 211 may have a wireless or IEEE 802.11 wireless configuration, although numerous other types of interfaces may be implemented for the interface 211. The local network interface 211 may also provide the functionality of the transmitter 122 and receiver 142, and the communication channel 130 may also be embodied in a local communication network 222.

[0062] The I / O interfaces 208 and 213 can provide either or both serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). A storage device 209 is provided and typically includes a hard disk drive (HDD) 210. Other storage devices, such as a floppy disk drive or magnetic tape drive (not shown), may also be used. An optical disk drive 212 is typically provided to serve as a non-volatile source of data. For example, an optical disk (e.g., CD-ROM, DVD, Blu-ray Disc) may be used. TM ), USB-RAM, portable, external hard drives, and portable memory devices such as floppy disks can be used as suitable sources of data to computer system 200. Typically, any of HDD 210, optical drive 212, and networks 220 and 222 may be configured to operate as video source 112 or as a destination for decoded video data to be stored for playback via display 214. Source device 110 and destination device 140 of system 100 may be embodied in computer system 200.

[0063] The components 205-213 of the computer module 201 typically communicate via an interconnected bus 204 and in a manner which results in a conventional mode of operation of the computer system 200 known to those skilled in the art. For example, the processor 205 is coupled to the system bus 204 using a connection 218. Similarly, the memory 206 and optical disk drive 212 are coupled to the system bus 204 by a connection 219. Examples of computers on which the described arrangements can be practiced include IBM-PCs and compatibles, SunSPARCstations, Apple Macs, and the like. TM , or a similar computer system.

[0064] Where appropriate or desirable, feature encoder 120 and feature decoder 150, as well as the methods described below, may be implemented using computer system 200. In particular, feature encoder 120, feature decoder 150, and the methods described may be implemented as one or more software application programs 233 executable within computer system 200. In particular, feature encoder 120, feature decoder 150, and the steps of the methods described are performed by instructions 231 (see FIG. 2B ) in software 233 executing within computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more specific tasks. The software may also be divided into two separate portions, with a first portion and corresponding code modules performing the methods described and a second portion and corresponding code modules managing the user interface between the first portion and the user.

[0065] The software may be stored on a computer-readable medium, including, for example, the storage devices described below. The software is loaded from the computer-readable medium into computer system 200 and then executed by computer system 200. Such software or a computer-readable medium having a computer program recorded thereon is a computer program product. Use of the computer program product in computer system 200 preferably results in advantageous apparatus for implementing source device 110 and destination device 140 and the methods described.

[0066] The software 233 is typically stored on the HDD 210 or in the memory 206. The software is loaded into the computer system 200 from a computer-readable medium and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 that is read by the optical disk drive 212.

[0067] In some examples, application program 233 may be provided to a user encoded on one or more CD-ROMs 225 and read via corresponding drive 212, or may be read by a user from network 220 or 222. Additionally, software may be loaded into computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-transitory, tangible storage medium that provides recorded instructions and / or data to computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROMs, DVDs, and Blu-ray Discs. TM , a hard disk drive, a ROM or integrated circuit, a USB memory, a magneto-optical disk, or a computer-readable card such as a PCMCIA card, such devices being internal or external to the computer module 201. Examples of transitory or non-tangible computer-readable transmission media that may also participate in providing software, application programs, instructions, and / or video data or encoded video data to the computer module 401 include wireless or infrared transmission channels, as well as network connections to other computers or networked devices, and the Internet or intranets, including email transmissions and information stored on websites and the like.

[0068] A second portion of application program 233 and the corresponding code modules described above may be executed to implement one or more graphical user interfaces (GUIs) that are rendered or represented on display 214. A user of computer system 200 and applications, typically through operation of keyboard 202 and mouse 203, may manipulate the interface in a functionally adaptable manner to provide control commands and / or input to applications associated with the GUI. Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface that utilizes speech prompts output via loudspeaker 217 and user voice commands input via microphone 280.

[0069] Figure 2B is a detailed schematic block diagram of processor 205 and "memory" 234. Memory 234 represents the logical collection of all memory modules (including HDD 209 and semiconductor memory 206) accessible by computer module 201 of Figure 2A.

[0070] When the computer module 201 is first powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in the ROM 249 of the semiconductor memory 206 of FIG. 2A. Hardware devices, such as the ROM 249, that store software are sometimes referred to as firmware. The POST program 250 inspects the hardware within the computer module 201 to ensure proper functionality and typically checks the processor 205, memory 234 (209, 206), and a basic input / output system software (BIOS) module 251, also typically stored in the ROM 249, for correct operation. If the POST program 250 executes successfully, the BIOS 251 boots the hard disk drive 210 of FIG. 2A. The booting of the hard disk drive 210 causes a bootstrap loader program 252, resident on the hard disk drive 210, to execute via the processor 205. This loads the operating system 253 into the RAM memory 206, and the operating system 253 begins operation. Operating system 253 is a system-level application executable by processor 205 and performs a variety of high-level functions, including processor management, memory management, device management, storage management, software application interface, and general-purpose user interface.

[0071] The operating system 253 manages the memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has enough memory to run without conflicting with memory allocated to another process. Furthermore, the different types of memory available in the computer system 200 of FIG. 2A need to be used appropriately so that each process can run effectively. Thus, the aggregate memory 234 is not intended to show how specific segments of memory are allocated (unless otherwise specified), but rather to provide a general view of the memory accessible by the computer system 200 and how such memory is used.

[0072] As shown in FIG. 2B, processor 205 includes several functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248, sometimes referred to as a cache memory. Cache memory 248 typically includes several storage registers 244-246 within a register section. One or more internal buses 241 functionally interconnect these functional modules. Processor 205 also typically has one or more interfaces 242 for communicating with external devices via system bus 204 using connection 218. Memory 234 is coupled to bus 204 using connection 219.

[0073] Application program 233 includes a set of instructions 231, which may include conditional branch and loop instructions. Program 233 may also include data 232 used in the execution of program 233. Instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative sizes of instructions 231 and memory locations 228-230, a particular instruction may be stored in a single memory location, as indicated by the instruction shown in memory location 230. Alternatively, an instruction may be segmented into several parts, each stored in a separate memory location, as indicated by the instruction segments shown in memory locations 228 and 229.

[0074] Generally, the processor 205 is given a set of instructions to execute therein. The processor 205 awaits a subsequent input to which the processor 205 responds by executing another set of instructions. Each input may be provided from one or more of several sources, including data generated by one or more of the input devices 202, 203, data received from an external source over one of the networks 220, 202, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into a corresponding reader 212, all shown in FIG. 2A. Execution of the set of instructions may in some cases result in the output of data. Execution may also involve storing data or variables in memory 234.

[0075] The feature encoder 120, feature decoder 150, and described methods may use input variables 254, which are stored in corresponding memory locations 255, 256, 257 in memory 234. The feature encoder 120, feature decoder 150, and described methods generate output variables 261, which are stored in corresponding memory locations 262, 263, 264 in memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0076] 2B, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 cooperate to perform the sequence of micro-operations required to perform a "fetch, decode, and execute" cycle for each instruction in the instruction set that makes up program 233. Each fetch, decode, and execute cycle: a fetch operation to fetch or read an instruction 231 from a memory location 228, 229, 230; a decode operation in which the control unit 239 determines which instructions have been fetched; and an execution operation in which the control unit 239 and / or the ALU 240 execute the instruction.

[0077] A further fetch, decode, and execute cycle for the next instruction may then be performed. Similarly, a store cycle may be performed by the control unit 239 storing or writing a value to the memory location 232.

[0078] Each step or sub-process in the described methods of FIGS. 7 and 8 is associated with one or more segments of program 233 and is typically performed by register sections 244, 245, 247, ALU 240, and control unit 239 within processor 205, which cooperate to perform fetch, decode, and execute cycles for all instructions in the instruction set for the referenced segment of program 233.

[0079] The feature coding system 100 performs a desired computer vision task using a particular CNN selected for the desired computer vision task. The selected CNN is partitioned into a first portion, which is a backbone network executed by the CNN backbone 116. The selected CNN is also partitioned into a different second portion, which is a head network executed by the CNN head 154. As described below, the CNN is partitioned into at least a backbone network and a head network, although the partition point (and thereby the backbone and head networks implemented) may vary. Figure 3 is a schematic block diagram illustrating a general policy architecture 300 that can define the partition point between the backbone network and the head network.

[0080] In the policy of Figure 3, the selected CNN belongs to an overall CNN architecture in which there is a canonical image classification CNN 310. Image classification is a fundamental computer vision task upon which more complex computer vision tasks are typically built. For example, the ResNet and ResNeXt CNN architectures are first described in terms of the configuration of neural network layers trained to accomplish the image classification task. Similarly, the YOLOv3 CNN architecture includes a backbone network called Darknet53 pre-trained for image classification, and the YOLOv4 CNN architecture includes a backbone network called CSP-Darknet53 pre-trained for image classification.

[0081] Video data 311 is input to image classification CNN 310. In the example of FIG. 3, the video data 311 is processed by an exemplary sequence of backbone layer modules 312, 314, and 316 to generate backbone feature tensors 313, 315, and 317, respectively. The number of backbone layer modules and the corresponding backbone feature tensors is only three, as an example. In general, in other implementations, any number of backbone feature tensors may be extracted from the backbone network of image classification CNN 310 as appropriate. The final backbone feature tensor 317 is input to image classification head 320 to generate classification task result 321.

[0082] In the policy of Figure 3, backbone feature tensors 313, 315, and 317 can alternatively be input to a CNN head 330 that performs a computer vision task other than image classification, generating task result 340. Referring to the example ResNet and ResNeXt CNN architectures above, for the same backbone used by image classification, these architectures define head networks that can perform object detection and object segmentation. The YOLOv3 and YOLOv4 CNN architectures define head networks that perform object detection from the output of their respective backbone networks.

[0083] In one configuration of the feature coding system 100, the selected CNN is split into a backbone network and a head network according to the policy of Figure 3. The policy of Figure 3 can be summarized as follows: The backbone network consists of the maximal set of backbone layer modules 312, 314, and 316 that are entirely contained within the canonical image classification CNN 310 for the overall CNN architecture and may also be used to perform at least one computer vision task other than image classification. The head network 330 takes as input the backbone feature tensors 313, 315, and 317. Thus, the split point between the backbone network and the head network is the midpoint that is entirely contained within the canonical image classification CNN 310. The tensor data received at the CNN head 330 is processed by at least a portion of the canonical image classification CNN 310, one or more of the CNN backbone layers 312, 314, and 316. The split point is indicated by signaling the syntax element network_split_points, which has the semantic meaning that the default split point is used. At the destination device 140, the CNN head 154 determines the overall CNN architecture from the cnn_architecture syntax element in the decoded metadata 152, and if the network_split_points syntax element is determined to be default, the pre-defined split points for the determined CNN architecture are used.

[0084] FIG. 4 shows a graph representation of an exemplary backbone network 400 for the ResNeXt 101-layer architecture. Backbone network 400 satisfies the policy described in connection with FIG. 3. That is, backbone network 400 is completely contained within the canonical ResNeXt 101-layer image classification network. Each node, such as node 410, in exemplary backbone network 400 represents a neural network layer or a group of neural network layers. Each arrow, such as arrow 420, in exemplary backbone network 400 represents a backbone feature tensor that crosses from one neural network layer to another. In exemplary backbone network 400, the outputs of nodes P2, P3, P4, P5, and P6 (402-406, respectively) are predetermined split points for the ResNeXt 101-layer architecture. The backbone, defined by the split points at the outputs of nodes P2, P3, P4, P5, and P6, is signaled by setting cnn_architecture to zero (which has the semantic meaning of the ResNeXt101 layer architecture in Table 1) and network_split_points to a default value of zero. Different computer vision tasks can be achieved by using different head networks that are compatible with backbone network 400. For example, object detection may be performed by using a "Faster R-CNN" head network, and object segmentation may be performed by using a "Mask R-CNN" head network.

[0085] Alternate split points are outputs of the node "stem" in the backbone network 400. In implementations that use split points after the node stem, the backbone network is signaled by setting cnn_architecture to zero and network_split_points to one. The range of values ​​that network_split_points can take and the corresponding semantic meaning depend on the overall CNN architecture, which is first determined by decoding cnn_architecture. An exemplary mapping of values ​​to semantics for network_split_points for the exemplary backbone network 400 of FIG. 4 is shown in Table 2 below. In the example of Table 2, a fixed number of values ​​of the network_split_points syntax element map to a fixed number of predetermined split points.

[0086] [Table 2]

[0087] For a particular computer vision task, the feature coding system 100 can flexibly use any one of several CNN architectures according to the tradeoff between task performance, complexity, and bit rate. Task performance can be estimated offline using test data with ground truth. Task performance metrics depend on the specific computer vision task. For example, object detection and segmentation performance can be measured by mean average precision (mAP), object tracking performance can be measured by multiple object tracking accuracy (MOTA), and video reconstruction performance can be measured by peak signal-to-noise ratio (PSNR). High task performance can be important for use cases that are sensitive to machine task errors, such as fully autonomous vehicles. The complexity of the feature coding system 100 is affected by the CNN architecture because smaller CNNs have fewer multiply-add operations. A lower-complexity CNN architecture is advantageous for reducing the cost of products that implement the feature coding system 100. The compression efficiency of the feature coding system 100 depends on two main factors: the size of the backbone features 117 and the efficiency of the feature encoder 120 in exploiting redundancy in the backbone features 117. The choice of CNN architecture affects the size of the backbone features 117 and therefore the bitrate of the bitstream 121.

[0088] Furthermore, for a particular CNN architecture used by feature coding system 100 and a desired computer vision task, CNN head 154 can flexibly select any one of several head networks according to a trade-off between task performance and complexity. The selected head network must be compatible with the decoded backbone features 151 and perform the desired computer vision task, but within these constraints, multiple head networks of varying complexity may be available for selection.

[0089] The ResNeXt101-layer network described with reference to FIG. 4 provides one CNN architecture option that provides high task performance, but also high complexity and a relatively high bitrate. One alternative is a low-complexity CNN architecture, such as the YOLOv3 architecture. The YOLOv3 architecture has lower task performance, but low complexity and also provides a relatively low bitrate. Relative bitrate performance is further described below with reference to Table 3.

[0090] FIG. 5 shows a graphical representation of a YOLOv3 network 500. Each node, such as node 510, in YOLOv3 network 500 represents a neural network layer. Each arrow, such as arrow 520, in YOLOv3 network 500 represents a feature tensor that spans from one neural network layer to another. Different types of neural network layers are represented in YOLOv3 network 500 by different node shapes. The different types of neural network layers, whose shapes are identified by table 530, include convolutional layers, downsampling layers, summation layers, concatenation layers, upsampling layers, detection layers, and pooling layers (not shown in FIG. 5).

[0091] For each type of neural network layer represented by a node in the YOLOv3 network 500, the operation of the neural network layer may include a batch normalization step and a nonlinear step. For example, a convolutional layer generally consists of a convolution step, followed by a batch normalization step, and finally a nonlinear step. The batch normalization step is sometimes referred to as a batch normalization layer, and the nonlinear step is sometimes referred to as a nonlinear layer.

[0092] Batch normalization multiplies each element of the input tensor by a scaling factor γ and then adds an offset β, where γ and β are learned during network training and are generally vectors with length equal to the number of input channels. In other words, γ and β apply the same values ​​across the batch and spatial dimensions. Assuming the input tensor is statistically invariant to the tensor, the batch normalization step is trained, and the output of the batch normalization is normalized to have a standard Gaussian distribution, as the name of the step suggests.

[0093] A nonlinear step may follow batch normalization, applying a nonlinear mapping to each element of the input tensor. The nonlinear mapping is sometimes called an "activation" or "activation function." The nonlinear mapping can be a sigmoid function, a rectified linear unit (ReLU), a "leaky" ReLU, a "mish" function, or some other nonlinear mapping function. The use of a nonlinear step within a neural network layer distinguishes neural networks from other machine learning techniques, such as support vector machines.

[0094] Convolutional layers are the most common type of neural network layer typically used in CNNs. The input tensor T to an exemplary convolutional layer has dimensions BxCxHxW, where B is the batch size, C is the number of input channels, H is the height of the input space, and W is the width of the input space. The operation of the convolutional layer is determined by a set of learned weights ω with dimensions CxYxXxO, where X and Y are the height and width of the convolutional support S, respectively, and O is the number of output channels. For a convolutional layer with a stride of 1, the output tensor U with dimensions BxOxHxW is calculated using the following equation (1):

[0095]

number

[0096] In the above equation, samples indexed from T that are spatially outside the range of T may be estimated by a bound extension policy. For example, the sample may be set to zero (zero bound extension), set equal to the spatially closest sample in T (constant bound extension), or set by some other policy. In convolutional layers where bound extension is not used, the output tensor U has smaller spatial dimensions, with a reduction in spatial dimensions related to the support size of S.

[0097] Convolutional layers are distinguished by the weights associated with them, which have spatial dimensions corresponding to the support size of the convolution operation. In contrast, fully connected layers have weights whose spatial dimensions are determined by the spatial dimensions of the input tensor. The spatial dimensions of the convolutional support size are usually relatively small compared to the spatial dimensions of the input tensor. For example, a convolutional layer may have a support size of 3x3, 5x5, or 7x7. Convolutional layers may be referred to by their support size as a "1x1 convolutional layer," a "3x3 convolutional layer," etc. By applying spatial weights in a sliding window fashion across the input tensor, convolutional layers are much less complex than fully connected layers and can also model spatially regularized behavior.

[0098] In the YOLOv3 network 500, the node representing a convolutional layer is for a convolutional layer with a stride of 1. The spatial dimension of the output tensor from each convolutional layer is equal to the spatial dimension of the corresponding input tensor. Conversely, a convolutional layer with a stride greater than 1 is represented in the YOLOv3 network 500 with a node labeled as a downsampling layer. A convolutional layer with a stride s has an operational operation that can be described by Equation (2).

[0099]

number

[0100] In the YOLOv3 network 500, the downsampling layers have a stride of 2, so the spatial dimensions of the output tensor from each downsampling layer are halved relative to the spatial dimensions of the corresponding input tensor. Downsampling layers are used to progressively reduce the spatial resolution of data passing through the neural network and are one building block used to generate the feature pyramid.

[0101] An addition layer is a relatively simple operation in a neural network. It can take multiple input tensors, each with the same dimensions. The output tensor is calculated by an element-wise sum over each of the input tensors. Because addition layers have no associated learned weights, they are sometimes called "addition operations" or "element-wise addition."

[0102] Although a simple operation, the summation layer plays a key role in forming the "residual block." A residual block typically ends with an summation layer that combines two tensors resulting from two parallel passes through the residual block. The first pass through the residual block is a direct copy of the input tensor to the residual block and is typically called a "shortcut connection." The second pass through the residual block passes through a sequence of convolutional layers. The sequence of convolutional layers is designed to minimize the complexity of the residual block without unduly restricting the residual block's expressibility. In the worst case, the shortcut connection copies the input tensor of the residual block to the output of the residual block, while the second pass through the residual block outputs a zero tensor. In the worst case, the residual block then implements a discriminant function. Thus, during network training, the second pass through the residual block learns only variations from the discriminant function implemented by the shortcut connection. The variations learned by the second pass are sometimes called the "residual function."

[0103] Similar to addition layers, concatenation layers are relatively simple operations with no associated learned weights. Concatenation layers are sometimes called "concatenation operations" or "channel-wise concatenation." Concatenation layers can take multiple input tensors, each with the same batch size and spatial dimension. The output tensor is computed by concatenation of the input tensors across the channel dimensions. In other words, the channel size of the output tensor is equal to the sum of the channel sizes of the input tensors. Channel-wise concatenation is a common step performed to non-destructively combine information from multiple sources.

[0104] An upsampling layer typically does not have associated learned weights. An upsampling layer is sometimes referred to as an "upsampling operation" or "interpolation." An upsampling layer receives an input tensor with dimensions BxCxHxW and produces an output tensor with dimensions BxCxHoxWo, where the output spatial dimensions Ho and Wo are larger than the input spatial dimensions. The way in which additional samples are predicted from the input tensor can be any one of a number of interpolation methods, such as bilinear interpolation, spline interpolation, bandlimited interpolation, or learned interpolation. In the case of learned interpolation, the upsampling layer consists of associated learned weights. In the YOLOv3 network 500, the spatial dimensions of the output tensor from the upsampling layer are doubled relative to the spatial dimensions of the corresponding input tensor. An upsampling layer is used to prepare tensors with different spatial resolutions so that the information contained in the tensors can be combined by a concatenation layer.

[0105] In the YOLOv3 network 500, the detection layer is a 1x1 convolutional layer trained to generate object detection proposals. From the output tensor generated by the detection layer, each spatial location can be called a detection cell. The number of output channels is designed based on the number P of object detection proposals each cell should generate and the number k of different object classes the detection layer is trained to recognize. For each object detection proposal, the detection layer generates a bounding box coordinate prediction, an "objectness" score, and a confidence score for each of the k object classes. The number of output channels is P*(k+5).

[0106] The detection layer of the YOLOv3 network 500 is an example of an output layer of a CNN and does not limit the applicability of the feature coding system 100 to other neural networks and other computer vision tasks. For other computer vision tasks, the output layer may be a convolutional layer with a different structure, or a fully connected layer, or another type of neural network layer. The only requirement of the output layer is that it produces raw results that can be interpreted or processed to produce the desired computer vision task result.

[0107] Although not shown in the YOLOv3 network 500, pooling layers, used in many neural networks, provide another means of reducing the spatial resolution of data passing through a neural network. From the input tensor to the pooling layer, groups of samples ("pools") are selected spatially spaced with a stride of s. The size of the pools may be s x s if the pools are non-overlapping. The size of the pools may be greater than s x s if the pools overlap. The output tensor of the pooling layer is generated by computing a representative value from each pool. For example, the representative value may be the average of the pooled samples, in which case the layer may be called an average pooling layer. Alternatively, the representative value may be the maximum of the pooled samples, in which case the layer may be called a "max pooling" layer. One special case of a pooling layer is when s is set to the spatial size of the input tensor, in which case a single representative value is selected for each input channel. This special case is called a "global pooling" layer.

[0108] In one arrangement of the feature coding system 100, the YOLOv3 network 500 is split into a backbone network and a head network according to the policy of FIG. 3 . That is, nodes 0 through 74 contained within the YOLOv3 network 500 are defined to constitute the backbone network because they are identified as being completely contained within the canonical image classification network of the YOLOv3 network 500. The remaining nodes 75 through 106 contained within the YOLOv3 network 500 are defined to constitute the head network. The split points between the backbone network and the head network (520 through 522, respectively) are the edges between the pairs of nodes (36, 98), (61, 86), and (74, 75). This split point can be signaled in the bitstream 121 by setting the syntax element network_split_points to a default value of 0. In the described implementation, the YOLOv3 network 500 performs object detection. However, in the general YOLOv3 architecture, the head network may be replaced with another head network trained to perform a desired computer vision task.

[0109] As mentioned above, the choice of CNN architecture affects the size of the backbone features 117 and therefore the bitrate of the bitstream 121. For example, the compression ratio of the backbone features extracted from the default segmentation points of the ResNeXt101-layer architecture shown in FIG. 4 and the compression ratio of the backbone features extracted from the default segmentation points of the YOLOv3 architecture shown in FIG. 5 can be evaluated by examining the size of the respective backbone features. The spatial size of the video frame data 113 is assumed to be HxW. The total number of video samples per frame may be 3xHxW for a video with a 444 chroma format, 2xHxW for a video with a 422 chroma format, 1.5xHxW for a video with a 420 chroma format, or HxW for a monochrome video. The spatial size of the preprocessed frame data 115 is assumed to be H * xW * where H is the resizing ratio r for which the goal is to optimize the performance of the CNN in computer vision tasks. * ≒rH, W * ≒rW. A typical resize ratio is 0.5.

[0110] Table 3 below shows the spatial size and number of channels for each of the backbone tensors resulting from the default split points for the ResNeXt101 layer architecture and the YOLOv3 architecture. In Table 3, and similar analyses of backbone tensor sizes below, the batch size of the backbone tensors is assumed to be 1. Batch sizes greater than 1 are typically used during CNN training but provide no benefit during inference. During inference, a batch size of B means that B consecutive video frames from the video frame data 113 are processed by the feature coding system 100 as a unit, rather than sequentially. Thus, the analysis of Table 3 for B=1 examines the resulting backbone tensor size per video frame.

[0111] As shown in Table 3, there are five backbone feature tensors corresponding to the default split points of the ResNeXt101 layer architecture. Each feature tensor has 256 channels, and collectively, the backbone feature tensors span a dyadic pyramid of spatial resolution. The total number of backbone tensor samples corresponding to the default split points of the ResNeXt101 layer architecture is H * xW * x21.3125. For a resize ratio of 0.5, the total number of backbone tensor samples for the spatial dimensions of video frame data 113 is HxWx5.328125. Thus, for a typical resize ratio of 0.5, the total number of backbone tensor samples corresponding to the default split point of the ResNeXt101 layer architecture is greater than the total number of video samples per frame for any of the common chroma formats mentioned above. One drawback of implementing feature coding system 100 using the ResNeXt101 layer architecture is the relatively high bitrate of bitstream 121.

[0112] [Table 3]

[0113] In contrast, the total number of backbone tensor samples corresponding to the default split points in the YOLOv3 architecture is H * xW *x7. When the resize ratio is 0.5, the total number of backbone tensor samples for the spatial dimensions of the video frame data 113 is H x W x 1.75. Therefore, at a typical resize ratio of 0.5, the total number of backbone tensor samples corresponding to the default split point of the YOLOv3 architecture is equivalent to the total number of video samples per frame in the common chroma format described above. One advantage of this configuration of the feature coding system 100 using the YOLOv3 architecture is that the bitrate of the bitstream 121 is relatively lower than that of the ResNeXt101 layer architecture and can be equivalent to the compressed bitstream generated by compressing the video frame data 113 using conventional video coding techniques. However, one disadvantage of this configuration using the YOLOv3 architecture is lower computer vision task performance than that achievable with the ResNeXt101 layer architecture.

[0114] The ResNeXt101 layer architecture provides high task performance but also a relatively high bitrate. The YOLOv3 architecture provides a relatively low bitrate but has lower task performance. Another alternative is a CNN architecture based on YOLOv4, which provides improved task performance compared to YOLOv3 while maintaining a low bitrate. YOLOv4's task performance is typically better than YOLOv3 but lower than that of the ResNeXt101 layer architecture.

[0115] FIG. 6 shows a graphical representation of a YOLOv4 network 600. Each node in the YOLOv4 network 600, such as node 610, represents a neural network layer. Each arrow in the YOLOv4 network 600, such as arrow 620, represents a feature tensor passing from one neural network layer to another. Different types of neural network layers are represented in the YOLOv4 network 600 by different node shapes. Different types of neural network layers are identified by different shapes, as shown in table 630. Different types of neural network layers include convolutional layers, downsampling layers, summation layers, concatenation layers, upsampling layers, detection layers, and pooling layers, as described above in connection with FIG. 5.

[0116] In one arrangement of the feature coding system 100, the YOLOv4 network 600 is split into a backbone network and a head network according to the policy of FIG. 3. That is, nodes 0-104 contained within the YOLOv4 network 600 are defined to constitute the backbone network because they are identified as being completely contained within the canonical image classification network of the YOLOv4 network 600. The remaining nodes 105-161 contained within the YOLOv4 network 600 are defined to constitute the head network. The split points between the backbone network and the head network are the edges between pairs of nodes (54, 129), (85, 119), and (104, 105) (shown as 620-622, respectively). The split points in this configuration can be signaled in the bitstream 121 by setting the syntax element network_split_points to a default value of 0. In this configuration, the YOLOv4 network 600 performs object detection. However, in the general YOLOv4 architecture, the head network may be replaced with another head network trained to perform a desired computer vision task.

[0117] Compared to the YOLOv3 network 500, the YOLOv4 network 600 has superior task performance, typically with a tradeoff of higher complexity. However, the number of split points is the same for the three spatial resolutions of the backbone features. Furthermore, the backbone features resulting from the default split points of the YOLOv4 architecture have the same dimensions, spatial resolution, and number of channels as the backbone features resulting from the default split points of the YOLOv3 architecture. The total number of backbone tensor samples corresponding to the default split points of the YOLOv4 architecture is also H * xW * x7. Therefore, one advantage of this configuration is that the YOLOv4 architecture provides improved task performance compared to the YOLOv3 architecture while maintaining the relatively low bitrate of the YOLOv3 architecture.

[0118] The configurations described above with reference to Figures 5 and 6 demonstrate that for lower complexity CNN architectures, such as the YOLOv3 and YOLOv4 architectures, default split points can be defined according to the policy of Figure 3 such that the resulting backbone features are comparable in total number of samples with the video frame data 113. However, the inventors have determined that neural network feature tensors with significantly fewer samples can be located at alternative split points in the CNN architecture.

[0119] For example, in the YOLOv3 network 500, the default split points 520-522 are indicated by the edges between pairs of nodes (36, 98), (61, 86), and (74, 75). The default split point with the highest spatial resolution 520 is indicated by edge (36, 98). As shown in Table 3, the backbone feature tensor extracted from the default split point (36, 98) has 256 channels. However, alternative split points for the highest spatial resolution (540-542, respectively) can be defined at the edges between pairs of nodes (99, 100), (101, 102), or (103, 104). For each of the alternative split points for the highest spatial resolution, the extracted backbone feature tensor has only 128 channels. The backbone feature tensor extracted from the default split point with intermediate spatial resolution 521, indicated by edge (61, 86), has 512 channels. Alternate split points for intermediate spatial resolution (550-552, respectively) can be defined at the edges between pairs of nodes (87, 88), (89, 90), or (91, 92). For each alternate split point for intermediate spatial resolution, the extracted backbone feature tensor has only 256 channels. The backbone feature tensor extracted from the default split point with the lowest spatial resolution, 522, indicated by edge (74, 75), has 1024 channels. Alternate split points for the lowest spatial resolution (560-562, respectively) can be defined at the edges between pairs of nodes (75, 76), (77, 78), or (79, 80). For each alternate split point for the lowest spatial resolution, the extracted backbone feature tensor has only 512 channels.

[0120] In one arrangement of the feature coding system 100, alternative split points for the YOLOv3 architecture are defined by selecting one of the alternative split points described above for each of the highest, medium, and lowest spatial resolutions. The selected alternative split points are signaled by the syntax element network_split_points using a signaling mechanism described further below with reference to Table 5. The total number of backbone tensor samples corresponding to the selected alternative split points is H * xW * x3.5. For a resize ratio of 0.5, the total number of backbone tensor samples for the spatial dimensions of the video frame data 113 is H x W x 0.875. Thus, for a typical resize ratio of 0.5, the total number of backbone tensor samples corresponding to the alternative split points of the YOLOv3 architecture is less than the total number of video samples per frame of the common chroma format described above. One advantage of this configuration for signaling alternative split points of the YOLOv3 architecture is that the bitrate of the feature coding system 100 can be significantly lower than the bitrate of a compressed bitstream produced by compressing the video frame data 113 using conventional video coding techniques.

[0121] Significantly lower bit rates can also be achieved by selecting alternative split points in the YOLOv4 architecture. In the YOLOv4 network 600, default split points 620-622 are indicated by the edges between pairs of nodes (54, 129), (85, 119), and (104, 105). The backbone feature tensor extracted from the default split point with the highest spatial resolution, indicated by edge (54, 129), has 256 channels. However, alternative split points for the highest spatial resolution (640-643, respectively) can be defined at the edges between pairs of nodes (130, 131), (132, 133), (134, 135), or (136, 137). For each of the alternative split points for the highest spatial resolution, the extracted backbone feature tensor has only 128 channels. The backbone feature tensor extracted from the default segmentation point with intermediate spatial resolution 621, indicated by edge (85, 119), has 512 channels. Alternate segmentation points for intermediate spatial resolution (650 through 656, respectively) can be defined at the edges between pairs of nodes (120, 121), (122, 123), (124, 125), (126, 142), (143, 144), (145, 146), or (147, 148). For each alternate segmentation point for intermediate spatial resolution, the extracted backbone feature tensor has only 256 channels. The backbone feature tensor extracted from the default segmentation point with the lowest spatial resolution 622, indicated by edge (104, 105), has 1024 channels. Alternative split points for the lowest spatial resolution (660 to 666, respectively) can be defined at the edges between pairs of nodes (105, 106), (107, 108), (114, 115), (116, 153), (154, 155), (156, 157), or (158, 159). For each of the alternative split points for the lowest spatial resolution, the extracted backbone feature tensor has only 512 channels.

[0122] In one arrangement of the feature coding system 100, alternative split points for the YOLOv4 architecture are defined by selecting one of the alternative split points described above for each of the highest, medium, and lowest spatial resolutions. The selected alternative split points are signaled by the syntax element network_split_points using a signaling mechanism described further below with reference to Table 5. The total number of backbone tensor samples corresponding to the selected alternative split points is H * xW * x3.5. For a resize ratio of 0.5, the total number of backbone tensor samples for the spatial dimensions of the video frame data 113 is H x W x 0.875. Thus, for a typical resize ratio of 0.5, the total number of backbone tensor samples corresponding to the alternative split points of the YOLOv4 architecture is less than the total number of video samples per frame of the common chroma format described above. One advantage of this configuration for signaling alternative split points of the YOLOv4 architecture is that the bitrate of the feature coding system 100 can be significantly lower than the bitrate of a compressed bitstream produced by compressing the video frame data 113 using conventional video coding techniques.

[0123] Table 4 below shows the total number of backbone tensor samples H * xW * We summarize alternative split points for both YOLOv3 and YOLOv4 architectures, resulting in x3.5.

[0124] [Table 4]

[0125] In one arrangement of feature coding system 100, alternative split points for the CNN architecture may be predetermined and signaled by setting the syntax element network_split_points to a fixed value. Table 2 provides an example where a small number of predetermined split points may be signaled using a fixed value for network_split_points.

[0126] In another arrangement of the feature coding system 100, alternative split points are not predetermined but instead explicitly signaled in the network_split_points syntax element. An exemplary signaling mechanism for alternative split points is shown in Table 5 below for the YOLOv3 architecture, which is signaled by setting cnn_architecture to a value of 4, and for the YOLOv4 architecture, which is signaled by setting cnn_architecture to a value of 5. In the example of Table 5, the network_split_points syntax element is a variable-length code. Bit position 0 of network_split_points is interpreted as a "default split point flag." When the default split point flag is zero, the signaled split point is the predetermined default split point defined by the policy of FIG. 3, and the network_split_points syntax element is 1 bit long. When the default split point flag is 1, the signaled split point is instead determined from three additional fixed-length codes parsed from the network_split_points syntax element. Three additional fixed-length codes are interpreted as offsets of the signaled split point from the predetermined default split point for the highest, medium, and lowest spatial resolution, respectively. The example in Table 5 shows that the offsets are interpreted as signed integers from the fixed-length codes, and that the alternate split point can be located earlier in the default backbone network if a negative offset is signaled, or later in the default head network if a positive offset is signaled. In another arrangement, the offsets may instead be interpreted as unsigned integers, in which case the alternate split points are predetermined by the signaled CNN architecture to be either completely contained within the default backbone network or completely contained within the default head network.

[0127] [Table 5]

[0128] In one configuration, an alternative split point may be determined from an offset by enumerating neural network layers, such as the node labels shown in FIGS. 5 and 6. For example, to signal a medium-resolution alternative split point (143, 144) for a YOLOv4 architecture, a medium-resolution split point offset of +25 may be signaled to indicate the difference between the starting layer 144 of the alternative head network and the starting layer 119 of the default head network. In another arrangement, an alternative split point may instead be determined from an offset by the number of layers earlier or later along the path of execution in the CNN architecture. For example, to signal a medium-resolution alternative split point (143, 144) for a YOLOv4 architecture, a medium-resolution split point offset of +10 may be signaled to indicate that the alternative split point is located 10 layers further along the path of execution after the default split point (85, 119). In both arrangements, the interpretation of the signaled offset is based on the definition of which operations in the CNN architecture are considered to be one “layer.” Different definitions of "stratum", such as when batch normalization is considered a stratum, result in correspondingly different offset values.

[0129] Alternative methods for signaling the splitting used in the bitstream may also be used, for example, by encoding the actual node number, encoding a selection from a subset of split points, encoding an offset from an outer layer, encoding a reference to a lookup table of split points associated with the CNN, etc.

[0130] The head CNN network can have different types of nodes or layers, or can be subject to some constraints based on different types of layers. For example, in some implementations, the starting layer of the CNN head can be limited to layers that are not additive layers. In other implementations, the starting layer of the CNN head is the layer immediately following the convolutional layer. The starting layer of the CNN head can effectively be limited to a set of layers from a predetermined default layer, e.g., layers that are within a predetermined distance or offset of layers from the default layer. In yet other configurations, the default split may be performed at the output. When signaling an offset from a default or output layer, the offset can be limited to a specific set of layers, e.g., layers that are not additive layers and / or within a given distance or offset from the output layer.

[0131] 7 illustrates a method 700 for encoding neural network features and associated metadata into the bitstream 121. The method 700 may be implemented, for example, on the source device 110 under the control of the processor 205 by execution of software 233 stored in the memory 206. The method 700 begins with an encode CNN architecture step 710. In the encode CNN architecture step 710, the method 700 determines a CNN architecture for the feature coding system 100. The CNN architecture may be predetermined if the source device 110 supports only one CNN architecture, or the CNN architecture may be selected from multiple architectures supported by the source device 110 according to a desired tradeoff between complexity, bit rate, and task performance. The determined CNN architecture is signaled by encoding a cnn_architecture syntax element into the bitstream 121. From step 710, the method 700 proceeds to a segmentation point encoding step 720 under the control of the processor 205.

[0132] In a split point encoding step 720, method 700 determines a backbone network for the determined CNN architecture and, correspondingly, determines the location of a split point separating the determined backbone network from the head network for the determined CNN architecture. Step 720 effectively identifies a split of the selected CNN into a first and second portion, each of which is distinct, and encodes into the bitstream information used to determine at least the starting layer of the second (head) portion of the neural network. If source device 110 supports only one backbone network for the determined CNN architecture, the split point may be predetermined, or the split point may be determined by selecting from the backbone networks supported by source device 110 according to a desired tradeoff between the complexity, bit rate, and computer vision task supported by the backbone network. The determined split point is signaled into bitstream 121 by encoding a network_split_points syntax element. From step 720, method 700 proceeds to a backbone inference step 730 under the control of processor 205.

[0133] Steps 710 and 720 effectively encode information for determining at least a starting layer of the head neural network into the bitstream. The starting layer of the second portion is determined from information associating the starting layer with the neural network. The encoded CNN architecture alone may, for example, identify a default split point or starting layer of the head network. Additionally or alternatively, the encoded split point itself may provide information for determining at least a starting layer of the head neural network, for example, by indicating an offset or distance difference between the starting layer of the head network and a predetermined layer.

[0134] In a backbone inference step 730, the method 700 processes a next video frame from the video frame data 113. The next video frame may be preprocessed by a frame processing module 114 implementing image processing steps such as image alignment, white-color balancing, or image resizing to result in a preprocessed frame. The preprocessed frame is inferred by the determined backbone network to generate backbone features 117. Backbone inference may be implemented by the CNN backbone module 116. From step 730, the method 700 proceeds to a feature encoding step 740 under the control of the processor 205.

[0135] The feature encoding step 740 may be performed by the feature encoder 120. In the feature encoding step 740, the method 700 encodes the backbone features 117 into the bitstream 121 or into a separate feature bitstream. Step 740 operates to encode data processed using the first (backbone) portion of the neural network into a bitstream. The method 700 proceeds from step 740 to an end of sequence test 750 under the control of the processor 205.

[0136] Steps 710-740 encode information for the data processed using the first part of the neural network, which is the feature channel generated by the backbone network. In the illustrated example, the information is encoded into a bitstream.

[0137] At the end of sequence test 750, method 700 checks whether there are any remaining frames from video frame data 113. If there are any remaining frames ("N" at step 750), method 700 proceeds to backbone inference step 730. Otherwise ("Y" at step 750), method 700 ends.

[0138] 8 illustrates a method 800 for decoding neural network features from a bitstream 143 to perform a desired computer vision task. Method 800 may be implemented, for example, on destination device 140 under control of processor 205 by execution of software 233 stored in memory 206. Method 800 begins with a CNN architecture decoding step 810. Step 810 may be implemented by feature decoder 150 receiving the bitstream based on data output by at least a first portion of a neural network in CNN backbone module 116. In CNN architecture decoding step 810, method 800 determines the CNN architecture by decoding a cnn_architecture syntax element from bitstream 143. The syntax element indicates the type of CNN architecture selected and encoded in step 710. From step 810, method 800 proceeds under processor control to a segmentation point decoding step 820.

[0139] In a split point decoding step 820, the method 800 determines the locations of the split points in the determined CNN architecture by decoding the network_split_points syntax element from the bitstream 143. From the determined CNN architecture and the determined split points, the method 800 determines the number, shape, and size of the corresponding backbone tensors. The method 800 proceeds to a feature decoding step 830 under the control of the processor 205.

[0140] Steps 810 and 820 effectively decode information for determining at least a starting layer of the head neural network. The starting layer of the second portion is determined from information associating the starting layer with the neural network. The decoded CNN architecture alone can, for example, identify a default split point or starting layer for the head network. Additionally or alternatively, the decoded split point itself can provide information for determining at least a starting layer of the head neural network, for example, by indicating an offset or distance difference between the starting layer and a predetermined layer of the head network.

[0141] The information decoded in step 820 may relate to a single split point or starting layer for the head network, or multiple split points and corresponding starting layers for the head network.

[0142] In a feature decoding step 830, the method 800 decodes backbone features 151 for the next video frame from the bitstream 143 or a separate feature bitstream. The decoded backbone tensors may be referred to as feature map data that represent the captured image data 113. The feature decoding may be guided by the determined number, shape, and size of the backbone tensors. From step 830, the method 800 proceeds, under control of the processor 205, to a head inference step 840.

[0143] The head inference step 840 is performed in the CNN head 154. In the head inference step 840, the method 800 determines a head network compatible with the determined CNN architecture, the determined split points, and the desired computer vision task. The computer vision task may be predetermined for the destination device 140 or selected by an algorithm or a human user. The head network is selected from head networks that satisfy compatibility constraints according to a trade-off between complexity and computer vision task performance and can be supported by the destination device 140. Backbone features 151 are inferred by the determined head network to generate a task result for the current video frame. In step 840, the head network is used to perform at least one computer vision task to decode the bitstream, thereby providing a computer vision result, such as identifying or tracking an object or tracking a pose. The method 800 proceeds from step 840 to an end of sequence test 850.

[0144] Steps 810-830 decode information for the data generated by the first portion of the neural network, the feature channels generated by the backbone network. In the illustrated example, the information is decoded from a bitstream.

[0145] At end of sequence test 850, method 800 checks whether there are any remaining frames to be decoded from bitstream 143. If there are any remaining frames ("N" at step 850), method 800 returns to feature decoding step 830 for the next frame. Otherwise ("Y" at step 850), method 800 ends.

[0146] In the above example, steps 810-840 are implemented on destination device 140. In other arrangements, steps 810-840 may be implemented across two or more devices. For example, steps 810 and 820 may be implemented on destination device 140 to decode the CNN architecture and segmentation points for the CNN. The decoded CNN architecture and segmentation points may be transmitted to an external processor-based device, such as a cloud server. Steps 830 and 840 may be implemented on an external device.

[0147] Different examples are described above for the YOLOv3 and YOLOv4 neural networks. The described configurations may also be used for different types of CNNs. The type of CNN may be encoded into a bitstream as described in connection with step 710 and decoded from the bitstream as described in connection with step 810. Correspondingly, split points indicating the backbone and head split of the CNN may be encoded into the bitstream as described in connection with step 720. The split points may be decoded as described in connection with step 820. The selection of different CNNs and split points may vary based on characteristics such as the required throughput and accuracy, the type of computer vision task required, and the structure of the neural network in terms of layers and channels per node.

[0148] Industrial Applicability The described arrangements are applicable to the computer and data processing industry, particularly digital signal processing for encoding and decoding of signals such as data related to video and image signals, to achieve high compression efficiency.

[0149] The ability to select different CNN split points and encode them into the bitstream allows for flexibility in compression efficiency, as appropriate split points can be identified for a trade-off between desired complexity, bitrate, and task performance. Furthermore, efficiency in encoding can also be realized when the backbone neural network is integrated into an edge device. Because different machine vision tasks can be implemented by different head CNNs, complexity at the decoder side can be reduced and flexibility increased. Reducing the tensor dimension output by the backbone CNN through split point selection can also increase efficiency at the decoder side. The described configuration also allows a different CNN (e.g., YOLOv3 or YOLOv4) to be selected and the selection coded in the bitstream, again allowing for increased options and flexibility. For example, the selection of low-complexity CNN architectures such as YOLOv3 and YOLOv4 and the selection of appropriate split points can produce feature coding that is competitive with traditional coding solutions.

[0150] The described arrangements are particularly useful for encoding and decoding tensors that represent features of data associated with video and image signals. The described arrangements can also be used to encode and decode other information that may be generated by a convolutional neural network having a backbone and head network.

[0151] The above describes only some embodiments of the present invention, modifications and / or variations may be made without departing from the scope and spirit of the present invention, and the embodiments are illustrative and not limiting.

Claims

1. 1. A computer-implemented method comprising: Decoding data generated by the first portion of the neural network from the bitstream; determining a starting layer of the second portion of the neural network, the second portion including the first portion and a second portion different from the first portion; the neural network includes an addition layer that generates a new tensor based on a plurality of tensors; The initiation layer is limited to a layer that is not the addition layer; The method, wherein the data is data to be processed by the second portion of the neural network.

2. 2. The method of claim 1, wherein the initial layer is a convolutional layer.

3. The method of claim 1 , wherein the starting layer is not a downsampling layer.

4. decoding means for decoding data generated by the first portion of the neural network from the bitstream; determining means for determining a starting layer of the second portion of the neural network, the second portion including the first portion and a second portion different from the first portion; Equipped with the neural network includes an addition layer that generates a new tensor based on a plurality of tensors; The initiation layer is limited to a layer that is not the addition layer; The data is data to be processed by the second portion of the neural network.

5. The apparatus of claim 4 , wherein the initial layer is a convolutional layer.

6. The apparatus of claim 4 , wherein the initiation layer is not a downsampling layer.

7. A computer program for causing a computer to carry out the method according to any one of claims 1 to 3.

Images