System and method for machine-oriented video coding using auto-encoder
Multi-task feature encoding is implemented in video encoding through the autoencoder, which solves the problem of inefficiency in traditional video encoding in machine applications, improves the encoding and decoding efficiency of video data, and adapts to the needs of different tasks and priorities.
Patent Information
- Application Number
- CN202380034517.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-02-17
- Filing Date
- 2023-02-15
- Publication Date
- 2024-11-26
AI Technical Summary
Traditional video encoding technology has problems of inefficiency in machine applications, especially in scenarios where real-time analysis and decision-making of large amounts of video data is required. Traditional methods require a lot of time and are difficult to meet the needs of efficiency and speed.
Adaptive neural network using autoencoder as a multi-task feature encoding, encode and decode image data through autoencoder, generate and reconstruct feature data to achieve efficient video encoding and decoding, suitable for the mixing of machines and humans. system.
It improves the encoding efficiency and decoding speed of video data, is suitable for multi-task processing, reduces computing complexity and storage requirements, and adapts to the needs of different tasks and priorities.
Smart Images

Figure CN119032558A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 311,333, filed on February 17, 2022, entitled “AUTOENCODER FRAMEWORK FOR VIDEOCODING FOR MACHINES,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field
[0003] The present invention generally relates to the field of video coding and decoding. In particular, the present invention relates to a system and method for adaptive neural network optimization of multi-task feature coding using an autoencoder. Background Art
[0004] A video codec may include electronic circuits or software that compress or decompress digital video. It can convert uncompressed video to a compressed format and vice versa. In the context of video compression, a device that compresses video (and / or performs some of its functions) may generally be referred to as an encoder, while a device that decompresses video (and / or performs some of its functions) may be referred to as a decoder.
[0005] The format of the compressed data may conform to standard video compression specifications. The compression may be lossy in that the compressed video lacks some information present in the original video. The result of this may include that the decompressed video may have a lower quality than the original uncompressed video because there is not enough information to accurately reconstruct the original video.
[0006] There can be a complex relationship between video quality, the amount of data used to represent the video (e.g., determined by the bitrate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, end-to-end delay (e.g., potential time), etc.
[0007] Motion compensation may include methods of predicting a video frame or a portion thereof by taking into account the motion of a camera and / or objects in the video given a reference frame, such as a previous and / or future frame. It may be used to encode and decode video data for video compression, such as encoding and decoding using the Advanced Video Coding (AVC) standard (also known as H.264) of the Moving Picture Experts Group (MPEG). Motion compensation may describe a picture based on a transformation from a reference picture to a current picture. A reference picture may be in the future when compared to the current picture, and may be earlier in time when compared to the current picture. Compression efficiency may be improved when images may be accurately synthesized based on previously transmitted and / or stored images.
[0008] While video content is typically thought of as intended for human consumption, there is a growing demand for video in industrial settings and other environments where machines rather than humans are required to evaluate the content.
[0009] Recent trends in robotics, surveillance, monitoring, IoT, etc. have introduced use cases where a large portion of all images and videos recorded in the field are consumed only by machines and never reach human eyes. These machines process images and videos with the goal of completing tasks such as object detection, object tracking, segmentation, event detection, etc. Recognizing that this trend is pervasive and will only accelerate in the future, international standardization bodies have begun efforts to standardize image and video coding that is optimized primarily for machine use. For example, in addition to established standards such as Compressed Descriptors for Visual Search and Compressed Descriptors for Video Analytics, standards such as JPEG Ai and Machine-Oriented Video Coding have been launched. Therefore, further improvements in encoding and decoding of video for machine use, as well as encoding and decoding in hybrid systems where both human viewers and machines use the video, are becoming increasingly important in the art.
[0010] In many applications, such as surveillance systems with multiple cameras, smart transportation, smart city applications, and / or smart industrial applications, traditional video encoding may require compressing large amounts of video from cameras and transmitting them over the network for both machine and human use. Subsequently, at the machine site, algorithms for feature extraction can be applied, typically using convolutional neural networks or deep learning techniques, including object detection, event action recognition, pose estimation, etc. Summary of the invention
[0011] The present disclosure relates to systems and methods for encoding and decoding image data for machine use using an autoencoder for encoding and decoding feature data.
[0012] In one embodiment, a system for transmitting compressed image data in a machine-oriented video coding application is provided. The system includes a video coding for machine (VCM) encoder, which includes an autoencoder encoder portion and an autoencoder bottleneck portion. The VCM encoder receives image information and generates an encoded bit stream including feature data of the image information. A VCM decoder is also provided. The VCM decoder includes: an autoencoder bottleneck portion, which is the same as used in an encoder coupled to an autoencoder decoder portion. The VCM decoder receives the encoded bit stream at the bottleneck portion and reconstructs the image information based on the feature data.
[0013] The autoencoder encoder part is typically a neural network comprising a plurality of consecutive encoder layers and a bottleneck part, each consecutive encoder layer having fewer neurons than the previous layer, and the bottleneck part having fewer neurons than the final encoder layer. Similarly, the autoencoder decoder part is a neural network comprising a plurality of consecutive encoder layers and a bottleneck part, each consecutive decoder layer having a higher number of neurons than the previous layer, and the bottleneck part having fewer neurons than the first decoder layer. In some embodiments, the number of encoder layers in the encoder part is the same as the number of layers in the decoder part.
[0014] Furthermore, the autoencoder encoder portion may communicate with the autoencoder decoder portion such that updates to the autoencoder may be communicated between the VCM encoder and the VCM decoder.
[0015] In some exemplary embodiments, the VCM encoder may further include: an autoencoder description encoder coupled to the autoencoder encoder portion, and a latent feature encoder coupled to the autoencoder encoder portion. A multiplexer may be provided, and the multiplexer may be coupled to the latent feature encoder and the autoencoder description encoder and provide an encoded bit stream.
[0016] In some embodiments, the VCM decoder further comprises a demultiplexer receiving the encoded bitstream, a latent feature decoder coupled to the demultiplexer, and an autoencoder description decoder coupled to the demultiplexer. Preferably, the autoencoder decoder is partially coupled to the autoencoder description decoder and the latent feature decoder.
[0017] These and other aspects and features of non-limiting embodiments of the present invention will become apparent to those skilled in the art from a reading of the following description of specific non-limiting embodiments of the present invention taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] For the purpose of illustrating the invention, the accompanying drawings show aspects of one or more embodiments of the invention. It should be understood, however, that the invention is not limited to the precise arrangements and instrumentalities shown in the accompanying drawings, in which:
[0019] Figure 1 is a block diagram illustrating an exemplary embodiment of a video encoding system;
[0020] Figure 2 is a block diagram illustrating an exemplary embodiment of a machine-oriented video encoding system;
[0021] Figure 3 is a block diagram illustrating an exemplary embodiment of a system for adaptive neural network optimization for multi-task feature encoding;
[0022] Figure 4is a block diagram illustrating an exemplary embodiment of a machine process that uses the output of a decoder as input for task completion;
[0023] Figure 5 is a block diagram of an exemplary embodiment of a bitstream structure;
[0024] Figure 6 is a schematic diagram illustrating an exemplary embodiment of a pruned neural network;
[0025] Figure 7 is a schematic diagram illustrating an exemplary embodiment of a complete neural network;
[0026] Figure 8 is a schematic diagram showing two exemplary embodiments of NNG-NNH split points;
[0027] Fig. 9 is a simplified schematic diagram illustrating an exemplary autoencoder topology according to the present disclosure;
[0028] Fig.10 is a block diagram illustrating an exemplary embodiment of a system for machine-oriented video coding having an autoencoder for task feature encoding;
[0029] Fig.11 is a block diagram illustrating an exemplary embodiment of a machine process including an autoencoder and having the output of a decoder as input for task completion;
[0030] Fig.12 is a block diagram of an exemplary embodiment of a bitstream structure for use with an autoencoder system;
[0031] Fig.13 is a block diagram illustrating an exemplary embodiment of a machine learning module;
[0032] Fig.14 is a schematic diagram illustrating an exemplary embodiment of a neural network;
[0033] Fig.15 is a schematic diagram illustrating an exemplary embodiment of a node of a neural network;
[0034] Fig.16 is a block diagram illustrating an exemplary embodiment of a video decoder;
[0035] Fig.17 is a block diagram illustrating an exemplary embodiment of a video encoder; and
[0036] Fig.18 is a block diagram of a computing system that can be used to implement any one or more of the methods disclosed herein, or any one or more portions thereof.
[0037] The drawings are not necessarily drawn to scale and may be illustrated by phantom lines, diagrammatic representations, and partial views. In certain instances, details that are not necessary for understanding the embodiments or that render other details difficult to perceive may have been omitted. DETAILED DESCRIPTION
[0038] In many applications, such as surveillance systems with multiple cameras, smart transportation, smart city applications, and / or smart industrial applications, traditional video encoding may require large amounts of video from cameras to be compressed and transmitted over the network to machines and humans. Subsequently, at the machine site, algorithms for feature extraction can typically be applied using convolutional neural networks or deep learning techniques, including object detection, event action recognition, pose estimation, etc.
[0039] Figure 1 An exemplary embodiment of a standard-compliant (e.g., HEVC, AV1, VVC, etc.) encoding / decoding system is shown, the system including a channel for machine application. Unfortunately, conventional approaches may require a large amount of video transmission from multiple cameras, which may require a significant amount of time for efficient and rapid real-time analysis and decision making. In certain embodiments, a machine-oriented video coding ("VCM") approach may address this problem by encoding the video and extracting some features at a transmitter site, and then transmitting the resulting encoded bitstream to a VCM decoder. At the decoder site, the video can be decoded for human vision, and the features can be decoded for a machine. As used herein, the term VCM refers generally to video encoding and decoding for machine use, and is not limited to a specific proposed protocol.
[0040] "Features" used in the present disclosure are specific structures and / or content attributes of data. Examples of features may include SIFT, audio features, color history, motion history, speech level, loudness level, etc. Features may be timestamped. Each feature may be associated with a single frame in a set of frames. Features may include high-level content features, such as timestamps, labels of people and objects in the video, coordinates of objects and / or regions of interest, frame masks based on region quantization, and / or any other features that may occur to those skilled in the art after reading the entire content of the present disclosure. As another non-limiting example, features may include features describing the spatial and / or temporal characteristics of a frame or a set of frames. Examples of features describing spatial and / or temporal characteristics may include motion, texture, color, brightness, edge count, blur, blockiness, etc. When in offline mode, all machine models as described in further detail below may be stored at the encoder and / or in the encoder's memory and / or in a memory accessible to the encoder. Examples of such models may include, but are not limited to, convolutional neural networks, key point extractors, edge detectors, saliency map constructors, etc., in whole or in part. When in online mode, one or more models may be transmitted to the feature extractor 220 by a remote machine in real time or at some point in time prior to extraction.
[0041] At the decoder site, it is understood that the video can be decoded for human vision and the features can be decoded for machine use. Systems that provide video for both human vision and machine use are sometimes referred to as hybrid systems. The systems and methods disclosed herein are intended to be applicable to both machine-based systems and hybrid systems.
[0042] Figure 1 1 is a high-level block diagram of a system for encoding and decoding video in a hybrid system that includes both human viewer use of video content and machine use. A video encoder 105 receives a source video that provides a compressed bitstream for transmission over a channel to a video decoder 110. The video encoder can encode video for human use as well as encode video for machine use. The video decoder 110 provides complementary processing of the compressed bitstream to extract video for human vision 115 and task analysis and feature extraction 120 for machine use. Feature extraction can be classified as any computer vision task, such as edge detection, line detection, object detection, or more recent techniques, such as convolutional neural networks, where the output of feature extraction can be spatially mapped back to the pixel space of the input video. Video encoding can include any standard video encoder and / or encoding technique, such as Advanced Video Codec (AVC), Versatile Video Coding (VVC), or High Efficiency Video Coding (HEVC).
[0043] Reference now Figure 2, an exemplary embodiment of an encoder for machine-oriented video coding (VCM) is shown. The VCM encoder 202 can be implemented using any circuit, including but not limited to digital circuits and / or analog circuits; the VCM encoder 202 can be configured using hardware configuration, software configuration, firmware configuration and / or any combination thereof. The VCM encoder 202 can be implemented as a computing device and / or a component of a computing device, and the computing device can include but is not limited to any computing device described below. In one embodiment, the VCM encoder 202 can be configured to: receive an input video 204 and generate an output bitstream 208. The reception of the input video 204 can be completed in any manner as described below. The bitstream can include but is not limited to any bitstream described below.
[0044] The VCM encoder 202 may include, but is not limited to, a preprocessor 205, a video encoder 210, a feature extractor 215, an optimizer 220, a feature encoder 225, and / or a multiplexer 230. The preprocessor 205 may receive an input video 204 stream and parse out video, audio, and metadata substreams of the stream. The preprocessor 205 may include and / or communicate with a decoder, as described in further detail below; in other words, the preprocessor 205 may have the ability to decode the input stream. In a non-limiting example, this may allow the input video 204 to be decoded, which may facilitate downstream pixel domain analysis.
[0045] Further references Figure 2 , VCM encoder 202 can operate in hybrid mode and / or video mode; when in hybrid mode, VCM encoder 202 can be configured to encode visual signals intended for human users and to encode feature signals intended for machine users; machine users can include, but are not limited to, any device and / or component, including, but not limited to, a computing device as described in further detail below. For example, when in hybrid mode, the input signal can be passed through preprocessor 205.
[0046] Still reference Figure 2, the video encoder 210 may include, but is not limited to, any video encoder 210, as further described below. When the VCM encoder 202 is in hybrid mode, the VCM encoder 202 may send an unmodified input video 204 to the video encoder 210, and send the same input video 204 and / or a copy of the input video 204 that has been modified in some way to the feature extractor 215. Modifications to the input video 204 may include any scaling, transformation, or other modifications that may occur to a person skilled in the art after reading the entire disclosure. For example, but not limited to, the input video 204 may be sized to a smaller resolution, a certain number of pictures in a sequence of pictures in the input video 204 may be discarded, thereby reducing the frame rate of the input video 204, and color information may be modified, for example, but not limited to, by converting RGB video to grayscale video, etc.
[0047] Still reference Figure 2 , the video encoder 210 and the feature extractor 215 are connected and can exchange useful information in both directions. For example, but not limited to, the video encoder 210 can transmit motion estimation information to the feature extractor 220, and vice versa. The video encoder 210 can provide a quantization map and / or data describing a quantization map to the feature extractor 215 based on a region of interest (ROI) identifiable by the video encoder 210 and / or the feature extractor 215, and vice versa. The video encoder 210 can provide data describing one or more partition decisions to the feature extractor 215 based on features present and / or identified in the input video 204, the input signal and / or any frame and / or subframe thereof; the feature extractor 215 can provide data describing one or more partition decisions to the video encoder 210 based on features present and / or identified in the input video 204, the input signal and / or any frame and / or subframe thereof. The video encoder 210 and the feature extractor 215 can share temporal information, and / or transmit temporal information to each other for optimal group of pictures (GOP) decision. Each of these techniques and / or processes may be performed without limitation, as described in further detail below.
[0048] Continue to refer Figure 2, the feature extractor 215 can operate in an offline mode or an online mode. The feature extractor 215 can identify and / or otherwise act on and / or manipulate features. "Features" used in the present disclosure are specific structures and / or content attributes of data. Examples of features may include SIFT, audio features, color history, motion history, speech level, loudness level, etc. Features may be time stamped. Each feature may be associated with a single frame in a set of frames. Features may include high-level content features such as timestamps, labels of people and objects in the video, coordinates of objects and / or regions of interest, frame masks based on region quantization, and / or any other features that may occur to a person skilled in the art after reading the entire contents of this disclosure. As another non-limiting example, features may include features that describe the spatial and / or temporal characteristics of a frame or a group of frames. Examples of features that describe spatial and / or temporal characteristics may include motion, texture, color, brightness, edge count, blur, blockiness, etc. When in offline mode, all machine models as described in further detail below may be stored at the encoder and / or in the encoder's memory and / or in a memory accessible to the encoder. Examples of such models may include, but are not limited to, whole or partial convolutional neural networks, keypoint extractors, edge detectors, saliency map constructors, etc. When in online mode, one or more models may be transmitted to the feature extractor 220 by a remote machine in real time or at some point in time prior to extraction.
[0049] Still reference Figure 2 , the feature encoder 225 is configured to encode a feature signal, such as but not limited to the feature signal generated by the feature extractor 215. In an embodiment, after extracting the features, the feature extractor 215 may pass the extracted features to the feature encoder 225. The feature encoder 225 may use entropy coding and / or similar techniques, such as but not limited to the techniques described below, to produce a feature stream, which may be passed to the multiplexer 230. The video encoder 210 and / or the feature encoder 225 may be connected through the optimizer 220; the optimizer 220 may exchange useful information between those video encoders 210 and the feature encoder 225. For example, but not limited to, information related to the codeword construction and / or length used for entropy coding may be exchanged and reused through the optimizer 220 for optimal compression.
[0050] In one embodiment, and with continued reference to Figure 2, the video encoder 210 can generate a video stream; the video stream can be passed to the multiplexer 230. The multiplexer 230 can multiplex the video stream with the feature stream generated by the feature encoder 225; alternatively or additionally, the video and feature bitstreams can be transmitted to different devices via different channels, different networks, and / or at different times or at different time intervals (time multiplexing). Each of the video stream and the feature stream can be implemented in any manner suitable for implementing any bitstream described in the present disclosure. In one embodiment, the multiplexed video stream and feature stream can generate a mixed bitstream, which can be transmitted as described in further detail below.
[0051] Still reference Figure 2 , where the VCM encoder 202 is in video mode, the VCM encoder 202 may use the video encoder 210 for both video encoding and feature encoding. The feature extractor 215 may transmit the features to the video encoder 210; the video encoder 210 may encode the features into a video stream, which may be decoded by a corresponding video decoder 250. It should be noted that the VCM encoder 202 may use a single video encoder 210 for both video encoding and feature encoding, in which case it may use different parameter sets for video and features; alternatively, the VCM encoder 202 may be two independent video encoders 210 that may operate in parallel.
[0052] Still reference Figure 2 , system 200 may include and / or communicate with a VCM decoder 240. VCM decoder 240 and / or its elements may be implemented using any circuit and / or configuration type suitable for the configuration of VCM encoder 202, as described above. VCM decoder 240 may include, but is not limited to, a demultiplexer 245. If the bitstream is multiplexed as described above, demultiplexer 245 may operate to demultiplex the bitstream. For example, but not limited to, demultiplexer 245 may separate a multiplexed bitstream containing one or more video bitstreams and one or more feature bitstreams into separate video and feature bitstreams.
[0053] Continue to refer Figure 2 , VCM decoder 240 may include video decoder 250. Video decoder 250 may be implemented in any manner suitable for a decoder, as described in further detail below. In one embodiment, but not limited to this, video decoder 250 may generate an output video that can be viewed by a person or other creature and / or device with visual sensory capabilities.
[0054] Still reference Figure 2, the VCM decoder 240 may include a feature decoder 255. In one embodiment, but not limited to this, the feature decoder 255 may be configured to provide one or more decoded data to the machine. The machine may include, but is not limited to, any computing device as described below, including but not limited to any microcontroller, processor, embedded system, system on chip, network node, etc. The machine may operate a machine model, store a machine model, train a machine model, receive input from a machine model, generate output for a machine model, and / or otherwise interact with a machine model, as described in further detail below. The machine may be included in the Internet of Things (IoT), which is defined as a network of objects having processing components and communication components, some of which may not be traditional computing devices, such as desktop computers, laptop computers, and / or mobile devices. Objects in the Internet of Things may include, but are not limited to, any device having an embedded microprocessor and / or microcontroller and one or more components for interfacing with a local area network (LAN) and / or a wide area network (WAN); one or more components may include, but are not limited to, a wireless transceiver, such as a wireless transceiver that communicates in the 2.4-2.485 GHz range, such as a BLUETOOTH transceiver that complies with the protocol promulgated by Bluetooth SIG, Inc., Kirkland, Washington, and / or a network communication component that operates according to the MODBUS protocol promulgated by Schneider Electric SE, Rueil-Malmaison, France, and / or the ZIGBEE specification of the IEEE 802.15.4 standard promulgated by the Institute of Electronics and Communications (IEEE). After reading the entire contents of this disclosure, those skilled in the art will be aware of various alternative or additional communication protocols and devices supporting such protocols that can be used in accordance with this disclosure, each of which is considered to be within the scope of this disclosure.
[0055] Continue to refer Figure 2Each of the VCM encoder 202 and / or the VCM decoder 240 may be designed and / or configured to perform any method, method step, or sequence of method steps in any embodiment described in the present disclosure in any order and with any degree of repetition. For example, each of the VCM encoder 202 and / or the VCM decoder 240 may be configured to repeatedly perform a single step or sequence until a desired or commanded result is achieved; repetition of a step or sequence of steps may be performed iteratively and / or recursively, using the output of a previous repetition as input to a subsequent repetition, aggregating the input and / or output of a repetition to produce an aggregated result, reducing or decrementing one or more variables, such as global variables, and / or dividing a larger processing task into a collection of smaller processing tasks that are addressed iteratively. Each of the VCM encoder 202 and / or the VCM decoder 240 may perform any step or sequence of steps described in the present disclosure in parallel, such as performing a step two or more times simultaneously and / or substantially simultaneously using two or more parallel threads, processor cores, etc.; task division between parallel threads and / or processes may be performed according to any protocol suitable for task division between iterations. After reading the entire disclosure, those skilled in the art will recognize various ways in which steps, sequences of steps, processing tasks and / or data may be subdivided, shared, or processed using iteration, recursion and / or parallel processing.
[0056] The present systems and methods are based on a machine learning architecture that supports multiple tasks for end users. The most common machine learning architecture used today is a neural network. One of the disadvantages of simple single-task neural networks is the time complexity and computational cost of training. To achieve high performance, neural networks must typically be trained using very large data sets with hundreds of thousands or even millions of samples, such as images and videos. Training a separate network every time a new use case emerges can be highly redundant and resource-wasting. Therefore, methods have been developed to reuse already trained portions of a neural network for multiple tasks. By training a portion of the network to support multiple tasks, users can save storage space, computing power, and reduce energy consumption.
[0057] Figure 3 is a block diagram illustrating an exemplary embodiment of a system for adaptive neural network optimization for multi-task feature coding; the system can be used in a method for adaptive neural network optimization, which can be used to implement multi-task feature coding in a system such as a standard for video coding for machines (VCM), or any other system that implements a neural network for image and video processing by a machine, such as the Internet of Things (IoT).
[0058] Typically, an input image or video passes through an encoder 300 that extracts relevant features. The encoder 300 includes a preprocessor 305 coupled to a neural network backbone (NNB) 310, the output of which is a feature set that can be further compressed using a feature or video encoder 315. The NNB 310 can also receive input and / or training data from a machine model 360. The output of the video encoder 315 is a feature substream. In addition to the features, the description of the NNB 310 can be compressed, and the description substream can be combined with the feature substream from the neural network description encoder 325 in a multiplexer 320 to form a bitstream 330.
[0059] The bitstream 330 is sent through a channel to a decoder 335, which decodes / decompresses the two substreams and produces an NNB description as well as a feature set. The bitstream 330 is received by a demultiplexer 340, which parses the bitstream into an NNB description substream applied to a neural network description decoder 345 and a feature set substream applied to a feature / video decoder 350. The end user (machine) 355 receives the feature set and passes it as input to a neural network head (NNH), which produces a decision at output related to one or more tasks performed by the machine. This is possible, for example, where a complete neural network (NN) is pre-trained and the parameters of the NNB are known and have not changed.
[0060] Figure 4 420. It is a block diagram showing an exemplary embodiment of a machine process with the output of the decoder used as input for task completion 425. In the case where NNB 415 has not been trained by the machine and the parameters of NNB 415 are unknown, the machine receives the NNB description from the decoder 435 and reconstructs the complete NN backbone 415, and optionally the machine fine-tunes the NNH to be consistent with the NNB or to support additional tasks not included in the original NNB training. In this article, NN retraining (NNH fine-tuning) is an optional step 420.
[0061] If the machine 455 knows the complete NN, such as by knowing all NNB parameters, then the NNB description does not need to be transmitted to the decoder. In this case, the bitstream may contain only the feature substream 410.
[0062] To facilitate flexibility on the encoder side, in the event that the task needs to change or the parameters of the current task need to change, the machine can send the NNB description to the encoder side that is consistent with the relevant machine model. Figure 1 Described in.
[0063] Along with the NNB description, the decoder can send a time interval for which a given NNB description is valid. After the time interval expires, the NNB description reverts to the previous description or the default description.
[0064] On the encoder side, feature / video encoders may be implemented using existing video and feature standards such as Versatile Video Coding (VVC), High Efficiency Video Coding (HEVC), AOMedia Video 1 (AV1), Compression Descriptor for Video Analytics (CDVA), etc.
[0065] NN description encoders can be implemented using simple techniques, such as variable length coding (VLC), or more advanced standards, such as Neural Network Compression and Representation (NNR).
[0066] Figure 5 is a block diagram of an exemplary embodiment of a bitstream structure;
[0067] The stream level header 505 contains a high level syntax describing the presence of a substream and contains parameters of such substream, such as length, duration, format, etc. The parser in the decoder uses this information to extract the substream.
[0068] The feature substream 510 contains a feature stream header 515 that describes the feature stream payload 520 in terms of length, format, and other relevant parameters. The feature stream header 515 is used by the feature decoder to extract and decode the feature stream payload 520.
[0069] The NN description substream 525 contains a NN description header 530, which contains parameters describing the length, format and type of the NN description payload 535. Alternatively, instead of a complete description of all NN parameters, the encoder can signal the index of the NN used in a lookup table, or a predetermined and agreed list. The list can be maintained by a central registration center, which updates the list and signals the updates to the end user. The NN description header is used by the feature decoder to extract and decode the NN description payload.
[0070] Depending on the level of abstraction and cross-task generalization, the split point between NNH and NNB can be adaptively chosen. In addition, when NNs are trained for multiple tasks and use cases, parts of the NN may be mostly or completely task-related, while other parts may be mostly or completely redundant for the specific task. In the following paragraphs, two adaptive optimization techniques are introduced: vertical optimization (task-specific ablation) and horizontal optimization (computation offloading).
[0071] Figure 6 is a schematic diagram illustrating an exemplary embodiment of a pruned neural network; such an embodiment may enable vertical optimization and / or task-specific ablation.
[0072] Ablation or pruning techniques are well established in the machine learning literature with a particular focus on neural networks (references: Newell, Allen (1975). D. Raj Reddy (ed.). A Tutorial on Speech Understanding Systems. In Speech Recognition: Invited Papers Presented at the 1974 IEEE Symposium. New York: Academic). Using this approach, parts of a neural network can be removed when training for a specific task. By pruning neurons or connections (synapses), a single NN can be optimized to support multiple tasks. Figure 6 An example of pruning is given in Figure 6 A complete neural network 600, an exemplary synapse pruned neural network 605, and an exemplary neuron pruned neural network 610 are shown.
[0073] Using ideas from information theory, a class of practical and near-optimal schemes for adapting the size of neural networks has been developed. By removing less important weights from the network, several improvements can be expected: better generalization, fewer training samples required, and increased speed of learning and / or classification. The approach can use second-order derivative information to trade off between network complexity and training set error.
[0074] For example, one part of a neural network can be trained for object detection, while another part is trained for object tracking or segmentation. At a lower level, one part of the network can be trained to detect one class of objects (e.g., people), while another part can be trained to detect another class of objects (e.g., cars).
[0075] Figure 7 is a schematic diagram showing an exemplary embodiment of a complete neural network. The computational complexity of the encoder depends linearly on the size of the NNB. Although vertical optimization can reduce the amount of computation in the NNB in some cases, the split point between the NNB and the NNH is the main tool for computational offloading. The split point needs to be selected based on the criteria of energy usage, computational power, and bandwidth between the encoder and decoder. Examples of neural networks are as follows Figure 7 shown. Figure 8 An example of the splitting options is described in
[0076] Figure 8is a schematic diagram showing two exemplary embodiments of NNB-NNH split points. In option 1, the split point is selected at a later stage in the NN, resulting in a relatively smaller size of the feature set. However, the benefit of having a smaller feature set and thus a smaller feature substream comes at the expense of higher computational complexity on the encoder side and a larger NN description.
[0077] In option 2, the split point is chosen at a lower level of the NN, resulting in a very small NNB and thus requiring minimal computational cost on the encoder side, as well as a minimal NN descriptor substream. However, this comes at the expense of a larger feature substream.
[0078] The trade-offs of different options need to be carefully considered based on cross-task similarities and computational requirements / energy demands, and the optimal implementation will vary for different use cases.
[0079] Alternatively, the encoder may automatically decide to implement vertical optimization and / or horizontal optimization based on available data collected by the encoder from the environment or received from the decoder.
[0080] The systems and methods described in the present disclosure may be implemented and / or interact with any systems, system components, methods, and / or method steps described in PCT application PCT / US22 / 53579, entitled “INTELLIGENT MULTI-STREAM VIDEO CODING FOR VIDEO SURVEILLANCE,” filed on December 22, 2022, the entire contents of which are incorporated herein by reference.
[0081] Autoencoder
[0082] The present disclosure further describes a framework for video coding for machines (VCM) that includes an autoencoder (AE) as a feature encoding and decoding element. The autoencoder described herein can be used as a special case of the general neural network described above, with some modifications.
[0083] refer to Fig. 9 , AE can be generally described as three parts. Preferably, the first part of AE - AE encoder 900 - is implemented on the encoder side of VCM (this can be implemented on an edge node or similar remote site / device). Preferably, the second part of AE - AE decoder 910 - is implemented as part of the VCM decoder (this can be implemented on a central server or similar centralized site / device). In an exemplary embodiment, the shared part of AE - bottleneck 905 - exists at both sites / devices. In one embodiment disclosed herein, all or substantially all features and AE updates are transmitted between the encoder and decoder using a bitstream.
[0084] An autoencoder (AE) is an unsupervised neural network that learns how to efficiently compress and encode high-dimensional input data, and then learns how to reconstruct / decode the data from the reduced encoded representation back to a representation as close to the original input as possible. In some cases, AE can reduce the dimensionality of the data by learning how to ignore the noise in the data. Noise is defined as redundant data in the sense of a specific task. For example, a network that detects faces can learn to retain the part of the input signal that contains the face and discard the rest of the input information. Therefore, AE is suitable for a variety of applications in visual signal processing and has therefore been used for some time, as described in Transforming Auto-encoders, Hinton, GE, Krizhevsky, A., & Wang, SD (2011. June) International conference on Artificial Neural Networks (pp. 44-51). Springer, Berlin, Heidelberg.
[0085] Reference again Fig. 9 , an AE network typically has three main high-level components: an AE encoder 900, an AE bottleneck 905, and an AE decoder 910. Using backpropagation, the unsupervised algorithm continuously trains the AE by setting the target output value as close to the input as possible given an allowed loss threshold. This forces the typically small AE bottleneck to use dimensionality reduction to reduce or eliminate noise.
[0086] A typical implementation of AE has the same number of layers (n) in the encoder 900 and the decoder 910 , and the number of neurons in these layers is usually significantly higher than the number of neurons in the bottleneck layer 905 .
[0087] Layers AEe1 915 to AEen 920 belong to the AE encoder 900 and typically have a decreasing number of neurons in each subsequent layer. Layer AEb 925 is the only layer in the AE bottleneck layer 905, and layer AEb 925 typically has fewer neurons than layer AEen 920. Layers AEd1 930 to AEdn935 belong to the AE decoder 910 and typically have an increasing number of neurons in each subsequent layer. In order to process visual information, the preprocessed image is passed to the input layer AEe1915 and is then transformed into a lower dimensional representation in each layer. The coding transformation results in a bottleneck, the so-called "latent" representation. This latent representation is acquired by the input layer AEd1 930 of the decoder and is then expanded and transformed into a close reconstruction of the original image, which is the output of the AEdn layer 935.
[0088] The problem that AE usually solves is to learn the functions A:Rn->Rp (encoder) and B:Rp->Rn (decoder) so that
[0089]
[0090] Where E is the expected value of the distribution of x, and Δ is the reconstruction loss function, which measures the distance between the output of the decoder and the input of the AE. Typically, the l2 norm is used for the loss function.
[0091] In the VCM framework, the input to the AE is usually an image or video from a source such as a camera, and the output is a reconstruction of that image, which is smaller than the input image in terms of bandwidth. When passed through the task neural network, this reconstructed image should produce the same or similar results as the original image.
[0092] Preferably, the loss of the task neural network is added to the loss function of the AE during the training phase to achieve the optimal parameters of the AE. The acceptable level of overall loss will determine the amount of compression that the AE can achieve. This trade-off framework allows the VCM encoder to increase or decrease the amount of compression based on conditions such as task type, task priority, bandwidth conditions of the communication network, etc.
[0093] During training, the connections and neurons of the AE can be further optimized and sparse, for example through the pruning process described herein, such as combining Figure 6 Compared to the full network, the pruned network occupies less space and in some cases can even achieve higher compression.
[0094] VCM AE framework
[0095] Given an AE pre-trained for a specific task, the AE encoder 900 and the AE decoder 910 can be split and implemented in the VCM encoder and VCM decoder. Fig.10 One such implementation is depicted in .
[0096] The VCM encoder receives an input image or video, such as from a camera, and passes the input image or video through a pre-processor unit 1005, which converts the given input into an appropriate format. Examples of pre-processing are: color conversion from camera output to RGB color space for a typical AE encoder input; image or frame downscaling from a high-definition resolution to a lower resolution typically accepted by the AE encoder; video frame rate reduction by the AE encoder for more efficient processing; etc.
[0097] The pre-processed image / video is then passed through an AE encoder 1010 having a bottleneck AEb layer 925 ( Fig. 9) as the last layer. The output of the AEb layer 925 is then further encoded using a latent feature encoder (LFE) 1015 to further compress the feature stream, using statistical coding such as entropy coding, variable length coding, or some other type of general coding structure that allows a trade-off between compression level and information loss. If any modifications to the AE encoder are required to support a given input signal, these modifications are encoded using an AE description encoder 1020, which encodes the modifications to the AE encoder 1010 using similar statistical or other general coding techniques. Examples of such modifications are changes to connection weights, changes to the activation functions of neurons, and pruning operations on network elements.
[0098] The VCM encoder may determine in adaptive mode to further modify the AE encoder for optimal operation based on conditions such as task type, task priority, bandwidth conditions of the communication network, etc.
[0099] The encoded AE description is then multiplexed 1025 with the encoded latent features into the bitstream sent to the decoder.
[0100] In addition to the autonomous modification of the AE encoder, the VCM encoder 1000 may receive a signal from the decoder or some other source with an updated machine model. The modification is then implemented in the AE encoder 1010, and no further information needs to be sent back to the decoder. The reason for this update may be a change in the task performed on the decoder side. For example, the decoder 1040 may be changed from detecting faces to detecting some other objects, or the type of task may change, such as from object detection to image segmentation or object tracking, etc.
[0101] Decoder
[0102] The VCM decoder 1040 receives the bitstream 1045 and passes it through the demultiplexer 1050, which uses a bitstream parser to extract and split the potential feature stream and the AE description stream. These streams are then passed through the appropriate decoders. The potential feature decoder 1055 takes the potential feature stream and passes it to layer AEd1 930 of the AE decoder 1060, which is then passed to other decoder layers, ultimately producing an image or video reconstruction at the output. The AE description decoder 1065 applies the inverse operation of the AE description encoder and produces a description of the modifications made by the AE encoder.
[0103] Fig.11 The component layout of VCM decoder 1040 is further shown in FIG.
[0104] If the AE encoder is modified, and the AE description is present in the bitstream, the decoder applies the described modifications to the instance of the original AE encoder stored at the decoder site. When the required modifications are applied, the entire AE may need to be retrained 1170 in order to obtain the optimal parameters for the AE decoder. After retraining 1170, the latent features are passed through the updated AE decoder 1175 to obtain the desired output representation of the original input image.
[0105] This representation is then passed by the machine to the task neural network 1180 that performs the desired task. Preferably, the performance of the task NN 1180 with AE image reconstruction should be the same or similar to the performance of the task NN with the original image as input to a degree of a predetermined threshold.
[0106] In order to achieve the desired performance threshold, the AE and the task NN can be jointly trained in a manner similar to the joint training method described in the prior application PCT / US22 / 40722 entitled "SYSTEMS AND METHODS FOR JOINTOPTIMIZATION TRAINING" (the entire contents of which are incorporated herein by reference), where the rate-distortion optimizer is replaced by the conventional loss function of the task NN.
[0107] BitTorrent
[0108] like Fig.12 As shown, the bitstream generated by the VCM encoder 1000 contains a feature substream and an AE description substream. Any of the substreams can be empty. The existence of the substream, as well as the length and type of the current substream are described by parameters in the stream level header 1205.
[0109] The feature substream contains a feature stream header 1210 describing the elements of the feature stream, and a feature stream payload 1215 containing a binary representation of the underlying features. The feature stream header 1210 contains parameters describing the type and length of the underlying feature elements.
[0110] The AE description substream contains an AE description stream header 1220 describing the elements of the AE description stream, and an AE description stream payload 1225 containing a binary representation of modifications to the AE encoder. The AE description stream header contains parameters describing the type and length of the AE description elements.
[0111] If the decoder does not have a complete description of a pre-trained AE encoder, the encoder can signal the description to the decoder using standard index table entries and network descriptions, such as Open Neural Network Exchange (ONNX). This saves bandwidth and does not require the full network description to be sent as part of the bitstream.
[0112] Reference now Fig.13 , showing an exemplary embodiment of a machine learning module 1300 that can perform one or more machine learning processes described in the present disclosure. The machine learning module can use machine learning processes to perform the determination, classification and / or analysis steps, methods, processes, etc. described in the present disclosure. A "machine learning process" as used in the present disclosure is a process that automatically uses training data 1304 to generate an algorithm that will be executed by a computing device / module to produce an output 908 given data provided as input 1312; this is in contrast to non-machine learning software programs in which the commands to be executed are determined in advance by the user and written in a programming language.
[0113] Still reference Fig.13 As used herein, "training data" is data that contains correlations that a machine learning process can use to model the relationship between two or more categories of data elements. For example, but not limited to, the training data 1304 may include multiple data entries, each representing a set of data elements that are recorded, received, and / or generated together; the data elements may be associated by shared presence in a given data entry, by proximity in a given data entry, and the like. The multiple data entries in the training data 1304 may indicate one or more trends of correlations between categories of data elements; for example, but not limited to, a higher value of a first data element belonging to a first category of data elements may tend to be associated with a higher value of a second data element belonging to a second category of data elements, thereby indicating a possible proportion or other mathematical relationship linking the values belonging to the two categories. According to various correlations, multiple categories of data elements may be related in the training data 1304; the correlation may indicate a causal and / or predictive connection between the data elements of each category, which may be modeled by a machine learning process as a relationship such as a mathematical relationship, as described in further detail below. The training data 1304 may be formatted and / or organized by each category of data elements, for example, by associating the data elements with one or more descriptors corresponding to the data elements of each category. As a non-limiting example, training data 1304 may include data entered by a person or process in a standardized form such that an entry for a given data element in a given field in a table may be mapped to a descriptor of one or more categories. Elements in training data 1304 may be linked to descriptors of categories by tags, tokens, or other data elements; for example, but not limited to, training data 1304 may be provided in a fixed-length format, a format that links data locations to categories such as a comma-separated value (CSV) format, and / or a self-describing format such as an extensible markup language (XML), JavaScript object representation (JSON), and the like, thereby enabling a process or device to detect data categories.
[0114] Alternatively or additionally, continue to refer to Fig.13 , the training data 1304 may include one or more unclassified elements; that is, the training data 1304 may not be formatted or contain descriptors for some data elements. The machine learning algorithm and / or other process may classify the training data 1304 according to one or more categories using, for example, natural language processing algorithms, tokenization, detecting associated values in the raw data, etc.; the categories may be generated using correlation and / or other processing algorithms. As a non-limiting example, in a text corpus, phrases consisting of a number "n" of compound words - such as nouns modified by other nouns - may be identified based on the statistically significant prevalence of n-grams containing such words in a particular order; such n-grams may be classified as elements of a language, such as "words" that are tracked similarly to single words, thereby generating new categories as a result of statistical analysis. Similarly, in a data entry that includes some text data, a person's name may be identified by a reference list, dictionary, or other summary of terms, thereby allowing for special classification by a machine learning algorithm, and / or automatically associating the data in the data entry with a descriptor or into a given format. The ability to automatically classify data entries can make the same training data 904 suitable for two or more different machine learning algorithms, as described in further detail below. The training data 1304 used by the machine learning module 1300 can associate any input data described in the present disclosure with any output data described in the present disclosure. As a non-limiting illustrative example
[0115] Further references Fig.13, one or more supervised and / or unsupervised machine learning processes and / or models may be used to filter, classify and / or select training data, as described in further detail below; such models may include, but are not limited to, training data classifiers 1316. Training data classifiers 1316 may include a "classifier," which, as used in the present disclosure, is a machine learning model as defined below, such as a mathematical model, a neural network, or a program generated by a machine learning algorithm referred to as a "classification algorithm," which, as described in further detail below, classifies inputs into categories or bins of data, thereby outputting categories or bins of data and / or labels associated with the categories or bins of data. A classifier may be configured to output at least one piece of data that labels or otherwise identifies data sets that are clustered together, are found to be close under a distance metric as described below, etc. The machine learning module 1300 may generate a classifier using a classification algorithm, which is defined as a process by which a computing device and / or any module and / or component operating thereon derives a classifier based on the training data 1304. Classification can be performed using, but not limited to, linear classifiers such as, but not limited to, logistic regression and / or naive Bayes classifiers, nearest neighbor classifiers such as k-nearest neighbor classifiers, support vector machines, least squares support vector machines, fisher linear discriminant, quadratic classifiers, decision trees, boosted trees, random forest classifiers, learning vector quantization, and / or neural network based classifiers.
[0116] Still reference Fig.13 , the machine learning module 1300 can be configured to perform a lazy learning process 1320 and / or protocol, which can be alternatively referred to as a "lazy loading" or "call when needed" process and / or protocol, which can be a process by which, upon receiving an input to be converted into an output, the algorithm to be used to generate the output on demand is derived by combining the input and the training set to perform machine learning. For example, an initial set of simulations can be performed to cover initial heuristics and / or "first guesses" for outputs and / or relationships. As a non-limiting example, the initial heuristics can include rankings of associations between inputs and elements of the training data 1304. Heuristics can include selecting a certain number of the highest ranked associations and / or training data 1304 elements. Lazy learning can implement any suitable lazy learning algorithm, including but not limited to the K nearest neighbor algorithm, the lazy naive Bayes algorithm, etc.; after reading the entire contents of this disclosure, those skilled in the art will be aware of the various lazy learning algorithms that can be applied to generate the outputs described in this disclosure, including but not limited to lazy learning applications of machine learning algorithms, as described in further detail below.
[0117] Alternatively or additionally, and with continued reference to Fig.13, the machine learning process described in the present disclosure can be used to generate a machine learning model 1324. A "machine learning model" as used in the present disclosure is a mathematical and / or algorithmic representation of a relationship between an input and an output, generated using any machine learning process including but not limited to any process described above, and stored in a memory; once the input is created, it is submitted to the machine learning model 1324, and the machine learning model 1324 generates an output based on the derived relationship. For example, but not limited to, a linear regression model generated using a linear regression algorithm can use coefficients derived during the machine learning process to calculate a linear combination of input data to calculate output data. As another non-limiting example, the machine learning model 1324 can be generated by creating an artificial neural network, such as a convolutional neural network including a node input layer, one or more intermediate layers, and a node output layer. Connections between nodes can be created via a process of "training" the network, where elements from a training data set 1304 are applied to the input nodes, and then a suitable training algorithm (such as Levenberg-Marquardt, conjugate gradient, simulated annealing, or other algorithms) is used to adjust the connections and weights between nodes in adjacent layers of the neural network to produce desired values at the output nodes. Sometimes, this process is referred to as deep learning.
[0118] Still reference Fig.13 , the machine learning algorithm may include at least a supervised machine learning process 1328. As defined herein, the supervised machine learning process 1328 includes at least an algorithm that receives a training set that associates multiple inputs with multiple outputs, and seeks to find one or more mathematical relationships that associate the inputs with the outputs, wherein each of the one or more mathematical relationships is optimal according to a certain criterion specified for the algorithm using a certain scoring function. For example, the supervised learning algorithm may include inputs and outputs as described above in the present disclosure, and a scoring function that represents the desired form of the relationship between the inputs and the outputs to be detected; for example, the scoring function may seek to maximize the probability that a given input, and / or a combination of element inputs, is associated with a given output to minimize the probability that a given input is not associated with a given output. The scoring function may be expressed as a risk function that represents the "expected loss" of the algorithm that associates the inputs with the outputs, wherein the loss is calculated as an error function that represents the degree to which the prediction generated by the association is incorrect when compared to a given input-output pair provided in the training data 1304. After reading the entire contents of the present disclosure, those skilled in the art will at least be aware of the various possible variations of the supervised machine learning process 928 that can be used to determine the relationship between the inputs and the outputs. The supervised machine learning process may include a classification algorithm as defined above.
[0119] Further references Fig.13, the machine learning process may include at least an unsupervised machine learning process 1332. As used herein, an unsupervised machine learning process is a process of deriving inferences in a data set without regard to labels; thus, an unsupervised machine learning process is free to discover any structure, relationship, and / or correlation provided in the data. An unsupervised process may not require a response variable; an unsupervised process may be used to discover interesting patterns and / or inferences between variables, to determine the degree of correlation between two or more variables, etc.
[0120] Still reference Fig.13 , the machine learning module 1300 can be designed and configured to create a machine learning model 1324 using techniques for developing linear regression models. The linear regression model can include a conventional least squares regression, the purpose of which is to minimize the square of the difference between the predicted outcome and the actual outcome according to an appropriate norm for measuring such a difference (e.g., a vector space distance norm); the coefficients of the resulting linear equation can be modified to improve the minimization. The linear regression model can include a ridge regression method, in which the function to be minimized includes a least squares function plus a term that multiplies the square of each coefficient by a scalar to penalize large coefficients. The linear regression model can include a least absolute shrinkage and selection operator (LASSO) model, in which ridge regression is combined with multiplying the least squares term by a factor of 1 divided by twice the number of samples. The linear regression model can include a multi-task lasso model, in which the norm applied in the least squares term of the lasso model is a Frobenius norm equal to the square root of the sum of the squares of all terms. The linear regression model may include an elastic net model, a multi-task elastic net model, a minimum angle regression model, a LARS lasso model, an orthogonal matching pursuit model, a Bayesian regression model, a logistic regression model, a stochastic gradient descent model, a perceptron model, a passive aggressive algorithm, a robust regression model, a Huber regression model, or any other suitable model that a person skilled in the art may think of after reading the entire disclosure. In one embodiment, the linear regression model may be generalized to a polynomial regression model, thereby seeking a polynomial equation (e.g., a second-order, third-order, or higher-order equation) that provides the best predicted output / actual output fit; a person skilled in the art will understand after reading the entire disclosure that a method similar to the above method may be applied to minimize the error function.
[0121] Continue to refer Fig.13, the machine learning algorithm may include, but is not limited to, linear discriminant analysis. The machine learning algorithm may include quadratic discriminant analysis. The machine learning algorithm may include kernel ridge regression. The machine learning algorithm may include support vector machines, including, but not limited to, regression processes based on support vector classification. The machine learning algorithm may include: stochastic gradient descent algorithms, including classification and regression algorithms based on stochastic gradient descent. The machine learning algorithm may include a nearest neighbor algorithm. The machine learning algorithm may include various forms of latent space regularization, such as variational regularization. The machine learning algorithm may include a Gaussian process, such as a Gaussian process regression. The machine learning algorithm may include: a cross decomposition algorithm, including partial least squares and / or canonical correlation analysis. The machine learning algorithm may include a naive Bayes method. The machine learning algorithm may include a decision tree-based algorithm, such as a decision tree classification or regression algorithm. The machine learning algorithm may include an integrated method, such as a bagging meta-estimator, a random tree forest, AdaBoost, a gradient tree boosting and / or a voting classifier method. The machine learning algorithm may include: a neural network algorithm, including a convolutional neural network process.
[0122] Reference now Fig.14 , an exemplary embodiment of a neural network 1400 is shown. Neural network 1400, also known as an artificial neural network, is a network of "nodes" or data structures having one or more inputs, one or more outputs, and functions that determine the outputs based on the inputs. Such nodes can be organized in a network, such as, but not limited to, a convolutional neural network, including a node input layer, one or more intermediate layers, and a node output layer. Connections between nodes can be created via a process of "training" the network, where elements from a training data set are applied to the input nodes, and then a suitable training algorithm (such as Levenberg-Marquardt, conjugate gradient, simulated annealing, or other algorithms) is used to adjust the connections and weights between nodes in adjacent layers of the neural network to produce the desired value at the output node. Sometimes, this process is referred to as deep learning. In a "feedforward" network, connections can extend only from the input nodes toward the output nodes, or the output of one layer can be fed back to the input of the same or different layer in a "recursive network."
[0123] Reference now Fig.15 , shows an exemplary embodiment of a node of a neural network. The node may include but is not limited to multiple inputs x i, Multiple input x i The node can receive values from the inputs to the neural network containing the node and / or from other nodes. The node can use the value multiplied by the corresponding input x i The weight w ito perform a weighted sum of the inputs. Additionally or alternatively, a bias b may be added to the weighted sum of the inputs so that an offset is added to each unit in a neural network layer that is independent of the input to that layer. The weighted sum may then be input to the function In this function Can generate one or more outputs y. Applied to input x i The weight w i It can be indicated that an input is "excitatory", indicating that it has a strong effect on one or more outputs y, for example by having a corresponding weight with a large value, and / or that an input is "inhibitory", indicating that it has a weak effect on one or more outputs y, for example by having a corresponding weight with a small value. The weights w can be determined by training a neural network using training data i This may be performed using any suitable process as described above.
[0124] Still reference Fig.15 , a "convolutional neural network" as used in this disclosure is a neural network in which at least one hidden layer is a convolutional layer that convolves the input to that layer with a subset of the input called a "kernel", and one or more additional layers, such as pooling layers, fully connected layers, etc. CNNs may include, but are not limited to, deep neural network (DNN) extensions, where a DNN is defined as a neural network with two or more hidden layers.
[0125] Fig.16 16 is a system block diagram illustrating an example decoder 1600 capable of adaptive cropping. The decoder 1600 may include an entropy decoder processor 1604, an inverse quantization and inverse transform processor 1608, a deblocking filter 1612, a frame buffer 1616, a motion compensation processor 1620, and / or an intra prediction processor 1624.
[0126] In operation, still refer to Fig.16 , the bitstream 1628 may be received by the decoder 1600 and input to the entropy decoder processor 1604, which may entropy decode a portion of the bitstream into quantized coefficients. The quantized coefficients may be provided to the inverse quantization and inverse transform processor 1608, which may perform inverse quantization and inverse transform to create a residual signal, which may be added to the output of the motion compensation processor 1620 or the intra-frame prediction processor 1624, depending on the processing mode. The output of the motion compensation processor 1620 and the intra-frame prediction processor 1624 may include a block prediction based on a previously decoded block. The sum of the prediction and the residual may be processed by the deblocking filter 1612 and stored in the frame buffer 1616.
[0127] In one embodiment, still referring to Fig.16, the decoder 1600 may include the following circuitry, which is configured to implement any of the operations described above in any of the embodiments described above in any order and any degree of repetition. For example, the decoder 1600 may be configured to repeatedly perform a single step or sequence until the desired or commanded result is achieved; the repetition of a step or sequence of steps may be performed iteratively and / or recursively, using the output of a previous repetition as the input of a subsequent repetition, aggregating the input and / or output of a repetition to produce an aggregated result, reducing or decrementing one or more variables, such as global variables, and / or dividing a larger processing task into a collection of smaller processing tasks addressed iteratively. The decoder may perform any step or sequence of steps described in the present disclosure in parallel, such as using two or more parallel threads, processor cores, etc. to perform the step two or more times simultaneously and / or substantially simultaneously; the task division between parallel threads and / or processes may be performed according to any protocol suitable for task division between iterations. After reading the entire contents of the present disclosure, those skilled in the art will recognize the various ways in which steps, sequence of steps, processing tasks, and / or data can be subdivided, shared, or processed using iteration, recursion, and / or parallel processing.
[0128] Fig.17 is a system block diagram illustrating an example video encoder 1700 suitable for VCM encoding. The example video encoder 1700 may receive an input video 1704 that may be initially split or divided according to a processing scheme such as a tree-structured macroblock partitioning scheme (e.g., a quadtree plus a binary tree). An example of a tree-structured macroblock partitioning scheme may include partitioning a picture frame into larger block elements called coding tree units (CTUs). In some implementations, each CTU may be further partitioned one or more times into multiple sub-blocks called coding units (CUs). The final result of this partitioning may include a set of sub-blocks that may be referred to as prediction units (PUs). Transform units (TUs) may also be utilized.
[0129] Still reference Fig.17 , the example video encoder 1700 may include: an intra-frame prediction processor 1708; a motion estimation / compensation processor 1712, which may also be referred to as an inter-frame prediction processor, capable of constructing a motion vector candidate list, including adding a global motion vector candidate to the motion vector candidate list; a transform / quantization processor 1716; an inverse quantization / inverse transform processor 1720; a loop filter 1724; a decoded picture buffer 1728; and / or an entropy coding processor 1732. The bitstream parameters may be input to the entropy coding processor 1732 to be included in the output bitstream 1736.
[0130] In operation, and continue to refer to Fig.17, for each block of a frame of the input video, it may be determined whether the block is to be processed via intra picture prediction or using motion estimation / compensation. The block may be provided to an intra prediction processor 1708 or a motion estimation / compensation processor 1712. If the block is to be processed via intra prediction, the intra prediction processor 1708 may perform processing to output a prediction value. If the block is to be processed via motion estimation / compensation, the motion estimation / compensation processor 1712 may perform processing including constructing a motion vector candidate list, including adding a global motion vector candidate to the motion vector candidate list where applicable.
[0131] Further references Fig.17 , a residual may be formed by subtracting the predicted value from the input video. The residual may be received by a transform / quantization processor 1716, which may perform a transform process (e.g., a discrete cosine transform (DCT)) to produce coefficients that may be quantized. The quantized coefficients and any associated signaling information may be provided to an entropy coding processor 1732 for entropy coding and inclusion in an output bitstream 1736. The entropy coding processor 1732 may support encoding of signaling information associated with encoding the current block. In addition, the quantized coefficients may be provided to an inverse quantization / inverse transform processor 1720, which may reproduce pixels, which may be combined with a predictor and processed by a loop filter 1724, the output of which may be stored in a decoded picture buffer 1728 for use by a motion estimation / compensation processor 1712, which may be capable of constructing a motion vector candidate list, including adding a global motion vector candidate to the motion vector candidate list.
[0132] Continue to refer Fig.17 Although some variations have been described in detail above, other modifications or additions are possible. For example, in some implementations, the current block may include any symmetric block (8×8, 16×16, 32×32, 64×64, 128×128, etc.) and any asymmetric block (8×4, 16×8, etc.).
[0133] In some implementations, reference is still made to Fig.17 , a quadtree plus binary decision tree (QTBT) can be implemented. In QTBT, at the coding tree unit level, the partition parameters of QTBT can be dynamically derived to adapt to local characteristics without any transmission overhead. Subsequently, at the coding unit level, the joint classifier decision tree structure can eliminate unnecessary iterations and control the risk of wrong prediction. In some implementations, the LTR frame block update mode can be an additional option available at each leaf node of the QTBT.
[0134] In some implementations, reference is still made to Fig.17, additional syntax elements can be signaled at different hierarchical levels of the bitstream. For example, a flag can be enabled for the entire sequence by including an enable flag encoded in a sequence parameter set (SPS). In addition, a CTU flag can be encoded at the coding tree unit (CTU) level.
[0135] Some embodiments may include a non-transitory computer program product (ie, a physically implemented computer program product) storing instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform the operations described herein.
[0136] Still reference Fig.17 , the encoder 1700 may include the following circuitry, which is configured to implement any of the operations described above in any embodiment in any order and at any degree of repetition. For example, the encoder 1700 may be configured to repeatedly perform a single step or sequence until the desired or commanded result is achieved; the repetition of a step or sequence of steps may be performed iteratively and / or recursively, using the output of a previous repetition as the input of a subsequent repetition, aggregating the input and / or output of a repetition to produce an aggregated result, reducing or decrementing one or more variables, such as global variables, and / or dividing a larger processing task into a collection of smaller processing tasks addressed iteratively. The encoder 1700 may perform any step or sequence of steps described in the present disclosure in parallel, such as using two or more parallel threads, processor cores, etc. to perform the steps two or more times simultaneously and / or substantially simultaneously; the task division between parallel threads and / or processes may be performed according to any protocol suitable for task division between iterations. After reading the entire contents of the present disclosure, those skilled in the art will be aware of various ways in which steps, sequence of steps, processing tasks, and / or data can be subdivided, shared, or processed using iteration, recursion, and / or parallel processing.
[0137] Continue to refer Fig.17, a non-transitory computer program product (i.e., a physically implemented computer program product) may store instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform the operations and / or steps thereof described in the present disclosure, including but not limited to any of the operations described above and / or any operations that the decoder and / or encoder may be configured to perform. Similarly, the following computer system is also described, which may include one or more data processors, and a memory coupled to one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more operations described herein. In addition, the method may be implemented by one or more data processors within a single computing system or distributed in two or more computing systems. Such computing systems may be connected via one or more connections, and data and / or commands or other instructions may be exchanged via one or more connections, including connections via a direct connection between one or more of the multiple computing systems through a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, etc.), etc.
[0138] It should be noted that any one or more aspects and embodiments described herein can be conveniently implemented using one or more machines (e.g., one or more computing devices used as user computing devices for electronic documents, one or more server devices such as document servers, etc.), arranged according to the teachings of this specification, which will be apparent to those of ordinary skill in the computer arts. A skilled programmer can readily prepare appropriate software coding based on the teachings of this disclosure, which will be apparent to those of ordinary skill in the software arts. The aspects and implementations discussed above that employ software and / or software modules may also include appropriate hardware for facilitating the implementation of the machine executable instructions of the software and / or software modules.
[0139] Such software can be a computer program product using a machine-readable storage medium. A machine-readable storage medium can be any of the following media, any of which can store and / or encode a sequence of instructions executed by a machine (e.g., a computing device) and cause the machine to perform any of the methods and / or embodiments described herein. Examples of machine-readable storage media include, but are not limited to, disks, optical disks (e.g., CDs, CD-Rs, DVDs, DVD-Rs, etc.), magneto-optical disks, read-only memory "ROM" devices, random access memory "RAM" devices, magnetic cards, optical cards, solid-state storage devices, EPROMs, EEPROMs, and any combination thereof. The machine-readable medium used herein is intended to include a single medium and a collection of physically separated media, for example, a collection of optical disks or one or more hard disk drives combined with a computer memory. As used herein, a machine-readable storage medium does not include a temporary form of signal transmission.
[0140] Such software may also include information (e.g., data) carried as a data signal on a data carrier such as a carrier wave. For example, machine executable information may be included as a data bearing signal embodied in a data carrier in which the signal encodes a sequence of instructions or a portion thereof for execution by a machine (e.g., a computing device), and any associated information (e.g., data structures and data) that causes the machine to perform any of the methods and / or embodiments described herein.
[0141] Examples of computing devices include, but are not limited to, electronic book reading devices, computer workstations, terminal computers, server computers, handheld devices (e.g., tablet computers, smart phones, etc.), network equipment, network routers, network switches, bridges, any machine capable of executing a sequence of instructions that specifies actions to be taken by the machine, and any combination thereof. In one example, the computing device may include and / or be included in a kiosk.
[0142] Fig.18 A diagram of one embodiment of a computing device in the exemplary form of a computer system 1800 is shown within which an instruction set for causing a control system to perform any one or more of the aspects and / or methods of the present disclosure may be executed. It is also contemplated that a plurality of computing devices may be utilized to implement a specifically configured instruction set for causing one or more devices to perform any one or more of the aspects and / or methods of the present disclosure. The computer system 1800 includes a processor 1804 and a memory 1808 that communicate with each other and with other components via a bus 1812. The bus 1812 may include any of several types of bus structures, including but not limited to a memory bus, a memory controller, a peripheral bus, a local bus, and any combination thereof using any of a variety of bus architectures.
[0143] Processor 1804 may include any suitable processor, such as but not limited to a processor incorporating logic circuits for performing arithmetic and logical operations, such as an arithmetic and logic unit (ALU), which may be regulated by a state machine and guided by operational inputs from memory and / or sensors; as non-limiting examples, processor 1804 may be organized according to von Neumann and / or Harvard architectures. Processor 1804 may include, incorporate and / or be incorporated into but not limited to a microcontroller, a microprocessor, a digital signal processor (DSP), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a graphics processing unit (GPU), a general-purpose GPU, a tensor processing unit (TPU), an analog or mixed signal processor, a trusted platform module (TPM), a floating point unit (FPU) and / or a system on a chip (SoC).
[0144] The memory 1808 may include various components (e.g., machine-readable media), including, but not limited to, random access memory components, read-only components, and any combination thereof. In one example, a basic input / output system 1816 (BIOS) may be stored in the memory 1808, including basic routines such as those that help transfer information between elements within the computer system 1800 during startup. The memory 1808 may also include instructions (e.g., software) 1820 (e.g., stored on one or more machine-readable media) that implement any one or more aspects and / or methods of the present disclosure. In another example, the memory 1808 may also include: any number of program modules, including, but not limited to, an operating system, one or more application programs, other program modules, program data, and any combination thereof.
[0145] The computer system 1800 may also include a storage device 1824. Examples of storage devices (e.g., storage device 1824) include, but are not limited to, hard disk drives, disk drives, optical drives combined with optical media, solid-state storage devices, and any combination thereof. The storage device 1824 may be connected to the bus 1812 via an appropriate interface (not shown). Example interfaces include, but are not limited to, SCSI, Advanced Technology Attachment (ATA), Serial ATA, Universal Serial Bus (USB), IEEE 1394 (Firewire), and any combination thereof. In one example, the storage device 1824 (or one or more of its components) may be removably docked with the computer system 1800 (e.g., via an external port connector (not shown)). Specifically, the storage device 1824 and the associated machine-readable medium 1828 may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for the computer system 1800. In one example, the software 1820 may reside entirely or partially in the machine-readable medium 1828. In another example, the software 1820 may reside entirely or partially in the processor 1804.
[0146] The computer system 1800 may also include an input device 1832. In one example, a user of the computer system 1800 may enter commands and / or other information into the computer system 1800 via the input device 1832. Examples of the input device 1832 include, but are not limited to, an alphanumeric input device (e.g., a keyboard), a pointing device, a joystick, a game controller, an audio input device (e.g., a microphone, a voice response system, etc.), a cursor control device (e.g., a mouse), a touch pad, an optical scanner, a video capture device (e.g., a still camera, a video camera), a touch screen, and any combination thereof. The input device 1832 may interface with the bus 1812 via any of a variety of interfaces (not shown), including, but not limited to, a serial interface, a parallel interface, a game port, a USB interface, a FireWire interface, a direct interface to the bus 1812, and any combination thereof. The input device 1832 may include a touch screen interface, which may be part of the display 1836 or separate from the display 1836, as will be discussed further below. The input device 1832 may be used as a user selection device for selecting one or more graphical representations in a graphical interface as described above.
[0147] The user may also input commands and / or other information to the computer system 1800 via a storage device 1824 (e.g., a removable disk drive, a flash drive, etc.) and / or a network interface device 1840. A network interface device such as the network interface device 1840 may be used to connect the computer system 1800 to one or more of various networks such as the network 1844, and one or more remote devices 1848 connected to the various networks. Examples of network interface devices include, but are not limited to, a network interface card (e.g., a mobile network interface card, a LAN card), a modem, and any combination thereof. Examples of networks include, but are not limited to, a wide area network (e.g., the Internet, an enterprise network), a local area network (e.g., a network associated with an office, a building, a campus, or other relatively small geographic space), a telephone network, a data network associated with a telephone / voice provider (e.g., a mobile communications provider data and / or voice network), a direct connection between two computing devices, and any combination thereof. A network such as the network 1844 may employ wired and / or wireless communication modes. In general, any network topology may be used. Information (eg, data, software 1820 , etc.) may be communicated with the computer system 1800 via the network interface device 1840 .
[0148] The computer system 1800 may also include a video display adapter 1852 for transmitting displayable images to a display device such as a display device 1836. Examples of display devices include, but are not limited to, a liquid crystal display (LCD), a cathode ray tube (CRT), a plasma display, a light emitting diode (LED) display, and any combination thereof. The display adapter 1852 and the display device 1836 may be used in conjunction with the processor 1804 to provide graphical representations of various aspects of the present disclosure. In addition to the display device, the computer system 1800 may also include: one or more other peripheral output devices, including, but not limited to, audio speakers, printers, and any combination thereof. Such peripheral output devices may be connected to the bus 1812 via a peripheral interface 1856. Examples of peripheral interfaces include, but are not limited to, serial ports, USB connections, FireWire connections, parallel connections, and any combination thereof.
[0149] The illustrative embodiments of the present invention have been described in detail above. Various modifications and additions may be made without departing from the spirit and scope of the present invention. The features of each of the various embodiments described above may be appropriately combined with the features of the other described embodiments so as to provide a variety of feature combinations in related new embodiments. In addition, although a plurality of separate embodiments have been described above, what has been described herein is only an illustration of the application of the principles of the present invention. In addition, although the specific methods herein may be shown and / or described as being performed in a particular order, the ordering is highly variable within the capabilities of the ordinary technician to implement the methods, systems, and software disclosed herein. Therefore, this description is meant only as an example, and is not otherwise intended to limit the scope of the present invention.
[0150] Exemplary embodiments have been disclosed above and shown in the accompanying drawings. It will be appreciated by those skilled in the art that various changes, omissions and additions may be made to the contents specifically disclosed herein without departing from the spirit and scope of the present invention.
Claims
1. An encoder for a video for a machine-oriented video coding application, the encoder comprising: Autoencoder encoder part; a latent feature encoder coupled to the autoencoder encoder portion; an autoencoder description encoder coupled to the autoencoder encoder portion; A multiplexer is coupled to the latent feature encoder and the autoencoder description encoder and provides an encoded bit stream.
2. The encoder according to claim 1, wherein The autoencoder encoder part is a neural network comprising a plurality of consecutive encoder layers and a bottleneck layer after the final encoder layer.
3. The encoder according to claim 2, wherein: Each successive encoder layer has fewer neurons than the previous layer, and the bottleneck layer has fewer neurons than the final encoder layer.
4. The encoder according to claim 3, wherein: The bottleneck layer has fewer neurons than the final encoder layer.
5. The encoder according to claim 2, wherein: The number of encoder layers is the same as the number of layers in the decoder part of a compatible decoder.
6. A decoder for coded video for a machine-oriented video coding application, the decoder comprising: a demultiplexer, receiving a bit stream encoded by the autoencoder; a latent feature decoder coupled to the demultiplexer; an autoencoder description decoder coupled to the demultiplexer; An autoencoder decoder portion is coupled to the autoencoder description decoder and the latent feature decoder and provides a decoded video for machine use.
7. The decoder according to claim 5, wherein: The autoencoder-decoder part is a neural network comprising a bottleneck layer as an input layer and a plurality of consecutive decoder layers.
8. The decoder according to claim 6, wherein: Each successive decoder layer has more neurons than the previous layer.
9. The decoder according to claim 7, wherein: The bottleneck layer has fewer neurons than the first decoder layer.
10. The decoder according to claim 6, wherein: The number of decoder layers is the same as the number of layers in the encoder part of the encoder that generates the encoded bitstream.
11. A system for encoding image data in a machine-oriented video encoding application, the system comprising: A machine-oriented video coding VCM encoder, the VCM encoder comprising an autoencoder encoder part and an autoencoder bottleneck part, the VCM encoder receiving image information and generating an encoded bit stream including feature data of the image information; as well as A VCM decoder, wherein the VCM decoder comprises the autoencoder bottleneck part and the autoencoder decoder part, and the VCM decoder receives the encoded bit stream and reconstructs the image information according to the feature data.
12. The system according to claim 11, wherein: The autoencoder encoder part is a neural network comprising a plurality of consecutive encoder layers, wherein each consecutive encoder layer has fewer neurons than the previous layer, and the bottleneck part has fewer neurons than the final encoder layer.
13. The system according to claim 12, wherein: The autoencoder-decoder part is a neural network comprising a plurality of consecutive encoder layers, wherein each consecutive decoder layer has more neurons than the previous layer, and the bottleneck part has fewer neurons than the first decoder layer.
14. The system according to claim 13, wherein: The number of encoder layers in the encoder part is the same as the number of layers in the decoder part.
15. The system according to claim 11, wherein: The autoencoder encoder portion is in communication with the autoencoder decoder portion, and updates to the autoencoder are communicated between the VCM encoder and the VCM decoder.
16. The system according to claim 11, wherein: The VCM encoder also includes: an autoencoder description encoder coupled to the autoencoder encoder portion; a latent feature encoder coupled to the autoencoder encoder portion; and A multiplexer is coupled to the latent feature encoder and the autoencoder description encoder and provides an encoded bit stream.
17. The system of claim 11, wherein: The VCM decoder further comprises: a demultiplexer, receiving the encoded bit stream; a latent feature decoder coupled to the demultiplexer; and The autoencoder describes a decoder, coupled to the demultiplexer, Wherein, the autoencoder decoder portion is coupled to the autoencoder description decoder and the latent feature decoder.
