Feature map encoding and tool selection for regular video encoding

Abstract

An apparatus for generating first encoded data and second encoded data. The apparatus includes a determination unit to determine whether the apparatus generates encoded data that includes a feature map based on a neural network for the encoded data. The apparatus also includes an encoding unit to generate, in a case that the apparatus generates first encoded data in a form of encoded video data that does not include the feature map, the first encoded data using a plurality of functions used to encode the video data. The encoding unit generates, in a case that the apparatus generates second encoded data that includes the feature map, the encoded data for the feature map using a first portion of the plurality of functions and not using a second portion of the plurality of functions.

Description

[0001] Cross-reference of related applications

[0002] This application claims the benefit of Australian Patent Application 2021202142, filed on 7 April 2021, pursuant to 35U.SC §119, which is incorporated herein by reference in its entirety as if set forth herein. Technical Field

[0003] This invention generally relates to digital video signal processing, and more particularly to methods, apparatus, and systems for encoding and decoding tensors from convolutional neural networks. The invention also relates to a computer program product comprising a computer-readable medium having recorded computer programs for encoding and decoding tensors from convolutional neural networks using video compression techniques. Background Technology

[0004] Video compression is a ubiquitous technology used to support many applications, including those for transmitting and storing video data. Numerous video coding standards have been developed, and others are currently under development. Recent advancements in video coding standardization have led to the formation of a group known as the Joint Video Experts Group (JVET). The JVET comprises members from two Standards Setting Organizations (SSOs): Study Group 16, Question 6 (SG16 / Q6) of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T), also known as the Video Coding Experts Group (VCEG); and Joint Technical Committee 1 / Subcommittee 29 / Working Group 11 (ISO / IEC JTC1 / SC29 / WG11), also known as the Moving Picture Experts Group (MPEG).

[0005] The Joint Video Experts Group (JVET) has developed a video compression standard called "Various Video Coding" (VVC).

[0006] Convolutional Neural Networks (CNNs) are an emerging technology for tackling use cases involving machine vision, such as object recognition, object tracking, human pose estimation, and action recognition. A CNN typically consists of many layers, such as convolutional layers and fully connected layers, where data is passed from one layer to the next in the form of "tensors." During the training phase, the weights of each layer are determined, where a very large amount of training data is passed through the CNN, and the determined results are compared to the ground truth associated with the training data. Processes such as stochastic gradient descent are applied to iteratively refine the network weights until the network performs at the desired level of accuracy. When the convolutional stage has a "stride" greater than 1, the output tensor from the convolution has a lower spatial resolution than the corresponding input tensor. Operations such as "max pooling" further reduce the spatial size of the output tensor compared to the input tensor. Max pooling produces an output tensor by splitting the input tensor into groups of data samples (e.g., 2×2 groups of data samples) and selecting the maximum value from each group as the corresponding value in the output tensor. The process of using the input to perform a CNN and progressively transforming the input into the output is often referred to as "inference".

[0007] Typically, a tensor has four dimensions: batch, channels, height, and width. When inferring video data, the first dimension, "batch," of size "one," indicates that one frame is passed through the CNN at a time. As the network is trained, the value of the batch dimension can be increased so that multiple frames are passed through the network before updating the network weights, according to a predetermined "batch size." Multi-frame video can be passed through as a single tensor whose batch dimension increases with the number of frames in a given video. However, for practical considerations related to memory consumption and access, inference of video data is usually performed frame-by-frame. The "channels" dimension indicates the number of concurrent "feature maps" for a given tensor, and the height and width dimensions indicate the size of the feature maps at a specific stage of the CNN. The number of channels passing through the CNN varies depending on the network architecture. The size of the feature maps also varies depending on the subsampling that occurs in a particular network layer.

[0008] The input to the first layer of a CNN is an image or video frame, which is typically resized to match the dimensions of the tensors input to the first layer. The dimensions of the tensors depend on the CNN architecture, which usually has some dimensions related to the input width and height, as well as additional "channel" dimensions.

[0009] Slicing a tensor based on its channels produces a set of "feature maps." This is because each slice of the tensor has some relation to its corresponding input image, thus capturing properties such as edges. In layers further from the network's input, these relationships may be more abstract. The "task performance" of a CNN is measured by comparing its results on a specific input to the provided ground truth (i.e., "training data"), where the ground truth is typically prepared by humans to indicate the "correct" result.

[0010] Once the network topology is determined, the network weights can be updated over time as more training data becomes available. A portion of the CNN can also be retrained, keeping the weights of the rest of the network constant. The overall complexity of CNNs tends to be quite high when performing a large number of multiplications (accumulation operations) and writing and reading a large number of intermediate tensors to and from memory. In some applications, CNNs are implemented entirely in the "cloud," leading to the need for high and expensive processing power. In other applications, CNNs are implemented in edge devices such as cameras or mobile phones, resulting in less flexibility but a more distributed processing load.

[0011] VVC is expected to address the ongoing demand for even higher compression performance, particularly as video formats become more capable (e.g., with higher resolution and higher frame rates), and the growing market demand for services delivered over WANs (where bandwidth costs are relatively high). VVC can be implemented in contemporary silicon processes and offers an acceptable trade-off between the achieved performance and implementation cost. Implementation cost can be considered in terms of one or more aspects such as silicon area, CPU processor load, memory utilization, and bandwidth. Part of the universality of the VVC standard lies in the wide selection of tools that can be used to compress video data and the broad range of applications to which VVC is suitable.

[0012] Video data consists of a sequence of frames of image data, each frame containing one or more color channels. Typically, one primary color channel and two secondary color channels are required. The primary color channel is usually referred to as the "luminance" channel, and the (one or more) secondary color channels are usually referred to as the "chrominance" channels. Although video data is typically displayed in the RGB (Red-Green-Blue) color space, this color space is highly correlated between its three corresponding components. The video data representation seen by the encoder or decoder typically uses a color space such as YCbCr. YCbCr concentrates luminance (mapped to "luminance" according to the transfer function) in the Y (primary) channel and chrominance in the Cb and Cr (secondary) channels. Due to the use of decorrelated YCbCr signals, the statistics of the luminance channel differ significantly from those of the chrominance channel. The main difference is that, after quantization, the chrominance channel contains relatively fewer effective coefficients for a given block compared to the coefficients of the corresponding luminance channel block. Furthermore, the Cb and Cr channels can be spatially sampled at a lower rate (subsampling) compared to the luma channel (e.g., half horizontally and half vertically (known as the "4:2:0 chroma format")). The 4:2:0 chroma format is commonly used in applications such as internet video streaming, broadcast television, and Blu-ray. TM Used in "consumer" applications (such as on-disk storage). When only luminance samples are present, the resulting monochrome frames are referred to as using the "4:0:0 chroma format".

[0013] The VVC standard specifies a "block-based" architecture in which frames are first divided into an array of square regions known as "Code Tree Units" (CTUs). CTUs typically occupy relatively large areas, such as 128×128 luma samples. However, the area of ​​CTUs at the right and bottom edges of individual frames may be smaller. Associated with each CTU is a "coding tree" ("shared tree") for both luma and chroma channels, or separate trees for each channel. The coding tree defines the breakdown of the CTU's region into a set of blocks (also called "coded blocks" (CBs)). When using a shared tree, a single coding tree specifies blocks for both luma and chroma channels; in this case, the set of juxtaposed coding blocks is called a "Code Unit" (CU) (i.e., each CU has coding blocks for its respective color channel). CBs are processed in a specific order for encoding or decoding. Because of the use of the 4:2:0 chroma format, a CTU with a luma coding tree for a 128×128 luma sample region has a corresponding chroma coding tree for a 64×64 chroma sample region (juxtaposed with the 128×128 luma sample region). When a single coding tree is used for both luma and chroma channels, the set of juxtaposed blocks for a given region is typically referred to as a “unit,” such as the CU mentioned above, as well as “prediction units” (PUs) and “transform units” (TUs). A single tree with CUs spanning the chroma channels of 4:2:0 chroma format video data produces chroma blocks that are half the width and height of the corresponding luma blocks. When a separate coding tree is used for a given region, the CBs mentioned above, as well as “prediction blocks” (PBs) and “transform blocks” (TBs), are used.

[0014] Despite the aforementioned distinction between “unit” and “block”, the term “block” can be used as a general term for a frame’s area or region for which operations are applied across all color channels.

[0015] For each CU, a prediction unit (PU) (“prediction unit”) is generated for the content (sample values) of the corresponding region of the generated frame data. Furthermore, a representation of the difference (or spatial domain “residual”) between the prediction and the content of the region seen at the encoder input is formed. The differences in the individual color channels can be transformed and encoded into a sequence of residual coefficients, forming one or more TUs for a given CU. The applied transformation can be a Discrete Cosine Transform (DCT) or other transformation applied to individual blocks of residual values. The transformation is applied separately (i.e., a two-dimensional transformation is performed in two processes). First, the block is transformed by applying a one-dimensional transformation to each row of samples in the block. Then, the partial result is transformed by applying a one-dimensional transformation to each column of the partial result to produce a final block of transform coefficients that are substantially decorrelated to the residual samples. The VVC standard supports transforms of various sizes, including transforms of rectangular blocks whose sides are powers of 2. The transform coefficients are quantized for entropy encoding into the bitstream.

[0016] VVC is characterized as intra-frame prediction and inter-frame prediction. Intra-frame prediction involves using previously processed samples from the frame being used to generate a prediction of the current data sample block in that frame. Inter-frame prediction involves using sample blocks obtained from previously decoded frames to generate a prediction of the current sample block in a frame. The sample blocks obtained from previously decoded frames are offset relative to the spatial location of the current block according to a motion vector, which is typically filtered. Intra-frame prediction blocks can be: (i) uniform sample values ​​(“DC intra-frame prediction”), (ii) a plane with offsets and horizontal and vertical gradients (“planar intra-frame prediction”), (iii) a total of blocks with neighboring samples applied in a specific direction (“angular intra-frame prediction”), or (iv) the result of matrix multiplication using neighboring samples and selected matrix coefficients. Further discrepancies between the prediction block and the corresponding input samples can be corrected to some extent by encoding the “residual” into the bitstream. The residuals are typically transformed from the spatial domain to the frequency domain to form residual coefficients in the "primary transform domain." These residual coefficients can be further transformed by applying a "secondary transform" to produce residual coefficients in the "secondary transform domain." Quantizing the residual coefficients according to the quantization parameters results in a loss of reconstruction accuracy of the samples generated at the decoder, but a reduction in the bit rate in the bitstream. Summary of the Invention

[0017] The object of the present invention is to substantially overcome or at least improve one or more disadvantages of the existing arrangement.

[0018] According to one aspect of this disclosure, an apparatus for generating first encoded data and second encoded data is provided, the apparatus comprising:

[0019] A determining unit is configured to determine whether the device generates encoded data including encoded data based on feature maps from a neural network; and

[0020] An encoding unit is configured to generate the first encoded data using multiple functions for encoding video data when the device generates the first encoded data in the form of encoded video data excluding the feature map, wherein when the device generates the second encoded data including the feature map, the encoding unit uses a first portion of the multiple functions but not a second portion of the multiple functions to generate the encoded data of the feature map.

[0021] According to another aspect of this disclosure, an apparatus is provided for decoding first encoded data and second encoded data, the apparatus comprising:

[0022] A determining unit is configured to determine whether the device decodes encoded data, including encoded data based on feature maps from a neural network; and

[0023] A decoding unit is configured to decode the first encoded data using multiple functions for decoding video data when the device decodes the first encoded data in the form of encoded video data that does not include the feature map, wherein when the device decodes the second encoded data that includes the encoded data of the feature map, the decoding unit decodes the encoded data of the feature map using a first portion of the multiple functions but not a second portion of the multiple functions.

[0024] According to another aspect of this disclosure, a method for generating first coded data and second coded data is provided, the method comprising:

[0025] Determine whether the device generates encoded data that includes encoded data based on neural network feature maps;

[0026] When the device generates first encoded data in the form of encoded video data that does not include the feature map, the first encoded data is generated using multiple functions for encoding video data; and

[0027] When the device generates the second encoded data including the encoded data of the feature map, the encoded data of the feature map is generated using the first part of the plurality of functions but not the second part of the plurality of functions.

[0028] According to another aspect of this disclosure, a method for decoding first encoded data and second encoded data is provided, the method comprising:

[0029] Determine whether the device decodes encoded data, including encoded data based on neural network feature maps;

[0030] When the device decodes the first encoded data in the form of encoded video data that does not include the feature map, multiple functions for decoding video data are used to decode the first encoded data; and

[0031] When the device decodes the second encoded data including the encoded data of the feature map, the encoded data of the feature map is decoded using the first part of the plurality of functions without using the second part of the plurality of functions.

[0032] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided, which stores a program for performing a method for generating first coded data and second coded data, the method comprising:

[0033] Determine whether the device generates encoded data that includes encoded data based on neural network feature maps;

[0034] When the device generates first encoded data in the form of encoded video data that does not include the feature map, the first encoded data is generated using multiple functions for encoding video data; and

[0035] When the device generates the second encoded data including the encoded data of the feature map, the encoded data of the feature map is generated using the first part of the plurality of functions but not the second part of the plurality of functions.

[0036] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided, which stores a program for performing a method of decoding first encoded data and second encoded data, the method comprising:

[0037] Determine whether the device decodes encoded data, including encoded data based on neural network feature maps;

[0038] When the device decodes the first encoded data in the form of encoded video data that does not include the feature map, multiple functions for decoding video data are used to decode the first encoded data; and

[0039] When the device decodes the second encoded data including the encoded data of the feature map, the encoded data of the feature map is decoded using the first part of the plurality of functions without using the second part of the plurality of functions.

[0040] Other aspects were also disclosed. Attached Figure Description

[0041] At least one embodiment of the invention will now be described with reference to the following figures and appendices, wherein:

[0042] Figure 1 This is a schematic block diagram illustrating a distributed machine task system;

[0043] Figure 2A and Figure 2B Formation that can be put into practice Figure 1 A schematic block diagram of a general-purpose computer system for distributed machine task systems;

[0044] Figure 3A This is a schematic block diagram showing the functional modules of the backbone of a CNN;

[0045] Figure 3B It is shown Figure 3A A schematic diagram of the remaining blocks;

[0046] Figure 3C It is shown Figure 3A A schematic block diagram of the remaining units;

[0047] Figure 3D It is shown Figure 3A A schematic block diagram of the CBL module;

[0048] Figure 4 This is a schematic block diagram illustrating the functional modules of the optional backbone of a CNN;

[0049] Figure 5 This is a schematic block diagram showing a feature map quantizer and a packer as part of a distributed machine task system;

[0050] Figure 6 This is a schematic block diagram showing the functional modules of a video encoder;

[0051] Figure 7 This is a schematic block diagram illustrating the functional modules of the video decoder;

[0052] Figure 8 This is a schematic block diagram showing a feature map inverse quantizer and unpacker as part of a distributed machine task system;

[0053] Figure 9A This is a schematic diagram showing the head of a CNN;

[0054] Figure 9B It is shown Figure 9A A schematic block diagram of the upgrader module;

[0055] Figure 9C It is shown Figure 9A A schematic block diagram of the detection module;

[0056] Figure 10 This is a schematic diagram showing the optional heads of a CNN;

[0057] Figure 11 This is a schematic block diagram showing the packing arrangement of feature maps in a monochrome frame;

[0058] Figure 12 This is a schematic block diagram illustrating the packing arrangement of alternative feature maps in a monochrome frame;

[0059] Figure 13 This is a schematic block diagram showing the packing arrangement of feature maps in a 4:2:0 chroma subsampled color frame;

[0060] Figure 14 This is a schematic block diagram showing the bitstream that preserves the encoded packing feature map and associated metadata;

[0061] Figure 15 This illustrates a method for performing the first part of a CNN and encoding the resulting feature maps;

[0062] Figure 16 This illustrates a method for decoding feature maps and performing the second part of a CNN;

[0063] Figure 17 This illustrates a method for determining the grouping of feature maps;

[0064] Figure 18 This demonstrates a method for selecting a set of encoding tools or functions from a video standard; and

[0065] Appendix A is a syntax table showing the format of the Supplemental Enhancement Information (SEI) message, which represents the metadata associated with feature map packing and quantization in the bitstream. Detailed Implementation

[0066] Where steps and / or features with the same reference numerals are referenced in any one or more drawings, unless the contrary is intended, these steps and / or features have the same (one or more) function or (one or more) operation for the purposes of this specification.

[0067] Distributed machine task systems can include edge devices, such as webcams or smartphones that generate intermediate compressed data. Distributed machine task systems can also include end devices, such as server farm (“cloud”) applications that manipulate the intermediate compressed data to produce task results. Furthermore, the functionality of the edge devices can be implemented in the cloud, and the intermediate compressed data can be stored for later processing (potentially for multiple different tasks as needed).

[0068] A convenient form of intermediate compressed data is compressed video bitstreams, thanks to the availability of high-performance compression standards and their implementations. Video compression standards typically operate on integer samples given a localization depth (e.g., 10 bits arranged in a planar array). Color video has three planar arrays, corresponding, depending on the application, to color components Y, Cb, Cr or R, G, B. CNNs typically operate on floating-point data in tensor form, which generally has a much smaller spatial dimension than the input video data to the CNN operation, but significantly more channels than the typical three channels of color video data.

[0069] Tensors typically have the following dimensions: frame, channel, height, and width. For example, a tensor with dimensions [1, 256, 76, 136] can be considered to contain 256 (256) feature maps, each of size 136 × 76. For video data, inference is usually performed one frame at a time, rather than using tensors containing multiple frames.

[0070] VVC encoders and decoders include capability signaling mechanisms known as “constraints.” Early on, in a bitstream, there existed a set of constraints indicating which capabilities of the VVC standard were not used in the bitstream. These constraints were signaled along with the bitstream’s “profile” and “level.” A profile broadly indicated which set of tools was needed to decode the bitstream. Constraints also provided fine-grained control over which tools were further constrained within a given profile. Further constraints on tools were similar to “sub-profiles,” but sub-profiles were defined outside the VVC standard, while the general constraint signaling semantics were defined within the VVC standard. Sub-profiles (e.g., equivalently defined) of a subset of tools are defined based on the type of data being encoded by the video encoder, allowing the decoder to know, before beginning bitstream decoding, the subset of encoding tools for the indicated profile of the bitstream to be used.

[0071] Figure 1 This is a schematic block diagram illustrating the functional modules of a distributed machine task system 100. System 100 can be used to implement a method for efficiently packing and quantizing feature maps into planar frames to encode and decode feature maps relative to encoded data, such that the associated overhead data is not too cumbersome, and the task performance regarding decoding feature maps is resilient to changes in the bit rate of the bit stream.

[0072] System 100 includes a source device 110 for generating encoded data in the form of encoded video information. System 100 also includes a destination device 140. A communication channel 130 is used to communicate encoded video information from the source device 110 to the destination device 140. In some arrangements, the source device 110 and the destination device 140 may include one or both of their respective mobile phones (e.g., "smartphones") or webcams and cloud applications. The communication channel 130 may be a wired connection such as Ethernet or a wireless connection such as WiFi or 5G. Furthermore, the source device 110 and the destination device 140 may include applications that capture encoded video data on some computer-readable storage medium, such as a hard drive in a file server.

[0073] like Figure 1 As shown, source device 110 includes a video source 112, a CNN backbone 114, a feature map quantizer and packer 116, a multiplexer 118, a video encoder 120, and a transmitter 122. Video source 112 typically includes a source of captured video frame data (denoted as 113), such as a camera sensor, a previously captured video sequence stored on a non-transitory recording medium, or a video feed from a remote camera sensor. Video source 112 can also be the output of a computer graphics card (e.g., video output from a display operating system and various applications running on a computing device (e.g., a tablet computer)). Examples of source devices 110 that may include a camera sensor as video source 112 include smartphones, video camcorders, professional cameras, and webcams.

[0074] The CNN backbone 114 receives video frame data 113 and processes specific layers of the overall CNN, such as layers corresponding to the "backbone" of the CNN. The CNN backbone layer can produce multiple tensors as outputs, for example, corresponding to different spatial scales of the input image represented by the video frame data 113. A "Feature Pyramid Network" (FPN) architecture can produce three tensors corresponding to the three layers output from the backbone 114, these three tensors having varying spatial resolutions and channel numbers. A feature map quantizer and packer 116 receives tensor 115 output from the CNN backbone 114. The feature map quantizer and packer 116 performs the action of quantizing the floating-point values ​​in tensor 115 into data samples packed into frame 119 to connect the inner layers of the overall CNN, which are the outputs of the CNN backbone 114, to the video encoder 120. The resolution of frame 119 can be based on the total region of the feature maps to be encoded and the target aspect ratio. If too many unused areas appear in frame 119 during packing, the frame size can be increased (e.g., the height can be increased) so that all feature maps can be placed in frame 119. For example, the resolution of frame 119 could be 2056×1224, and the bit depth of frame 119 could be ten (10) bits. Determining the placement of feature maps in frame 119 only needs to be done when the dimensions of tensor 115 are established. Slicing tensor 115 along the channel dimensions results in the extraction of a feature map for each channel, where the feature map of a given tensor has a specific size determined by the additional dimensions of the tensor. In the case of using FPN, multiple tensors are generated for each incoming frame, comprising multiple sets of feature maps, each set having a different spatial resolution. Feature maps from all layers are packed into planar video frames, such as packed feature map frame 117. If source device 110 is configured to encode feature maps, multiplexer 118 selects the packed feature map frame 117; or if source device 110 is configured to encode video data, multiplexer 118 selects frame data 113, thereby outputting frame 119 to an encoding unit in the form of video encoder 120. The selection between feature maps and regular video data is encoded into the bitstream using the “frame_type” syntax element in the metadata SEI message. Refer to Appendix A for a description of the metadata SEI message. Frame 119 is input to video encoder 120, where lossy compression is applied to frame 119 to produce bitstream 121. Bitstream 121 is supplied to transmitter 122 for transmission via communication channel 130, or bitstream 121 is written to storage unit 132 for later use.

[0075] After being converted into tensors by the CNN backbone 114, the resulting feature maps no longer identify individuals that are clearly identifiable in the video data 113. In particular, regarding the pseudonymization or anonymization requirements of the European General Data Protection Regulation (GDPR), from a user privacy perspective, storing the feature maps in storage unit 132 (e.g., in compressed form) may be more secure.

[0076] Source device 110 supports a specific network for the CNN backbone 114. However, destination device 140 can use one of several networks for the CNN head 150. In this way, partially processed data in the form of packaged feature maps can be stored for later use in various tasks without having to operate the CNN backbone 114 again. Video encoder 120 encodes frame data 119 using a specific set of VVC encoding tools (or "grades").

[0077] Bitstream 121 is transmitted by transmitter 122 via communication channel 130 as encoded video data (or "encoded video information"). In some implementations, bitstream 121 may be stored in storage unit 132 until it is later transmitted via communication channel 130 (or in lieu of transmission via communication channel 130), wherein storage unit 132 is a non-transitory storage device, such as a "flash" memory or hard disk drive. For example, encoded video data can be provided to clients as needed via a wide area network (WAN) for video streaming applications.

[0078] Destination device 140 includes receiver 142, video decoder 144, demultiplexer 146, feature map unpacker and inverse quantizer 148, CNN head 150, CNN task 152, and display device 160. Receiver 142 receives encoded video data from communication channel 130 and passes the received video data as a bitstream (indicated by arrow 143) to video decoder 144. Video decoder 144 then outputs decoded frame data (indicated by arrow 145) to demultiplexer 146. Decoding metadata 155 is also extracted by video decoder 144 from bitstream 143 and passed to feature map unpacker and inverse quantizer 148. Supplemental Enhancement Information (SEI) messages 1413 (see [link to relevant documentation]) typically present in bitstream 143 are also included. Figure 14Decoding metadata 155 is obtained. Appendix A shows example syntax of decoding metadata 155 and the semantics of each example syntax element. Decoding metadata 155 may exist and be decoded from the bitstream for each frame. Decoding metadata 155 may exist and be decoded less frequently relative to each frame. For example, decoding metadata 155 may exist and be decoded only for intra-frame images in bitstream 143. When decoding metadata 155 does not exist for a given frame, the most recently available metadata is used. If destination device 140 is configured to perform a CNN task (as indicated by the “frame_type” syntax element in SEI message 1413 of bitstream 143), frame data 145 is output as feature map frame data 147 to the feature map unpacker and inverse quantizer 148. Otherwise, if destination device 140 is configured to decode video data, frame data 145 is output as frame data 159 and supplied to display device 160 for display as video. The feature map unpacker and inverse quantizer output tensor 147, which is fed to the CNN head 150. The CNN head 150 performs the subsequent layers of the task starting from the CNN backbone 114 to produce task result 151, which is stored in task result buffer 152. Examples of display devices 160 include cathode ray tubes, liquid crystal displays, such as smartphones, tablets, computer monitors, or standalone televisions. The functionality of the source device 110 and the destination device 140 can also be embodied in a single device, examples of which include mobile phones, tablets, and cloud applications.

[0079] Although the example apparatus has been described above, the source apparatus 110 and the destination apparatus 140 can each typically be configured within a general-purpose computer system via a combination of hardware and software components. Figure 2AThis computer system 200 is shown, comprising: a computer module 201; input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 configured as a video source 112, and a microphone 280; and output devices including a printer 215, a display device 214 configured as a display device 160, and a speaker 217. The computer module 201 can communicate with a communication network 220 via a connection 221 using an external modem transceiver device 216. The communication network 220, which may represent the communication channel 130, can be a WAN, such as the Internet, a cellular telecommunications network, or a private WAN. If the connection 221 is a telephone line, the modem 216 can be a conventional dial-up modem. Alternatively, if the connection 221 is a high-capacity (e.g., cable or optical) connection, the modem 216 can be a broadband modem. A wireless modem can also be used for wireless connection to the communication network 220. The transceiver device 216 can provide the functions of a transmitter 122 and a receiver 142, and the communication channel 130 can be embodied in the connection part 221.

[0080] Computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, memory unit 206 may have semiconductor random access memory (RAM) and semiconductor read-only memory (ROM). Computer module 201 also includes multiple input / output (I / O) interfaces, including: an audio-video interface 207 coupled to a video display 214, speakers 217, and a microphone 280; an I / O interface 213 coupled to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and optionally a joystick or other human-machine interface device (not shown); and an interface 208 for external modems 216 and printers 215. Signals from the audio-video interface 207 to the computer monitor 214 are typically output from the computer's graphics card. In some implementations, modem 216 may be integrated within computer module 201, for example, within interface 208. Computer module 201 also has a local network interface 211, which allows computer system 200 to be coupled to a local area communication network 222, known as a local area network (LAN), via connection 223. Figure 2A As shown, the local area communication network 222 can also be connected to the wide area network 220 via the connection unit 224, wherein the local area communication network 222 typically includes a so-called "firewall" device or a device with similar functionality. The local network interface 211 may include Ethernet. TM Circuit card, BluetoothTM The interface can be wireless or IEEE 802.11 wireless; however, various other types of interfaces can be implemented for interface 211. The local network interface 211 can also provide the functions of transmitter 122 and receiver 142, and the communication channel 130 can also be embodied in the local area communication network 222.

[0081] I / O interfaces 208 and 213 can provide either serial connectivity or parallel connectivity, wherein the former is typically implemented according to the Universal Serial Bus (USB) standard and has a corresponding USB connector (not shown). A storage device 209 is provided, and storage device 209 typically includes a hard disk drive (HDD) 210. Other storage devices (not shown), such as floppy disk drives and tape drives, may also be used. An optical disc drive 212 is typically provided as a non-volatile source of data. Optical discs, such as CD-ROMs, DVDs, and Blu-ray discs, may be used. TM Portable storage devices such as USB-RAM, portable external hard disk drives, and floppy disks serve as suitable sources of data for computer system 200. Typically, any of HDD 210, optical disc drive 212, network 220, and 222 can also be configured to operate as a video source 112 or as a destination for decoded video data to be stored for reproduction via display 214. The source device 110 and destination device 140 of system 100 can be embodied within computer system 200.

[0082] Components 205-213 of computer module 201 typically communicate via interconnect bus 204 and in a manner known to those skilled in the art to the conventional operating mode of computer system 200. For example, processor 205 is coupled to system bus 204 via connector 218. Similarly, memory 206 and optical disc drive 212 are coupled to system bus 204 via connector 219. Examples of computers in which this arrangement can be practiced include IBM-PC and compatible machines, Sun SPARCstation, and Apple Macs. TM Or a similar computer system.

[0083] Where appropriate or desired, the video encoder 120 and the video decoder 144, as well as the methods described below, can be implemented using the computer system 200. Specifically, the video encoder 120, the video decoder 144, and the methods to be described can be implemented as one or more software applications 233 executable within the computer system 200. In particular, instructions 231 (see [link to instructions]) executed within the computer system 200 in the software 233 are utilized. Figure 2BThe steps of implementing the video encoder 120, the video decoder 144, and the method are described. The software instructions 231 can be configured as one or more code modules, each for performing one or more specific tasks. Alternatively, the software can be divided into two separate parts, where a first part and its corresponding code modules perform the method, and a second part and its corresponding code modules manage the user interface between the first part and the user.

[0084] For example, the software can be stored in a computer-readable medium including the storage means described below. The software is loaded from the computer-readable medium into computer system 200, and then executed by computer system 200. The computer-readable medium having such software, or the computer program recorded on such computer-readable medium, is a computer program product. Using the computer program product in computer system 200 preferably implements advantageous means for carrying out the source device 110 and the destination device 140, as well as the method.

[0085] Software 233 is typically stored in HDD 210 or memory 206. The software is loaded from a computer-readable medium into computer system 200 and executed by computer system 200. Thus, for example, software 233 can be stored on an optically readable disk storage medium (e.g., CD-ROM) 225 read by optical disk drive 212.

[0086] In some instances, application 233 is supplied to the user encoded on one or more CD-ROMs 225 and read via the corresponding drive 212, or alternatively, the user may read application 233 from network 220 or 222. Furthermore, software may also be loaded into computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and / or data to computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, and Blu-ray Discs. TM This includes hard disk drives, ROMs or integrated circuits, USB storage devices, magneto-optical disks, or computer-readable cards such as PCMCIA cards, regardless of whether these devices are inside or outside the computer module 201. Examples of temporary or non-tangible computer-readable transmission media that may also participate in the provision of software, applications, instructions, and / or video data or encoded video data to the computer module 201 include: radio or infrared transmission channels and network connections to other computers or networked devices, as well as the Internet or intranet including email sending and information recorded on websites.

[0087] The second part of the aforementioned application 233 and the corresponding code modules can be executed to implement one or more graphical user interfaces (GUIs) to be drawn or otherwise presented on the display 214. By typically operating the keyboard 202 and mouse 203, users and applications of the computer system 200 can operate the interface in a functionally applicable manner to provide control commands and / or input to applications associated with these (one or more) GUIs. Other functionally applicable forms of user interfaces can also be implemented, such as audio interfaces utilizing voice prompts output via speaker 217 and user voice commands input via microphone 280.

[0088] Figure 2B This is a detailed schematic block diagram of processor 205 and "memory" 234. Memory 234 represents... Figure 2A The computer module 201 can access the logical aggregation of all memory modules (including storage device 209 and semiconductor memory 206).

[0089] When the computer module 201 is initially powered on, the power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in... Figure 2A The ROM 249 of the semiconductor memory 206 is sometimes referred to as firmware. The POST program 250 checks the hardware within the computer module 201 to ensure proper operation, and typically checks the processor 205, the memory 234 (209, 206), and the Basic Input / Output System Software (BIOS) module 251, which is also typically stored in the ROM 249, for correct operation. Once the POST program 250 has successfully run, the BIOS 251 boots. Figure 2A The hard disk drive 210 is booted. Booting the hard disk drive 210 causes the bootloader 252 residing on the hard disk drive 210 to be executed via the processor 205. This loads the operating system 253 into the RAM memory 206, where the operating system 253 begins operation. The operating system 253 is a system-level application executable by the processor 205 to implement various advanced functions, including processor management, memory management, device management, storage management, software application interfaces, and general user interfaces.

[0090] The operating system 253 manages memory 234 (209, 206) to ensure that each process or application running on computer module 201 has sufficient memory to execute without conflicting with memory allocated to other processes. Furthermore, appropriate use of memory is required. Figure 2AThe computer system 200 contains different types of memory to enable efficient operation of various processes. Therefore, aggregate memory 234 is not intended to illustrate how specific segments of memory are allocated (unless otherwise stated), but rather to provide an overview of the memory accessible to the computer system 200 and how such memory is used.

[0091] like Figure 2B As shown, processor 205 includes multiple functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and local or internal memory 248, sometimes referred to as cache memory. Cache memory 248 typically includes multiple storage registers 244-246 in its register section. One or more internal buses 241 functionally interconnect these functional modules. Processor 205 also typically has one or more interfaces 242 for communicating with external devices via system bus 204 using connection 218. Memory 234 is coupled to bus 204 using connection 219.

[0092] Application program 233 includes a sequence of instructions 231, which may contain conditional branch instructions and loop instructions. Program 233 may also include data 232 used when executing program 233. Instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative size of instructions 231 and memory locations 228-230, as described by the instructions shown in memory location 230, a particular instruction may be stored in a single memory location. Alternatively, as described by the instruction segments shown in memory locations 228 and 229, an instruction may be divided into multiple parts, each stored in a separate memory location.

[0093] Typically, a set of instructions is assigned to processor 205, which is then executed within processor 205. Processor 205 awaits subsequent input, which it responds to by executing another set of instructions. Inputs can be provided from one or more sources, including data generated by one or more input devices 202, 203, data received from external sources via one of networks 220, 202, data retrieved from one of storage devices 206, 209, or data retrieved from storage medium 225 inserted into the corresponding reader 212 (all of which are within...). Figure 2A (As shown in the diagram). Executing a set of instructions may result in output data in some cases. Execution may also involve storing data or variables into memory 234.

[0094] The video encoder 120, video decoder 144, and the method can use input variables 254 stored in corresponding memory locations 255, 256, and 257 within memory 234. The video encoder 120, video decoder 144, and the method generate output variables 261 stored in corresponding memory locations 262, 263, and 264 within memory 234. Intermediate variables 258 can be stored in memory locations 259, 260, 266, and 267.

[0095] refer to Figure 2B The processor 205, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 work together to perform micro-operation sequences required for "fetch, decode, and execute" cycles of instructions in the instruction set constituting program 233. Each fetch, decode, and execute cycle includes:

[0096] The fetch operation is used to fetch or read instruction 231 from memory locations 228, 229, and 230;

[0097] Decoding operation, in which the control unit 239 determines which instruction was extracted; and

[0098] The operation is performed, in which the control unit 239 and / or ALU 240 executes the instruction.

[0099] Then, a further fetch, decode, and execute cycle for the next instruction can be performed. Similarly, a storage cycle can be performed, in which the control unit 239 stores or writes a value to or writes it to memory location 232.

[0100] To explain Figure 15 , Figure 16 , Figure 17 and Figure 18 Each step or subprocess in the method is associated with one or more segments of program 233 and is typically performed by working together in the register units 244, 245, 247, ALU 240 and control unit 239 in processor 205 to fetch, decode and execute cycles of instructions in the instruction set of the segment of program 233.

[0101] Figure 3A This is a schematic block diagram illustrating the functional modules of the backbone 310 of a CNN that can be used as the backbone 114 of a CNN. The backbone 114 is sometimes referred to as "DarkNet-53," although different backbones are possible, implementing different numbers and dimensions of layers for tensors 115 for each frame. Reference Figure 14The “backbone_id” syntax element in SEI message 1413, as described in Appendix A, indicates the type of the backbone. When the type of the backbone is unknown, the tensor dimensions are specified using the number of feature maps per layer (“fm_cnt”) and the feature map dimensions per layer (“fm_width” and “fm_height”).

[0102] like Figure 3A As seen, video data 113 is passed to resizing module 304, which resizes the frames to a resolution suitable for processing by the CNN backbone 310, resulting in resized frame data 312. If the resolution of frame data 113 is already suitable for the CNN backbone 310, the operation of resizing module 304 is unnecessary. The resized frame data 312 is then passed to convolutional batch normalization leaky rectified linear (CBL) module 314 to generate tensor 316. CBL 314 contains a reference... Figure 3D The CBL module 360 ​​shown is the module described.

[0103] CBL module 360 ​​takes tensor 361 as input, which is passed to convolutional layer 362 to produce tensor 363. When convolutional layer 362 has a stride of 1, tensor 363 has the same spatial dimension as tensor 361. When convolutional layer 362 has a larger stride (such as 2), tensor 363 has a smaller spatial dimension compared to tensor 361; for example, for a stride of 2, the size of tensor 363 is halved. Regardless of the stride, the channel dimension of tensor 363 can vary compared to the channel dimension of tensor 361 for a given CBL block. Tensor 363 is passed to batch normalization module 364, which outputs tensor 365. Batch normalization module 364 normalizes the input tensor 363, applying a scaling factor and offset value to produce output tensor 365. The scaling factor and offset value are derived from the training process. Tensor 365 is passed to LeakyReLU module 366 to produce tensor 367. Module 366 provides an activation function through which positive values ​​in the tensor are passed through, and negative values ​​are significantly reduced in magnitude, for example, reduced to 0.1 times their previous value.

[0104] Tensor 316 is passed from CBL block 314 to residual block 11 module 320, which contains a cascade of 11 residual units.

[0105] For reference Figure 3BThe ResBlock 340 shown describes the remaining block. ResBlock 340 receives tensor 341, which is zero-padded by zero-padding module 342 to produce tensor 343. Tensor 343 is passed to CBL module 344 to produce tensor 345. Tensor 345 is passed to remaining cell 346, and remaining cell 346 of remaining block 340 contains a series of cascaded remaining cells. The last remaining cell in remaining cell 346 outputs tensor 347. (See reference...) Figure 3C The ResUnit 350 described here describes the remaining units. ResUnit 350 takes tensor 351 as input, which is passed to CBL module 352 to produce tensor 353. Tensor 353 is passed to a second CBL unit 354 to produce tensor 355. Addition module 356 adds tensor 355 to tensor 351 to produce tensor 357. Addition module 356 can also be called a "shortcut" because the input tensor 351 essentially affects the output tensor 357. For an untrained network, ResUnit 350 performs actions to pass through the tensors. During training, CBL modules 352 and 354 perform actions to deviate tensor 357 from tensor 351 based on the training data and ground truth data.

[0106] Res11 module 320 outputs tensor 322, which is output as one of the layers from the backbone module 310 and also provided to Res8 module 324. Res8 module 324 is a residual block (i.e., 340) comprising eight residual units (i.e., 350). Res8 module 324 produces tensor 326, which is passed to Res4 module 328 and also output as one of the layers from the backbone module 310. Res4 module 324 is a residual block (i.e., 340) comprising four residual units (i.e., 350). Res4 module 324 produces tensor 329, which is output as one of the layers from the backbone module 310. Overall, layer tensors 322, 326, and 329 are output as tensor 115. The backbone CNN 310 can take 1088×608 video frames as input and produce three tensors corresponding to the three layers, with the following dimensions: [1, 256, 76, 136], [1, 512, 38, 68], and [1, 1024, 19, 34]. Although... Figure 3A , Figure 3B , Figure 3C and Figure 9A , Figure 9B , Figure 9CThe overall CNN depicted can be partitioned as shown in the figure, but other partitions of the overall CNN are also possible. The tensors output from the first convolutions in CBL blocks 912, 926, and 940 (i.e., tensor 363 in the respective CBL modules) can be tapped as outputs from the backbone, in which case the first convolutions of upscaler modules 922 and 936 and CBL modules 912, 926, and 940 are included in the backbone CNN 310. The resulting tensors have dimensions of [1, 512, 34, 19], [1, 256, 68, 38], and [1, 128, 136, 76]. When enumerating all layers and operations of the YOLOv3 network, the tapped tensor 363 at CBL modules 912, 926, and 940 corresponds to the tapped tensors at modules 75, 90, and 105 in the YOLOv3 network, respectively. Compared to the outputs of “DarkNet-53” (i.e., 322, 326, and 329), the resulting tensors have half the number of feature maps at each resolution.

[0107] Figure 4This is a schematic block diagram illustrating a functional module of an alternative backbone 400 of a CNN that can be used as the backbone 114 of a CNN. The backbone 400 implements a residual network with a feature pyramid network (“ResNet FPN”) and is an alternative to the CNN backbone 114. Frame data 113 is input and passed through stem network 408, res2 module 412, res3 module 416, res4 module 420, res5 module 424 and max pooling module 428 via tensors 409, 413, 417, 425, whereby max pooling module 428 produces tensor 429 as output. Stem network 408 includes 7×7 convolutions and max pooling operations with a stride of two (2). Res2 module 412, res3 module 416, res4 module 420 and res5 module 424 perform convolution operations, i.e., LeakyReLU activation. Each of the modules 421, 416, 420, and 424 also halves the resolution of the processed tensors via a stride setting of 2. Tensors 409, 413, 417, and 425 are passed to 1×1 lateral convolution modules 440, 442, 444, and 446 to produce tensors 441, 443, 445, and 447. Tensor 441 is passed to a 3×3 output convolution module 470, which produces the output tensor P5 471. Tensor 441 is also passed to an upsampler module 450 to produce an upsampled tensor 451. A summing module 460 sums tensors 443 and 451 to produce tensor 461, which is passed to an upsampler module 452 and a 3×3 lateral convolution module 472. Module 472 outputs tensor P4 473. The upsampler module 452 produces the upsampled tensor 453. Summation module 462 sums tensors 445 and 453 to produce tensor 463, which is passed to 3×3 lateral convolution module 474 and upsampler module 454. Module 474 outputs tensor P3 475. Upsampler module 454 outputs upsampled tensor 455. Summation module 464 sums tensors 447 and 455 to produce tensor 465, which is passed to 3×3 lateral convolution module 476. Module 476 outputs tensor P2 477. Upsampler modules 450, 452, and 454 use nearest neighbor interpolation to reduce computational complexity. Tensors 429, 471, 473, 475, and 477 form the output tensor 115 of CNN backbone 400.

[0108] Figure 5This is a schematic block diagram illustrating a feature map quantizer and packer 116 as part of a distributed machine task system 100. Tensors 115 from the CNN backbone 114 are input to a group determiner module 510, a range determiner module 514, and a quantizer module 518. In other words, the quantizer module 518 implements a mapping function or transfer function from floating-point values ​​to integer values. The group determiner module 510 assigns the feature maps (channels) of the input tensor 115 to feature map groups 512 based on predetermined criteria or certain metrics based on the data present in the tensor 115. Feature map groups 512 can span tensors across different layers or can be limited to a single layer. Feature map groups 512 are passed to the range determiner module 514 and output as part of metadata 125. The range determiner module 514 determines, for each group, the quantization range indicating the maximum amplitude value present in the feature maps belonging to the corresponding group, resulting in a quantization range 516. The range determiner module 514 can determine a new quantization range for each frame, or it can determine a new quantization range less frequently, for example, only for intra-frame images.

[0109] Bitstream 121 includes a “qr_update” flag in its metadata indicating whether the quantization range has been updated (see Appendix A). A single quantization range can be used to represent the maximum amplitude of any value prior to quantization within the feature map of the group to which that quantization range belongs. In another arrangement, separate quantization ranges are used for the maximum positive value within the feature map group and the maximum negative value within the feature map, resulting in an asymmetric quantization range with two values ​​for each group.

[0110] Tensor 115 typically has 32-bit floating-point precision, so the various quantization ranges are also floating-point values. Other floating-point precisions are possible, such as 16-bit and 8-bit, and various bit allocations for the exponent and fractional part of the floating-point value are also possible.

[0111] The quantization range 516 is passed to the quantizer module 518 and output as part of the metadata 125. The quantizer module 518 quantizes the individual feature maps into sample values ​​in two stages. First, the quantization range of the feature map group to which the feature map belongs is used to normalize the feature map values ​​to obtain values ​​in the range [-1, 1]. Second, the normalized feature map values ​​are scaled to a sample range corresponding to the bit depth of the video encoder 120. For 10-bit operations, the normalized feature map is multiplied by the feature map group 512, then the offset of the feature map group 512 is added, and the sum is converted to integer precision and output as an integerized feature map 520. The multiplication and addition operations result in at least one value at the minimum or maximum allowed sample value in the feature map of the given feature map group (i.e., zero (0) or one thousand and twenty-three (1023) for 10-bit video). To provide some flexibility for overshoot that may occur at the output of the video decoder 144, the multiplication factor applied to the normalized feature map can be reduced compared to the maximum possible multiplication factor that can be used without introducing clipping. For regular video represented in the YCbCr color space, a “video range” is defined as sixteen (16) to two hundred and thirty-five (235) for 8-bit video data or sixty-four (64) to nine hundred and forty (940) for 10-bit video data. Therefore, the multiplication factor can be reduced to 7 / 8 of the full value, resulting in a sample range similar to the sample range seen in the video range of YCbCr video data. The resulting multiplication factor will be 7 / 8 × (1 << (bit_depth - 1)). The offset factor used to shift negative tensor values ​​into the positive range is left at the middle point, i.e., 1 << (bit_depth - 1), corresponding to the default predictor of the unavailable reference sample for intra-frame prediction, such as the reference. Figure 6 and Figure 7 The process is described above. If the integer value generated by quantization exceeds the range allowed by the bit depth of the samples in the frame, clipping is applied to ensure that the integer value remains within the bit depth of the samples in the frame. The integerized feature map 520 is passed to the packer module 522, which generates a packed feature map frame 117, which includes the individual feature maps of the integerized feature map 520 arranged according to the packing format. (See reference...) Figures 11 to 13 The packet format is further described. The resulting packet feature frame 117 is passed to the video encoder 120 via multiplexer 118.

[0112] Figure 6 This is a schematic block diagram showing the functional modules of the video encoder 120. Figure 7 This is a schematic block diagram illustrating the functional modules of the video decoder 144. Typically, data is transferred between the functional modules within the video encoder 120 and the video decoder 144 in groups of samples or coefficients (such as the partitioning of blocks into fixed-size sub-blocks) or as arrays. Figure 2Aand Figure 2B As shown, the video encoder 120 and video decoder 144 can be implemented using a general-purpose computer system 200, wherein various functional modules can be implemented using dedicated hardware within the computer system 200 and executable software within the computer system 200 (such as one or more software code modules of a software application program 233 residing on the hard disk drive 205 and executed by the processor 205). Alternatively, the video encoder 120 and video decoder 144 can be implemented using a combination of dedicated hardware and software executable within the computer system 200. Alternatively, the video encoder 120, video decoder 144, and the method can be implemented in dedicated hardware such as one or more integrated circuits that perform the functions or sub-functions of the method. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), a dedicated standard product (ASSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or one or more microprocessors and associated memory. Specifically, the video encoder 120 includes modules 610-690, and the video decoder 144 includes modules 720-796, wherein each of these modules can be implemented as one or more software code modules of the software application 233.

[0113] although Figure 6The video encoder 120 is an example of a Universal Video Coding (VVC) video coding pipeline, but other video codecs can also be used for the processing stages described herein. The video encoder 120 receives frame data 119, such as a sequence of frames, each frame including one or more color channels. The frame data 119 can be any chroma format and bit depth supported by the used profile, such as 4:0:0 or 4:2:0 of ​​the VVC standard's "Main 10" profile, with a sample precision of eight (8) to ten (10) bits. The block partitioner 610 first divides the frame data 119 into CTUs, typically square in shape and configured to use a specific size of CTU. The maximum effective size of the CTU can be, for example, 32×32, 64×64, or 128×128 luma samples, configured by the "sps_log2_ctu_size_minus5" syntax element present in the "Sequence Parameter Set". The CTU size also provides the maximum CU size, as a CTU without further splitting will contain one CU. Block partitioner 610 further divides each CTU into one or more CBs based on the luma coding tree and chroma coding tree. The luma channel can also be referred to as the primary color channel. The individual chroma channels can also be referred to as secondary color channels. CBs have various sizes and can include both square and non-square aspect ratios. However, in the VVC standard, CBs, CUs, PUs, and TUs always have side lengths that are powers of two. Therefore, based on the luma coding tree and chroma coding tree of the CTU, the block partitioner 610 outputs the current CB, represented as 612 (progressing based on iterations on one or more blocks of the CTU). CUs or CBs are generated by recursively partitioning the CTU using quadtree partitioning (split into four sub-regions arranged as 2×2 subdivisions of the parent region), binary tree partitioning (split horizontally or vertically into two equal-sized sub-regions of the parent region), and ternary tree partitioning (split horizontally or vertically into three sub-regions with an area ratio of 1:2:1).

[0114] Although operations are typically described on a per-CTU basis, the video encoder 120 and video decoder 144 can operate on smaller sized regions to reduce memory consumption. For example, individual CTUs can be partitioned into smaller regions (known as “Virtual Pipeline Data Units” (VPDUs) of size 64×64). VPDUs form a data granularity more suitable for pipelined processing in the hardware architecture; the reduced memory footprint compared to operating on the entire CTU reduces silicon area, thereby reducing cost. When the CTU size is 128×128, the allowed coding tree is appropriately limited to ensure that processing of a VPDU is fully completed before moving to the next VPDU. For example, at the root node of the coding tree for a 128×128 CTU, ternary tree splitting is prohibited because the resulting CUs (such as 32×128 / 128×32 or further decompositions thereof) cannot be processed in the manner required to progress from a 64×64 region to subsequent 64×64 regions. When the CTU size is 64×64, regardless of the encoding tree selected by the encoder, processing must complete one 64×64 zone (i.e., from one CTU to the next CTU) before moving on to the next 64×64 zone.

[0115] The CTUs obtained from the first segment of frame data 113 can be scanned in raster scan order, and these CTUs can be grouped into one or more "slices". A slice can be an "intra-frame" (or "I") slice. An intra-frame slice (I slice) indicates that each CU in the slice is intra-predicted. Typically, the first picture in a coding layer video sequence (CLVS) contains only I slices and is referred to as an "intra-frame picture". CLVS can contain periodic intra-frame pictures that form "random access points" (i.e., intermediate frames in the video sequence from which decoding can begin). Alternatively, slices can be single-predicted or double-predicted ("P" or "B" slices, respectively), indicating the additional availability of single and double prediction within the slice.

[0116] When using a chroma format other than 4:0:0, within an I-slice, the coding trees of individual CTUs can diverge into two separate coding trees below the 64×64 level, one for luma and the other for chroma. Using separate trees allows for different block structures for luma and chroma within the luma 64×64 region of the CTU. For example, large chroma CBs can be juxtaposed with many smaller luma CBs, and vice versa. In P- or B-slices, a single coding tree of the CTU defines a common block structure for both luma and chroma. The resulting blocks from a single tree can be intra-predicted or inter-predicted.

[0117] For each CTU, the video encoder 120 operates in two phases. In the first phase (referred to as the “search” phase), the block partitioner 610 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has an associated “candidate” CB. The first phase involves testing various candidate CBs to select the CB that provides relatively high compression efficiency and relatively low distortion. The testing typically involves Lagrangian optimization, thereby evaluating the candidate CBs based on a weighted combination of rate (i.e., coding cost) and distortion (i.e., error relative to the input frame data 119). The “best” candidate CB (i.e., the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding into the bitstream 121. Included in the evaluation of the candidate CBs are options for using the CB for a given region, or for further splitting the region according to various splitting options and encoding the individual smaller resulting regions with further CBs, or even further splitting the region. Thus, both the coding tree and the CB are selected in the search phase.

[0118] Video encoder 120 generates prediction blocks (PBs) indicated by arrow 620 for each CB (e.g., CB 612). PB 620 is a prediction of the content of the associated CB 612. Subtractor module 622 generates a difference (or “residual”, which refers to the difference in the spatial domain) between PB 620 and CB 612, represented as 624. Difference 624 is the block size difference between the corresponding samples in PB 620 and CB 612. Difference 624 is transformed, quantized, and represented as a transform block (TB) indicated by arrow 636. PB 620 and the associated TB 636 are typically selected from one of several possible candidate CBs, for example, based on the evaluated cost or distortion.

[0119] The candidate coded block (CB) is a CB obtained from the video encoder 120 for one of the prediction modes available for the associated PB and the resulting residual. When combined with the prediction PB in the video encoder 120, the TB 636 reduces the difference between the decoded CB and the original CB 612 at the cost of additional signal notification in the bitstream.

[0120] Therefore, each candidate coded block (CB) (i.e., the prediction block (PB) combined with the transform block (TB)) has an associated coding cost (or "rate") and an associated difference (or "distortion"). The distortion of the CB is typically estimated as the difference in sample values, such as the sum of absolute differences (SAD), the sum of squared differences (SSD), or the Hadamard transform applied to the difference. The mode selector 686 uses the difference 624 to determine the estimate obtained from each candidate PB to determine the prediction mode 687. The prediction mode 687 indicates the decision to use a specific prediction mode (e.g., intra-frame prediction or inter-frame prediction) for the current CB. The coding cost associated with each candidate prediction mode and the corresponding residual coding can be estimated at a significantly lower cost compared to entropy coding of the residual. Therefore, even in a real-time video encoder, multiple candidate modes can be evaluated to determine the optimal mode in a rate-distortion sense.

[0121] Determining the optimal mode based on rate distortion is typically achieved using a variation of Lagrange optimization.

[0122] Lagrange or similar optimization processes can be employed to select both the optimal partitioning of the CTU to the CB (using block partitioner 610) and the optimal prediction mode from multiple possibilities. By applying Lagrange optimization of candidate modes in the mode selector module 686, the intra-prediction mode with the lowest cost metric is selected as the "optimal" mode. The lowest cost mode includes the selected quadratic transform index 688, which is also encoded into bitstream 121 by entropy encoder 638.

[0123] In the second phase of the operation of the video encoder 120 (referred to as the "encoding" phase), the determined coding trees for each CTU are iterated within the video encoder 120. For CTUs using individual trees, the luma coding tree is encoded first for each 64×64 luma zone of the CTU, followed by the chroma coding tree. Within the luma coding tree, only the luma CB is encoded, and within the chroma coding tree, only the chroma CB is encoded. For CTUs using shared trees, a single tree describes the CU (i.e., luma CB and chroma CB) according to the common block structure of the shared tree.

[0124] The entropy encoder 638 supports bitwise encoding of syntactic elements using variable-length and fixed-length codewords, as well as arithmetic encoding modes for syntactic elements. Sections of the bitstream, such as "parameter sets" (e.g., Sequence Parameter Sets (SPS) and Picture Parameter Sets (PPS)), use a combination of fixed-length and variable-length codewords. Slices (also called contiguous sections) have a slice header using variable-length encoding, followed by slice data using arithmetic encoding. The slice header defines parameters specific to the current slice, such as slice-level quantization parameter offsets. The slice data includes the syntactic elements of the individual CTUs within the slice. The use of variable-length and arithmetic encoding requires sequential parsing within the various sections of the bitstream. These sections can be described using starter codes to form "Network Abstraction Layer Units" or "NAL Units." Arithmetic encoding is supported using context-adaptive binary arithmetic encoding processing.

[0125] Arithmetically encoded syntactic elements consist of a sequence of one or more "bins" (binary files). Like bits, bins have values ​​of "0" or "1". However, bins are not encoded as discrete bits in bitstream 121. A bin has an associated prediction (or "possible" or "maximum probability") value and an associated probability (known as "context"). The "maximum probability symbol" (MPS) is encoded when the actual bin to be encoded matches the predicted value. Encoding the maximum probability symbol is relatively inexpensive in terms of bit consumption in bitstream 121, including a cost of less than one discrete bit in total. The "minimum probability symbol" (LPS) is encoded when the actual bin to be encoded does not match the possible value. Encoding the minimum probability symbol is relatively expensive in terms of bit consumption. Bin encoding techniques enable efficient encoding of bins that can skew the probabilities of "0" vs. "1". For syntactic elements with two possible values ​​(i.e., "flag"), a single bin is sufficient. For syntactic elements with many possible values, a sequence of bins is required.

[0126] The presence of later bins in a sequence can be determined based on the value of an earlier bin. Furthermore, each bin can be associated with more than one context. The selection of a specific context can depend on earlier bins in a syntactic element, the bin values ​​of adjacent syntactic elements (i.e., adjacent syntactic elements from adjacent blocks), etc. Each time a context-encoded bin is encoded, the context selected for that bin (if it exists) is updated to reflect the new bin value. Therefore, the binary arithmetic coding scheme is called adaptive.

[0127] The entropy encoder 638 also supports bins lacking context (referred to as "bypass bins"). Bypass bins are encoded using an equiprobable distribution between "0" and "1". Therefore, each bin has a one-bit encoding cost in the bitstream 121. The absence of context saves memory and reduces complexity, thus using bypass bins with an unskewed distribution of values ​​for a particular bin. An example of an entropy encoder employing context and adaptation is known in the art as CABAC (Context Adaptive Binary Arithmetic Encoder), and many variations of this encoder have been adopted in video coding.

[0128] The entropy encoder 638 encodes the quantization parameter 692 using a combination of context-coded and bypass-coded bins, and encodes the LFNST index 388 if applicable to the current CB. The quantization parameter 692 is encoded using an "incremental QP". The incremental QP is signaled at most once in each region known as a "quantization group". The quantization parameter 692 is applied to the residual coefficients of the luma CB. The adjusted quantization parameter is applied to the residual coefficients of the juxtaposed chroma CB. The adjusted quantization parameter may include mapping from the luma quantization parameter 692 according to a mapping table and a CU-level offset selected from an offset list. The secondary transform index 688 is signaled when the residuals associated with the transform block include valid residual coefficients only in the positions of those coefficients transformed into principal coefficients by applying a secondary transform.

[0129] Multiplexer module 684 outputs PB 620 from intra-prediction module 664 based on the determined optimal intra-prediction mode selected from the test prediction modes of each candidate CB. Candidate prediction modes do not need to include every conceivable prediction mode supported by video encoder 120. Intra-prediction is divided into three types: first, “DC intra-prediction,” which involves filling PB with a single value representing the average of nearby reconstructed samples; second, “planar intra-prediction,” which involves filling PB with samples based on the plane, utilizing DC offsets, and vertical and horizontal gradients derived from nearby reconstructed neighboring samples. Nearby reconstructed samples typically include a row of reconstructed samples above the current PB, extending to the right of the PB, and a column of reconstructed samples to the left of the current PB, extending downwards beyond the PB; and third, “angular intra-prediction,” which involves filling PB with reconstructed neighboring samples filtered and propagated across the PB in a specific direction (or “angle”). In VVC, sixty-five (65) angles are supported, where rectangular blocks can utilize additional angles unavailable to square blocks to produce a total of eighty-seven (87) angles.

[0130] A fourth type of intra-frame prediction can be used for chroma blocks, thereby generating blocks from juxtaposed luma reconstruction samples according to a "Cross-Component Linear Model" (CCLM) mode. Three different CCLM modes are available, each using a different model derived from adjacent luma and chroma samples. The derived model is used to generate sample blocks for the chroma block from the juxtaposed luma samples. Intra-frame prediction of luma blocks can be performed using matrix multiplication of reference samples, using a matrix selected from a predefined set of matrices. This matrix intra-frame prediction (MIP) achieves gain by using matrices trained on a large set of video data, where the matrices represent the relationship between reference samples and prediction blocks that is not easily captured in angular, planar, or DC intra-frame prediction modes.

[0131] Module 664 can also generate prediction units by copying blocks from the vicinity of the current frame using an "intra-block copy" (IBC) method. The location of the reference block is constrained to an area equivalent to a CTU (which is divided into 64×64 areas known as VPDUs), covering the processed VPDUs of the current CTU and the VPDUs of the previous CTU, up to a region limit of one CTU. This area is known as the "IBC virtual buffer" and limits the IBC reference area, thereby limiting the required storage space. The IBC buffer is filled with reconstructed samples 654 (i.e., before in-loop filtering), thus requiring a separate buffer for frame buffer 672.

[0132] The residuals of predicted blocks when encoding feature map data differ from those seen for natural video. This natural video is typically captured by imaging sensors or screen content, as commonly seen in operating system user interfaces, etc. Feature map residuals tend to contain a lot of detail, which, relative to the significantly low-frequency coefficients of various transforms, is more suitable for transform-skip coding. Experiments show that feature map residuals have sufficient local similarity to benefit from transform coding. However, the distribution of feature map residual coefficients does not cluster towards the DC (top-left) coefficients of the transform blocks. In other words, there is sufficient correlation for transforms indicating gain when encoding feature map data, and this is also true when using intra-block replication to generate predicted blocks of feature map data. Therefore, when encoding feature map data, Hadamard cost estimation can be used to evaluate the residuals generated by candidate block vectors used for intra-block replication, rather than relying solely on SAD or SSD cost estimation. SAD or SSD cost estimation tends to select block vectors with residuals more suitable for transform-skip coding and may miss block vectors with residuals that will be tightly encoded using transforms. When encoding feature map data, the VVC standard’s Multiple Transform Selection (MTS) tool can be used, which allows combinations of DST-7 and DCT-8 transforms, in addition to DCT-2 transform, to be used horizontally and vertically for residual coding.

[0133] Intra-predictive luminance coding blocks can be partitioned vertically or horizontally into sets of prediction blocks of equal size, each block having a minimum region of sixteen (16) luminance samples. This intra-fractional sub-partitioning (ISP) method allows individual transform blocks to contribute to the generation of prediction blocks from one sub-partition to the next from the luminance coding block, thereby improving compression efficiency.

[0134] When previously reconstructed adjacent samples are unavailable, such as at the edge of a frame, a default halftone value of half the sample range is used. For example, for 10-bit video, a value of 512 (512) is used. Since no previous samples are available for the CB located in the upper left position of the frame, the angular and planar intra-frame prediction modes produce the same output as the DC prediction mode (i.e., a flat plane of samples with halftone values ​​as amplitude).

[0135] For inter-frame prediction, the motion compensation module 680 generates a prediction block 682 using samples from one or two frames preceding the current frame in the coded sequence of the bitstream, and the multiplexer module 684 outputs this prediction block 682 as PB620. Furthermore, for inter-frame prediction, a single coding tree is typically used for both the luma and chroma channels. The order of the coded frames in the bitstream may differ from the order of the frames at the time of capture or display. When one frame is used for prediction, the block is called "single prediction" and has one associated motion vector. When two frames are used for prediction, the block is called "double prediction" and has two associated motion vectors. For P-slices, each CU can be intra-predicted or single-predicted. For B-slices, each CU can be intra-predicted, single-predicted, or double-predicted.

[0136] Frames are typically encoded using a "picture group" structure to achieve a temporal hierarchy. A frame can be divided into multiple slices, each slice encoding a portion of the frame. This temporal hierarchy allows frames to reference preceding and following images in the order they are displayed. Images are encoded in the necessary order to ensure the relevance for decoding each frame. Affine inter-frame prediction modes are available where, instead of using one or two motion vectors to select and filter the reference sample block for a prediction unit, the prediction unit is divided into multiple smaller blocks, and a motion field is generated such that each smaller block has a different motion vector. The motion field uses the motion vectors of nearby points of the prediction unit as "control points." Affine prediction allows encoding motions other than translations with less need for depth-splitting coding trees. VVC's dual prediction mode geometrically blends two reference blocks along a selected axis and signals the angle and offset relative to the block's center. This geometric partitioning mode ("GPM") allows the use of larger coding units along the boundary between two objects, with the geometry of the coding unit's boundary serving as the angle and center offset. Motion vector differences can be encoded as direction (up / down / left / right) and distance (supporting a set of distances that are powers of 2), instead of using Cartesian (x, y) offsets. Motion vector predictors are obtained from neighboring blocks ("merge mode") as if no offset had been applied. The current block will share the same motion vectors with the selected neighboring blocks.

[0137] Samples are selected based on motion vector 678 and a reference image index. Motion vector 678 and the reference image index are applied to all color channels, thus describing inter-frame prediction primarily in terms of operations on the PU rather than the PB. A single coding tree describes the decomposition of each CTU into one or more inter-frame prediction blocks. Inter-frame prediction methods can vary in the number and precision of motion parameters. Motion parameters typically include a reference frame index, indicating which reference frame(s) from the reference frame list will be used, along with spatial translations of the individual reference frames, but can include more frames, specific frames, or complex affine parameters (such as scaling and rotation). Additionally, predetermined motion refinement processes can be applied to generate dense motion estimates based on the reference sample blocks.

[0138] PB 620 has been identified and selected, and at subtractor 622, PB 620 is subtracted from the original sample block to obtain the residual (denoted as 624) with the lowest encoding cost, which is then subjected to lossy compression. The lossy compression process includes transform, quantization, and entropy coding steps. Forward master transform module 626 applies a forward transform to the difference 624, converting it from the spatial domain to the frequency domain, and produces master transform coefficients indicated by arrow 628. The maximum master transform size in one dimension is a 32-point DCT-2 or 64-point DCT-2 transform configured by the "sps_max_luma_transform_size_64_flag" in the sequence parameter set. If the CB being encoded is larger than the maximum supported master transform size represented as the block size (e.g., 64×64 or 32×32), the master transform 626 is applied in a tiling manner to transform all samples of the difference 624. In the case of using a non-square CB, tiling is also performed using the maximum available transform size in each dimension of the CB. For example, when using a maximum transform size of 32 (32), a 64×16 CB uses two 32×16 primary transforms arranged in a tiling manner. When the size of the CB is larger than the maximum supported transform size, the CB is filled with TBs arranged in a tiling manner. For example, a 128×128 CB with a maximum transform size of 64-pt is filled with four 64×64 TBs arranged in a 2×2 manner. A 64×128 CB with a maximum transform size of 32-pt is filled with eight 32×32 TBs arranged in a 2×4 manner.

[0139] The application of transform 626 yields multiple TBs for the CB. When the various applications of the transform operate on TBs with a difference 624 greater than 32×32 (e.g., 64×64), all resulting master transform coefficients 628 outside the top-left 32×32 region of the TB are set to zero (i.e., discarded). The remaining master transform coefficients 628 are passed to quantizer module 634. The master transform coefficients 628 are quantized according to quantization parameter 692 associated with the CB to produce master transform coefficients 632. In addition to quantization parameter 692, quantizer module 634 may also apply a "scaling list" to allow non-uniform quantization within the TB by further scaling the residual coefficients according to their spatial location within the TB. Quantization parameter 692 may differ for the luma CB and the individual chroma CBs. Master transform coefficients 632 are passed to forward quadratic transform module 630 to produce transform coefficients indicated by arrow 636 by performing an inseparable quadratic transform (NSST) operation or bypassing the quadratic transform. The forward master transform is typically separable, thus transforming the row sets and then the column sets of each TB. For luminance TBs with a width and height not exceeding 16 samples, the forward master transform module 626 uses a Type II Discrete Cosine Transform (DCT-2) in both the horizontal and vertical directions, or a transform bypass in both the horizontal and vertical directions, or a combination of a Type VII Discrete Sine Transform (DST-7) and a Type VIII Discrete Cosine Transform (DCT-8) in either the horizontal or vertical direction. In the VVC standard, the use of a combination of DST-7 and DCT-8 is referred to as the "Multiple Transform Selection Set" (MTS).

[0140] The forward quadratic transform of module 630 is typically an inseparable transform, applied only to the residuals of the intra-frame prediction CU, and can, however, be bypassed. The forward quadratic transform operates on sixteen (16) samples (arranged in the top-left 4×4 sub-block of the main transform coefficients 628) or forty-eight (48) samples (arranged in three 4×4 sub-blocks of the top-left 8×8 coefficients of the main transform coefficients 628) to produce a set of quadratic transform coefficients. The set of quadratic transform coefficients can be smaller than the set of main transform coefficients from which they are derived. Because the quadratic transform is applied only to the set of coefficients that are adjacent to each other and include DC coefficients, it is called the “low-frequency inseparable quadratic transform” (LFNST). Furthermore, when the LFNST is applied, all residual coefficients in the TB are zero in both the main transform domain and the quadratic transform domain.

[0141] The quantization parameter 692 is constant for a given TB, and thus a uniform scaling of the residual coefficients generated in the main transform domain for the TB is obtained. The quantization parameter 692 can be periodically varied with a signaled "incremental quantization parameter". For a CU contained within a given region (called a "quantization group"), the incremental quantization parameter (incremental QP) is signaled once. If the CU is larger than the quantization group size, the incremental QP is signaled once using one of the TBs of the CU. That is, the incremental QP is signaled once for the first quantization group of the CU by the entropy encoder 638, but not for any subsequent quantization groups of the CU. Non-uniform scaling can also be achieved by applying a "quantization matrix", whereby the scaling factor applied to each residual coefficient is derived from a combination of the quantization parameter 692 and the corresponding entries in the scaling matrix. The scaling matrix can have a size smaller than the TB, and when applied to the TB, the nearest neighbor method is used to provide scaling values ​​for each residual coefficient from a scaling matrix smaller than the TB size. The residual coefficients 636 are fed to the entropy encoder 638 for encoding in the bit stream 121. Typically, according to the scan pattern, the residual coefficients of each TB with at least one valid residual coefficient are scanned from the TU to produce an ordered list of values. The scan pattern typically scans the TB as a sequence of 4×4 “sub-blocks,” thus providing regular scan operations at a granularity of 4×4 sets of residual coefficients, where the arrangement of the sub-blocks depends on the size of the TB. Scanning within each sub-block and progress from one sub-block to the next typically follow a reverse diagonal scan pattern. Additionally, the quantization parameter 692 is encoded into bitstream 121 using incremental QP syntax elements, and the quadratic transform index 688 is encoded into bitstream 121.

[0142] As described above, the video encoder 120 needs access to the frame representation corresponding to the decoded frame representation seen in the video decoder 144. Therefore, the residual coefficients 636 are passed through the inverse quadratic transform module 644, which operates according to the quadratic transform index 688 to produce intermediate inverse transform coefficients, indicated by arrow 642. The intermediate inverse transform coefficients 642 are inversely quantized by the dequantizer module 640 according to the quantization parameter 692 to produce inverse transform coefficients, indicated by arrow 646. The dequantizer module 640 can also perform inverse non-uniform scaling of the residual coefficients using a scaling list, corresponding to the forward scaling performed in the quantizer module 634. The inverse transform coefficients 646 are passed to the inverse master transform module 648 to produce residual samples of the TU (indicated by arrow 650). The inverse master transform module 648 applies the DCT-2 transform horizontally and vertically, constrained by the maximum available transform size described with reference to the forward master transform module 626. The type of inverse transformation performed by inverse quadratic transformation module 644 corresponds to the type of forward transformation performed by forward quadratic transformation module 630. The type of inverse transformation performed by inverse master transformation module 648 corresponds to the type of master transformation performed by master transformation module 626. Summation module 652 adds residual sample 650 and PU 620 to generate reconstructed sample of CU (indicated by arrow 654).

[0143] Reconstructed sample 654 is passed to reference sample cache 656 and in-loop filter module 668. Reference sample cache 656, typically implemented using static RAM on an ASIC (to avoid expensive off-chip memory access), provides the minimum sample storage required to satisfy the dependencies used to generate intra-frame PBs for subsequent CUs in a frame. Minimal dependencies typically include a “line buffer” of samples below a row of CTUs for use by the next row of CTUs, as well as a column buffer (whose extent is set by the height of the CTU). Reference sample cache 656 feeds reference samples (indicated by arrow 658) to reference sample filter 660. Sample filter 660 applies a smoothing operation to produce filtered reference samples (indicated by arrow 662). Filtered reference samples 662 are used by intra-frame prediction module 664 to generate intra-frame prediction blocks of samples, indicated by arrow 666. For each candidate intra-frame prediction mode, intra-frame prediction module 664 generates sample blocks (i.e., 666). Sample blocks 666 are generated by module 664 using techniques such as DC, planar, or angular intra-frame prediction. Alternatively, a matrix multiplication method can be used to generate sample block 666, where adjacent reference samples are used as input, and the video encoder 120 selects a matrix from the matrix set, using an index to signal in the bit stream 120 which matrix in the matrix set will be used by the video decoder 144.

[0144] The in-loop filter module 668 applies several filtering stages to the reconstructed sample 654. The filtering stages include a "deblocking filter" (DBF), which applies smoothing aligned with the CU boundaries to reduce artifacts caused by discontinuities. The deblocking filter smooths block edges where coding artifacts caused by misaligned boundaries along the block boundaries due to the transform basis function are visible; these artifacts are more pronounced at higher values ​​of the quantization parameter 692. At lower values ​​of the quantization parameter 692, the filtering strength of the deblocking filter decreases. Another filtering stage present in the in-loop filter module 668 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. Another available filtering stage in the in-loop filter module 668 is a "sample adaptive offset" (SAO) filter. The SAO filter operates by first classifying the reconstructed sample into one or more categories and then applying an offset at the sample level according to the assigned category.

[0145] The filtered sample, indicated by arrow 670, is output from the in-loop filter module 668. The filtered sample 670 is stored in the frame buffer 672. The frame buffer 672 typically has a capacity to store several (e.g., up to sixteen (16)) images and is therefore stored in memory 206. Due to the large memory consumption required, on-chip memory is typically not used to store the frame buffer 672. Thus, access to the frame buffer 672 is expensive in terms of memory bandwidth. The frame buffer 672 provides reference frames (indicated by arrow 674) to the motion estimation module 676 and the motion compensation module 680.

[0146] Motion estimation module 676 estimates multiple "motion vectors" (denoted as 678), each a Cartesian space offset relative to the current CB position, thus referencing a block of one of the reference frames in frame buffer 672. A filtered block of reference samples (denoted as 682) is generated for each motion vector. The filtered reference samples 682 form further candidate modes for potential selection by mode selector 686. Furthermore, for a given CU, PB 620 can be formed using one reference block ("single prediction") or two reference blocks ("double prediction"). For the selected motion vector, motion compensation module 680 generates PB 620 based on filtering that supports sub-pixel precision in the motion vector. Thus, motion estimation module 676 (which operates on many candidate motion vectors) can perform simplified filtering compared to motion compensation module 680 (which operates only on the selected candidate), achieving reduced computational complexity. When video encoder 120 selects inter-frame prediction for a CU, motion vector 678 is encoded into bitstream 121.

[0147] Although the reference to Universal Video Coding (VVC) explains Figure 6The video encoder 120 can be used, but other video encoding standards or implementations can also utilize the processing stages of modules 610-690. Frame data 119 (and bit stream 121) can also be sourced from memory 206, hard disk drive 210, CD-ROM, Blu-ray disc. TM The frame data 119 can be read from (or written to) memory 206, hard disk drive 210, CD-ROM, Blu-ray disc, or other computer-readable storage media. Additionally, the frame data 119 (and bitstream 121) can be received from (or sent to) an external source, such as a server or RF receiver connected to the communication network 220. The communication network 220 may offer limited bandwidth, necessitating rate control in the video encoder 120 to avoid network saturation when the frame data 119 is difficult to compress. Furthermore, the bitstream 121 can be constructed from one or more slices representing spatial portions (sets of CTUs) of the frame data 119, generated by one or more instances of the video encoder 120, and operated in a coordinated manner under the control of the processor 205.

[0148] exist Figure 7 The video decoder 144 is shown. Although... Figure 7 The video decoder 144 is an example of a Universal Video Coding (VVC) video decoding pipeline, but other video codecs can also be used for the processing stages described herein. Figure 7 As shown, bitstream 143 is input to video decoder 144. Bitstream 143 can be read from memory 206, hard disk drive 210, CD-ROM, Blu-ray disc, or other non-transitory computer-readable storage media. Alternatively, bitstream 143 can be received from an external source, such as a server connected to communication network 220 or an RF receiver. Bitstream 143 contains encoded syntax elements representing the captured frame data to be decoded.

[0149] Bitstream 143 is input to entropy decoder module 720. Entropy decoder module 720 extracts syntactic elements from bitstream 143 by decoding the "bin" sequence and passes the values ​​of the syntactic elements to other modules in video decoder 144. Entropy decoder module 720 uses variable-length and fixed-length decoding to decode SPS, PPS, or slice headers, and uses an arithmetic decoding engine to decode the syntactic elements of the slice data into a sequence of one or more bins. Each bin can use one or more "contexts," where the context describes the probability levels of "one" and "zero" values ​​to be used to encode the bin. When multiple contexts are available for a given bin, a "context modeling" or "context selection" step is performed to select one of the available contexts to decode the bin. The processing of decoding bins forms a sequential feedback loop, so that individual slices can be decoded by a given entropy decoder instance 720 within the slice as a whole. A single (or several) high-performance entropy decoder instances 720 can decode all slices of a frame from bitstream 143, while multiple lower-performance entropy decoder instances 720 can decode slices of a frame from bitstream 143 simultaneously.

[0150] The entropy decoder module 720 applies an arithmetic coding algorithm, such as Context Adaptive Binary Arithmetic Coding (CABAC), to decode syntactic elements from the bitstream 143. The decoded syntactic elements are used to reconstruct parameters within the video decoder 144. These parameters include residual coefficients (indicated by arrow 724), quantization parameters 774, quadratic transform index 770, and mode selection information such as intra-frame prediction modes (indicated by arrow 758). The mode selection information also includes information such as motion vectors and partitioning each CTU into one or more CBs. Typically combined with sample data from previously decoded CBs, the parameters are used to generate a PB.

[0151] The residual coefficients 724 are passed to the inverse quadratic transform module 736, where a quadratic transform is applied according to the quadratic transform index, or no operation is performed (bypass). The inverse quadratic transform module 736 generates reconstructed transform coefficients 732, i.e., main transform coefficients, from the quadratic transform domain coefficients. The reconstructed transform coefficients 732 are input to the dequantizer module 728. The dequantizer module 728 performs inverse quantization (or "scaling") on the residual coefficients 732 (i.e., in the main transform coefficient domain) to create reconstructed intermediate transform coefficients, indicated by arrow 740, based on the quantization parameter 774. The dequantizer module 728 can also apply a scaling matrix to provide non-uniform dequantization within the TB, corresponding to the operation of the dequantizer module 640. If a non-uniform inverse quantization matrix is ​​indicated in the bitstream 143, the video decoder 144 reads the quantization matrix from the bitstream 143 as a sequence of scaling factors and arranges the scaling factors into a matrix. Inverse scaling combines the quantization matrix with the quantization parameter to create the reconstructed intermediate transform coefficients 740.

[0152] The reconstructed transform coefficients 740 are passed to the inverse master transform module 744. Module 744 transforms the coefficients 740 from the frequency domain back to the spatial domain. The inverse master transform module 744 applies the inverse DCT-2 transform horizontally and vertically, constrained by the maximum available transform size as described in the reference forward master transform module 626. The result of the operation of module 744 is a residual sample block indicated by arrow 748. The residual sample block 748 is equal in size to the corresponding CB. The residual sample 748 is fed to the summing module 750.

[0153] At summing module 750, residual sample 748 is added to decoding PB (denoted as 752) to produce a reconstructed sample block indicated by arrow 756. Reconstructed sample 756 is fed to reconstructed sample cache 760 and in-loop filtering module 788. In-loop filtering module 788 produces a reconstructed block of frame samples, denoted as 792. Frame sample 792 is written to frame buffer 796.

[0154] The reconstruction sample cache 760 operates in a manner similar to the reconstruction sample cache 656 of the video encoder 120. The reconstruction sample cache 760 provides storage for reconstructed samples required for intra-frame prediction of subsequent CBs in the absence of memory 206 (e.g., by using data 232, typically on-chip memory, instead). The reference sample, indicated by arrow 764, is obtained from the reconstruction sample cache 760 and supplied to the reference sample filter 768 to produce the filtered reference sample, indicated by arrow 772. The filtered reference sample 772 is supplied to the intra-frame prediction module 776. Module 776 generates blocks of intra-frame prediction samples, indicated by arrow 780, based on the intra-frame prediction mode parameters 758 signaled in bitstream 133 and decoded by the entropy decoder 720. The intra-frame prediction module 776 supports the modes of module 664, including IBC and MIP. Sample blocks 780 are generated using modes such as DC, planar, or angular intra-frame prediction.

[0155] When the prediction mode of a color block (CB) is indicated to use intra-prediction in bitstream 143, intra-prediction samples 780 are used to form decoded blocks (PBs) 752 via multiplexer module 784. Intra-prediction produces a prediction block (PB) of samples, where the PB is a block in a color component derived using "neighboring samples" in the same color component. Neighboring samples are those adjacent to the current block and have already been reconstructed because they are earlier in the block decoding order. In the case of juxtaposition of luma and chroma blocks, luma and chroma blocks can use different intra-prediction modes. However, two chroma CBs share the same intra-prediction mode.

[0156] When the prediction mode of CB is indicated as inter-frame prediction in bitstream 143, motion compensation module 734 generates a block of inter-frame prediction samples, represented as 738. The block 738 of inter-frame prediction samples is generated by selecting and filtering sample blocks 798 from frame buffer 796 using motion vectors decoded from bitstream 143 by entropy decoder 720 and a reference frame index. Sample blocks 798 are obtained from previously decoded frames stored in frame buffer 796. For dual prediction, two sample blocks are generated and blended together to produce samples for decoding PB 752. Frame buffer 796 is filled with filtered block data 792 from in-loop filtering module 788. Similar to in-loop filtering module 668 of video encoder 120, in-loop filtering module 788 applies any of the DBF, ALF, and SAO filtering operations. Generally, motion vectors are applied to both the luma and chroma channels, although the filtering processes used for subsample interpolation in the luma and chroma channels differ.

[0157] Figure 6 and Figure 7 The following module, not shown, is used to preprocess the video before encoding and postprocess it after decoding to shift sample values, thereby achieving a more uniform use of the range of sample values ​​within each chroma channel. A multi-segment linear model is derived in the video encoder 120 and signaled in the bitstream for use by the video decoder 144 to undo the sample shift. This Linear Model Chroma Scaling (LMCS) tool provides compression benefits for specific color spaces and content that exhibit some inhomogeneity in the use of their sample spaces, particularly the utilization of a limited range, which could lead to a higher quality loss from quantization applications.

[0158] Figure 8 This is a schematic block diagram illustrating a feature map inverse quantizer and unpacker 148 as part of a distributed machine task system 100. A decoded frame 147 is input to an unpacking module 810, where feature maps are extracted from each frame according to the packet format to generate an unpacked feature map 812. The unpacked feature map 812 includes sample values ​​as present in the decoded frame 147. (See reference...) Figures 11 to 13The packing format is further described. Based on the feature map set 820 obtained from the decoded metadata 155, the set of feature maps in the unpacked feature map 812 is assigned to groups such that each feature map belongs to a group, and one or more groups are indicated in the feature map set 820. The inverse quantizer 814 then performs scaling to convert the integer sample values ​​present in the unpacked feature map 812 into floating-point values ​​present in the tensor 149. Scaling uses the quantization range of a set of feature maps. The quantization range is obtained from the quantization range 822, which is extracted from the decoded metadata 155. The quantization range specifies the maximum amplitude of any floating-point value seen in the feature map belonging to the corresponding group. The inverse quantizer 814 normalizes the samples from the feature map 812 in each group to a range centered at zero and reaching 1 or -1, based on the sign of the maximum amplitude value found to be positive or negative. In the rare case where positive and negative values ​​have equal maximum amplitudes, a range of [-1, 1] is observed. Then, the normalized samples of a set of feature maps are multiplied (scaled) by the quantization range of that set of feature maps.

[0159] Once all groups of feature maps have been scaled, the results are output as intermediate data in the form of Tensor 149. For example, when the CNN backbone 114 includes an FPN, Tensor 149 can contain multiple tensors, each with a different spatial resolution. Besides using zero-centered linear symmetric quantization, other quantization methods are possible. For example, an asymmetric method can be used, in which the positive and negative quantization ranges are signaled for each group of feature maps. The positive and negative quantization ranges map the range of floating-point values ​​utilized by the group of feature maps to the full sample range provided by the bit depth of the samples. This results in asymmetric quantization because the midpoint of the sample range is no longer guaranteed to correspond to a zero floating-point value. The “quant_type” syntax element in SEI message 1413 selects the quantization method, and this syntax element is described with reference to Appendix A.

[0160] Although the quantization range of a given set of feature maps is derived from the values ​​within that set of feature maps, the quantization range needs to maintain the same data type as the values ​​within that set of feature maps. A coarser floating-point precision can be used, and rounding can be applied so that the range is not reduced when represented back in the original floating-point format (e.g., 32-bit IEEE 754 format). For example, in step 1550, coarser floating-point precision can be used in conjunction with rounding up. Rounding up can be achieved by adding a constant value ε to the quantization range qr to produce an adjusted quantization range qr. adjust To achieve this, such that ε = 2 floor(log2(qr)) / 2 fract_prec Where `frac_prec` is the number of decimal places to retain, and the "floor" operator rounds to the next more negative integer. Then, `qr`... adjustThe leftmost bit of the fractional part of fract_prec can be used and encoded into the SEI message; the remaining bits are truncated, and qr adjust The value will never be less than qr. The precision of the quantization range in terms of the bits allocated to the fractional part is selected using the “qr_fraction_precision” syntax element described in Appendix A. Setting qr_fraction_precision(fract_prec) to 5 (v) allows for precise setting of the quantization range, increasing the precision by approximately 3% in the worst case compared to the fractional precision of the original floating-point value, i.e., before reducing the fractional precision to five bits. To produce the mantissa of the quantization range, a leading “1” is appended to the fractional part (i.e., the quantization range cannot be a “non-normal” value). Since the quantization range is always positive, it is not necessary to encode the sign bit for each quantization range. The quantization range can be greater than or less than one, therefore a sign bit is required for the quantization range exponent. In the arrangement of System 100, quantization ranges less than 1.0 are not allowed, and the quantization exponent sign bit can be omitted from SEI message 1413. When the quantization exponent sign bit is not encoded, quantization ranges less than 1.0 are clipped to a value of 1.0 in the quantization range determiner module 514.

[0161] Although the operations of the inverse quantizer module 814 and the quantizer module 518 are referred to as "quantization," the operations of modules 518 and 814 differ from the quantization operations of the video encoder 120 and the video decoder 144, which involve the use of quantization parameters. Furthermore, the operations of modules 518 and 814 can be viewed as a form of tone mapping operation, which involves a conversion between the floating-point domain of a tensor and the sample domain of a frame. While scaling (i.e., the quantization range via the respective sets of feature maps) exists to utilize a wide range of sample value spaces, no quantization parameters are available for modules 518 and 814 to further modify the quantizer step size.

[0162] Figure 9AThis is a schematic block diagram showing the head 150 of a CNN used for object detection. Different networks can replace the CNN head 150 depending on the task to be performed in the destination device 140. The incoming tensor 149 is split into tensors for each layer (i.e., tensors 910, 920, and 934). Tensor 910 is passed to the CBL module 912 to produce tensor 914, which is then passed to the detection module 916 and the upgrade module 922. Bounding boxes 918 in the form of detection tensors are passed to the non-maximum suppression (NMS) module 948 to produce detection results 151. To generate bounding boxes addressing coordinates in the raw video data 113, the original video width and height are scaled before resizing the backbone of the network 114 (see “orig_source_width” and “orig_source_height” as described in decoding from SEI message 1413 and with reference to Appendix A). Upgrade module 922 generates an upgrade tensor 924, which is passed to CBL module 926, which generates tensor 928 as output. Tensor 928 is passed to detection module 930 and upgrade module 936. Detection module 930 generates a detection tensor 932, which is supplied to NMS module 948. Upgrade module 936 is another instance of module 960 and outputs the upgrade tensor 938. Upgrade tensor 938 is passed to CBL module 940, which outputs tensor 942 to detection module 944. CBL modules 912, 926, and 940 each contain a cascade of five CBL modules. Upgrade modules 922 and 936 are as follows: Figure 9B Examples of the upgrader module 960 shown.

[0163] Upgrade module 960 accepts tensor 962 as input, which is passed to CBL module 966 to generate tensor 968. Tensor 968 is passed to upsampler 970 to generate upsampled tensor 972. Cascade module 974 generates tensor 976 by cascading upsampled tensor 972 with input tensor 964. Detection modules 916, 930, and 944 are as follows: Figure 9CAn example of detection module 980 is shown. Detection module 960 receives tensor 982, which is passed to CBL module 984 to produce tensor 986. Tensor 986 is passed to convolution module 988, which implements the detection kernel. The detection kernel is a 1×1 kernel used to produce the output of feature maps at three layers. The detection kernel is 1×1×(B×(5+C)), where B is the number of bounding boxes that a particular unit can predict, typically three (3), and C is the number of classes, which can be eighty (80), making the kernel size two hundred and fifty-five (255) detection attributes (i.e., tensor 990). The constant “5” represents four bounding box attributes (box center x, y and size scale x, y) and one object confidence level (“objectivity”). The result of the detection kernel has the same spatial dimensions as the input feature map, but the depth of the output corresponds to the detection attributes. The detection kernel is applied to various layers, typically three layers, which yields a large number of candidate bounding boxes. The NMS module 948 applies non-maximum suppression to the obtained bounding boxes to discard redundant boxes, such as overlapping predictions with similar proportions, thereby obtaining the final set of bounding boxes as the output of object detection.

[0164] Figure 10 This is a schematic block diagram illustrating the optional head 1000 of the CNN. Head 1000 forms part of the overall network known as “Faster RCNN” and includes a feature network (i.e., the backbone 400), a region proposal network, and a detection network. Input to head 1000 is tensor 149, which includes tensors 1010, 1012, 1014, 1016, and 1018 for layers P2-P6. Tensors 1010, 1012, 1014, 1016, and 1018 for layers P2-P6 are input to the region proposal network (RPN) head module 1020. The RPN head module 1020 convolves the input tensors to produce intermediate tensors, which are fed into two subsequent sibling layers: one for classification and one for bounding box or “region of interest” (ROI) regression to a classification and bounding box 1022. The classification and bounding boxes 1022 are passed to the NMS module 1024, which prunes redundant bounding boxes by removing overlapping boxes with lower scores to produce pruned bounding boxes 1026. Bounding boxes 1026 are then passed to the Region of Interest (ROI) pooler 1028. The ROI pooler 1028 uses max pooling to produce fixed-size feature maps from various input-size maps, where subsampling takes the maximum value from each group of input values ​​to produce one output value in the output tensor.

[0165] The input to the ROI pooler 1028 consists of P2-P5 feature maps 1010, 1012, 1014, and 1016, along with region of interest (ROI) proposals 1026. Each proposed ROI from 1026 is associated with a portion of feature maps 1010-1016 to produce a fixed-size map. The size of this fixed-size map is independent of the lower-level portions of feature maps 1010-1016. For example, one of the feature maps 1010-1016 is selected according to the following rule such that the resulting cropped map has sufficient detail: floor(4+log2(sqrt(box_area) / 224)), where 224 is the canonical box size. The ROI pooler 1028 thus crops the input feature map according to proposal 1026, producing tensor 1030. Tensor 1030 is fed into the fully connected (FC) neural network head 1032. The FC (Full-Connected) head 1032 performs two fully connected layers to generate class scores and bounding box prediction increment tensors 1034. The class scores are typically 80-element tensors, each element corresponding to the predicted score for the corresponding object class. The bounding box prediction increment tensor is an 80 × 4 = 320-element tensor containing the bounding boxes for the corresponding object classes. Final processing is performed by the output layer module 1036, which receives tensor 1034 and performs filtering to generate a filtered tensor 1038. Low-scoring (low-classification) objects are no longer considered. The non-maximum suppression module 1040 removes overlapping bounding boxes by removing those with lower classification scores, resulting in the inference output tensor 151.

[0166] Figure 11 This is a schematic block diagram showing the feature map packing arrangement 1100 in a two-dimensional array in the form of a monochrome frame 1102. Feature maps of three layers, such as feature map 1110, feature map 1112, and feature map 1114, can be arranged in frame 1102. Figure 11In the example, frame 1102 includes regions each corresponding to a feature map (e.g., feature map 1110). Feature maps 1110, 1112, and 1114 are placed in a raster scan arrangement that fills monochrome frame 1102. The size of frame 1102 is initially set based on the area of ​​all feature maps to be placed in frame 1102, where the aspect ratio is approximately the aspect ratio of the target UHD frame, i.e., 3840 / 2160 = 1.78. The resolution can be increased in width and height to be a multiple of the minimum block size, for example, such that the width and height are each a multiple of four. When placing the feature maps, the final frame height can be increased to provide sufficient space due to the misalignment between the feature map size and the frame width (since feature maps cannot be packed together without any unused space, some unused space is allowed). The sample values ​​in the unused space in frame 1102 (such as unused space 1104) are set to the midpoint of the bit depth of the frame, i.e., 512 (512) for a 10-bit frame. The size of the feature maps depends on the CNN backbone 114. For the “DarkNet-53” backbone, the size can be 136×76 for feature map 1110 with 256 (256) instances, 68×38 for feature map 1112 with 512 (512) instances, and 34×19 for feature map 1120 with 1024 (1024) instances. To be clear, Figure 12 Frame 1202 is shown, which includes fewer feature maps than those found in typical applications, but as described below, in Figure 12 The text indicates three layers and their relative resolution. Different CNNs and different divisions between the "stem" and "head" parts of a CNN can result in different dimensions and numbers of feature maps for each layer, as well as different numbers of layers (i.e., the number of layers other than three).

[0167] When feature maps are placed in a two-dimensional array as monochrome frames 1102, feature maps from the same group of frames are placed adjacent to each other in frame 1102. For example, group 1106 contains feature map 1110, and groups 1108 and 1109 contain the remaining feature maps in the layer. Furthermore, in the case of two additional groups for the layer, group 1114 contains feature map 1112. For simplicity, grouping is not shown for the layer containing the smallest feature map (i.e., feature map 1120), but the same grouping method is used. Within each group, the feature maps exist in a defined order, and their placement in monochrome frames 1102 reflects this order.

[0168] After placing the feature map Figure 12In a monochrome frame 1202, alignment with specific boundaries (such as 4×4 grid boundaries) can be maintained. When the feature map size is not a multiple of this alignment, unused sample space exists between adjacent feature maps. For example, placing a 34×19 feature map occupies a 36×20 sample region, where the unused space is occupied by midtone sample values. The existence of unused space between feature maps reduces the occurrence of coding artifacts in a feature map caused by content in adjacent feature maps and improves the alignment of the feature map to the underlying block structure of the video codec. For example, for VVC, a minimum block size of 4×4 is typically used.

[0169] In addition to aligning feature maps to a specific alignment grid, minimum padding can be enforced between feature maps (such as two samples). When the feature map size is a multiple of the alignment grid, minimum padding helps prevent artifacts in one feature map caused by content from adjacent feature maps. For example, a 136×76 feature map fits into a 4×4 alignment grid, with no unused sample space inserted between the feature map itself and its neighboring feature maps. Minimum padding regions ensure some separation between adjacent feature maps, which can help reduce encoding artifacts that cross from one feature map to an adjacent feature map.

[0170] Figure 12 This is a schematic block diagram illustrating an alternative feature map packing arrangement 1200 in a monochrome frame 1202. The feature map packing arrangement 1200 is suitable for feature map packing with multiple groups of four feature maps. Figure 12 Grouping can be based on the spatial similarity between feature maps, resulting in groups of similar feature maps. Spatial similarity can be measured using the sum of absolute differences, the sum of squared differences, or some other similarity measure. Grouping applies to feature maps within the same layer and does not span multiple layers. Figure 12As seen, group 1210 comprises four feature maps. The feature maps of group 1210 are placed in monochrome frame 1202 using sample interleaving to occupy a 2×2 region of the component feature maps. Sample interleaving allows the higher structural details of the four feature maps to be shared by the same coding tree structure, where the details between the four feature maps vary with samples. Thus, a common coding tree structure and shared residuals (besides the local differences required to encode adjacent samples of different feature maps) are achieved, thereby improving compression efficiency. Once all groups of size four have been packed into monochrome frame 1202 for a given layer, the remaining feature maps (such as feature map 1214) are packed adjacently based on groups rather than in an interleaved manner. The remaining feature maps can be assigned to groups of any size because their group composition does not affect the packing process (except for the packing order). For the next layer, groups of four (such as group 1220) are packed in a sample interleaved manner, followed by feature maps belonging to other sizes (such as feature map 1224). For the final layer, groups of four (such as group 1230) are packed in a sample-interleaved manner, followed by feature maps belonging to other groups of different sizes (such as feature map 1234).

[0171] Figure 13 This is a schematic block diagram illustrating the feature map packing arrangement 1300 in a 4:2:0 chroma subsampled color frame 1301. Feature map groups containing two or three feature maps with high similarity and belonging to different layers are placed in different color channels within the juxtaposition area of ​​color frame 1301. Thus, the position of at least a portion of a first feature map in one layer relatively corresponds to the position of at least a portion of a second feature map in another layer. For two feature maps in adjacent layers, the larger feature map is placed in the luminance plane 1302, such as feature map 1304. The smaller feature map in these two feature maps is placed in the chroma plane 1310, such as feature map 1314. In the case where the group includes three feature maps, the third feature map is smaller than the feature map placed in the chroma plane 1310, and the third feature map is packed into the second chroma plane 1320, doubling its size, resulting in a doubled-packed feature map 1324. Since the two or three feature maps in the group are grouped based on spatial similarity, Figure 13 In the example, when encoding color frame 1301, coding tools targeting inter-channel correlations can be used to improve compression efficiency. For example, tools that attempt to predict chroma samples from luminance based on a poor model (such as a linear model for cross-color component prediction) can be applied. For inter-frame slices, where a shared coding tree specifies luminance and chroma coding blocks, a single coding tree is used to encode the block structure of two or three feature maps, instead of requiring separate coding trees as in cases where feature maps are placed in different locations.

[0172] Figure 14This is a schematic block diagram illustrating bitstream 1400, which holds encoded packaged feature maps and associated metadata. Bitstream 1400 corresponds to bitstream 121 generated by video encoder 120 or bitstream 143 decoded by video decoder 144. The bitstream contains a group of syntaxes that begins with a “Network Abstraction Layer” unit header. For example, NAL unit header 1408 precedes Sequence Parameter Set (SPS) 1410. SPS 1410 may include “profile level tier (PLT)” units of syntax 1438, which may include “General Constraint Information” (GCI) units of the syntax (i.e., constraint flag 1440). Constraint flag 1440 exists in SPS 1410 when “gci_present_flag” is present in SPS 1410 and equal to 1; otherwise, constraint flag 1440 does not exist in SPS 1410. When constraint flags are present in SPS1410, any activated flag indicates that bitstream 1400 conforms to a restricted subset of tools or functions indicated in the signaled profile of bitstream 1400 (which may correspond to a sub-profile). When constraint flags are not present in SPS1410, the individual constraint flags that would otherwise be signaled are inferred to have a value of zero, and the bitstream conforms to the signaled profile of bitstream 1400. Each flag in constraint flag 1440, when set, indicates the disabling of a specific tool in the VVC standard, where the semantics of the flag are defined in the VVC standard. A separate set of syntax elements (zero or more instances of the ptl_num_sub_profiles and general_sub_profile_idc syntax elements) identifies a specific sub-profile that the bitstream conforms to, where the definition of the sub-profile is defined outside the VVC standard. GCI includes a set of flags where each flag constrains a specific encoding tool not to be used in bitstream 1400. PLT 1438 can signal a specific set of tools available in bitstream 1400, known as a "grade". An example of a grade is "Main 10", which provides 8- to 10-bit video in 4:0:0 or 4:2:0 chroma format and is intended for broad deployment. GCI can indicate further constraints on the toolset to subsets of tools for a grade, which may correspond to subgrades. Typically, when video encoder 120 is encoding video samples (i.e., from video source 112 via multiplexer 118), all tools of a given grade can be used to efficiently encode frame data. When video encoder 120 is encoding feature maps packed into frames (i.e., from module 116), some tools of the VVC standard no longer provide compression benefits. Tools that do not provide compression benefits for packed feature maps do not need to be attempted by video encoder 120 and can be signaled in GCI as not being used in bitstream 1400.SPS 1410 also indicates the chroma format, bit depth, and resolution of the frame data represented by bit stream 1400.

[0173] The Picture Parameter Set (PPS) 1412 includes syntactic elements that control the low-level behavior of the control tools (including control of the deblocking filter). PPS 1412 includes the `pps_deblocking_filter_control_present_flag`, which, when set, indicates the control of the deblocking filter settings in PPS 1412. When `pps_deblocking_filter_control_present_flag` is set, the `pps_deblocking_filter_disabled_flag` is present in PPS 1412. When `pps_deblocking_filter_disabled_flag` is present in PPS 1412 and set to 1, the deblocking filter is disabled for all pictures referencing PPS 1412 unless further overriding of the deblocking control occurs in the picture header or the slice header 1418 of the picture. When pps_deblocking_filter_disabled_flag exists in PPS1412 and is set to 1, pps_deblocking_filter_override_enabled_flag also exists in PPS1412. When pps_deblocking_filter_override_enabled_flag exists in PPS1412 and is set to 1, the slice header 1418 or the image header of each individual image includes additional flags that can override the enabled or disabled deblocking filter indicated by pps_deblocking_filter_disabled_flag.

[0174] SEI message 1413 encodes the feature map grouping 1430 determined by group determiner module 510 and the quantization range 1432 determined by range determiner module 514. Appendix A shows example syntax and semantics of SEI message 1413. The packing format used by packer module 522 can also be encoded in SEI message 1413, thereby selecting a feature packing format from an enumeration of all available feature packing formats using an index. An index can also be used in SEI message 1413 to indicate a specific CNN backbone used to generate the feature maps, selecting a CNN backbone from an enumeration of a predetermined set of CNN backbones (some or all of which are available to source device 110). Based on the CNN backbone type index, the number of layers and the number of channels in each layer, as well as the resolution of each feature map in each layer, can be determined. For groupings where feature maps within a given group are located in the same layer, a separate list of feature map indices is encoded for each layer. For groupings where feature maps in a given group may span multiple layers, feature map indices and layer index pairs are encoded as items in each group. For groups where each layer contains at most one feature map, and for groups existing in adjacent layers, a layer index is only needed for the first feature map in the group. If the group includes feature maps from all layers (e.g., all three layers), no group index is needed because the feature map index is implicitly applied to a single feature map within each layer. If all feature maps of a given layer belong to a distinct layer, a quantization range for each layer is encoded.

[0175] Each frame is encoded as an "access unit" in bitstream 1400, such as Figure 14 Access unit 1414 is shown in the image. Each access unit includes one or more slices, such as slice 1416. For the first access unit of the bitstream, and typically for "random access point" access units, intra-frame slices are used to avoid any predictive dependencies on other access units in the bitstream 1400. Slice 1416 includes a slice header 1418, followed by slice data 1420. Slice data 1420 includes a sequence of CTUs, providing an coded representation of the frame data. CTUs are square and typically 128×128 in size, which is not well aligned with the typical feature map size. Aligning the feature map to a minimum block size (such as a 4×4 grid) partially improves this misalignment.

[0176] Figure 15A method 1500 for performing the first part of a CNN and encoding the resulting feature maps of frames of video data is shown. Method 1500 can be implemented using devices such as configured FPGAs, ASICs, or ASSPs. Optionally, as described below, method 1500 can be implemented by source device 110 as one or more software code modules of application program 233 under the execution of processor 205. The software code module of application program 233 implementing method 1500 can reside, for example, in hard disk drive 210 and / or memory 206. Method 1500 is repeated for each frame of video data generated by video source 112. Method 1500 can be stored in computer-readable storage media and / or memory 206.

[0177] Method 1500 begins with step 1510, which performs the first part of the CNN. In step 1510, the CNN backbone 114, under the execution of processor 205, performs a subset of the layers of a specific CNN to transform the input frame 113 into an intermediate tensor 115. Tensor 115 may contain multiple tensors due to the use of a prediction head or FPN. Method 1500 is used to encode the tensor corresponding to a frame of video data from video source 112. Control in processor 205 then proceeds from step 1510 to step 1520, which determines feature map similarity. The intermediate tensor 115 may be stored, for example, in memory 206 and / or hard disk 210.

[0178] In step 1520, which determines feature map similarity, module 116, under the execution of processor 205, generates a similarity matrix containing a measure of the similarity between each feature map and other feature maps within each layer. The similarity matrix can be stored, for example, in memory 206 and / or hard disk 210. The similarity measure can be the mean squared error (MSE) of two feature maps, the sum of absolute differences (SAD) of two feature maps, or some other difference measure. When it is desired to measure the similarity of feature maps in different layers, feature maps with lower spatial resolution can be upgraded (e.g., using nearest neighbor interpolation) to produce a resolution compatible with higher spatial resolution for the purpose of difference measurement. To reduce computational overhead, step 1520 is performed only occasionally, for example, only for the first image of the CLVS, or for each random access point in the CLVS. Control in processor 205 then proceeds from step 1520 to step 1530, which determines feature map grouping.

[0179] In step 1530, which determines the feature map groups, the group determiner 510, executed by the processor 205, determines the set of groups to which the feature maps are assigned. The groups of feature maps can be stored, for example, in memory 206 and / or hard disk drive 210. (See reference) Figure 17The operation of the group determiner 510 is described. Step 1530 is required when the similarity matrix of step 1520 is determined, for example, for the first image of CLVS or for each random access point in CLVS. Control in processor 205 proceeds from step 1530 to step 1540, which determines the placement of the feature map.

[0180] In step 1540, which determines the placement of feature maps, the packer module 522, executed by the processor 205, determines the position in the frame where each feature map will be placed. When the frame is a monochrome frame, the feature maps are placed in a raster scan order that fills the frame region, wherein the frame region is initialized based on the total area of ​​all feature maps to be packed into the frame and the target aspect ratio. (See reference...) Figures 11 to 13 Describes the packing arrangement. The packing format used is determined by the "packing_format" syntax element decoded from SEI message 1413 as described in Appendix A. Feature maps belonging to a given group are packed and unpacked sequentially according to the order in which they are listed in the corresponding group. Groups of two or three feature maps belonging to different layers are spatially juxtaposed but packed in different color channels, as described in Reference A. Figure 13 As the number and size of the feature maps do not change during the operation of the source device 110, the positions can be determined once and saved for subsequent frames. Packed frames can be stored, for example, in memory 206 and / or hard disk 210. Control in processor 205 then proceeds from step 1540 to step 1550, which determines the group range.

[0181] In step 1550, when determining the group range, range determiner 514, under the execution of processor 205, determines the range of floating-point data in each group of feature maps determined in step 1530. The determined range can be stored, for example, in memory 206 and / or hard disk drive 210. For symmetric operations, the range of a group is the maximum amplitude (absolute) value of the values ​​in the feature maps belonging to that group. This range provides a value for normalization of the feature map data before conversion and quantization to integer sample values. For asymmetric operations, positive and negative ranges are determined for each group of feature maps, indicating the maximum positive and maximum negative values ​​encountered within that group of feature maps. Quantization ranges are determined for each group of feature maps in tensor 115. Quantization ranges can be determined for tensors of each frame of video data, or less frequent updates can be applied. To reduce the overhead of signal notification, quantization ranges can be determined only for intra-frame pictures or randomly accessed pictures in the video bitstream. The range of floating-point data tensors in subsequent frames for which quantization ranges have not been determined may exceed the previously determined quantization range. A safety margin can be introduced by increasing the amplitude of the determined quantization range via some specified scaling factor. The quantization range is multiplied by a fixed factor (e.g., 8 / 7) to compress the range of data samples used to a range that roughly corresponds to the range of video used in YCbCr video data. Later frames, where the quantization range may not be determined, may have some margin exceeding this range up to the limit of sample bit depth (e.g., [0…1023] for 10-bit video). Control in processor 205 then proceeds from step 1550 to step 1560, which involves quantizing the feature map.

[0182] In step 1560 of quantizing feature maps, the quantizer module 518, executed by the processor 205, quantizes each feature map from floating-point values into integer sample values according to the quantization range of the group to which the feature map belongs. The determined integer sample values can be stored in, for example, the memory 206 and / or the hard disk drive 210. Scaling to a normalization range with a maximum magnitude of 1.0 is performed first, followed by multiplication to the sample range and addition of an offset, so that most of the sample magnitude is utilized. For 10-bit video, a multiplication factor of five hundred and twelve (512) is used, and an offset quant_offset of five hundred and twelve (512) is also used. To reduce nonlinear effects from overshoot that may be introduced by the video encoder 120 and the video encoder 144, a smaller multiplication factor can be used. If the quantization range has not been adjusted by a fixed factor (such as 8 / 7) to align with the video range commonly used in YCbCr video data, a scaling factor scale_f of 7 / 8*512=448 can be used. For 8-bit video data, for the video range alignment operation, an offset of one hundred and twenty-eight (128) and a scaling factor of one hundred and twenty-eight (128) or one hundred and twelve (112) can be used. In the case where the quantization range has been determined for the tensor from the previous frame and has not been updated for the current frame, the incoming floating-point values may exceed the quantization range of the feature map group to which the feature map belongs. To prevent overflow when mapping floating-point values to integer sample values, a clipping operation is applied. In one arrangement of the quantizer module 518, floating-point values are clipped into the range indicated by the quantization range to prevent overflow. Clipping floating-point values to the quantization range ensures that all samples are within the range [quant_offset-scale_f, quant_offset+scale_f]. In another arrangement of the quantizer module 518, after applying quant_offset and scale_f (at this point, the determined value may fall outside the range indicated by the bit depth), and before converting to integer sample values, clipping is applied. Clipping is applied to ensure that the integer sample values are within the range indicated by the bit depth, that is, [0…(1<<bit_depth)-1]. Clipping after scaling and before integer conversion, combined with the scale_f value for a smaller range (such as a video range), allows some margin for subsequent frames to exceed the quantization range determined from earlier frames. It also allows a certain degree of overshoot in the operations of the video encoder 120 and the video decoder 144 before clipping introduces nonlinear distortion into the conversion from floating-point tensors to integers and back to floating-point tensors. Control in the processor 205 then proceeds from step 1560 to step 1570 of packing feature maps.

[0183] In step 1570, when packing the feature maps, the packer module 522, under the execution of the processor 205, packs the integer feature maps 520 to produce a packed feature map frame 117. The quantized feature maps 520 corresponding to the feature maps from the various layers of tensor 115 can be stored in a memory buffer, configured, for example, within memory 206 and / or hard disk drive 210, to hold a frame of video data. Reference Figures 11 to 13 The packaging format of the feature map is described. Control in processor 205 then proceeds from step 1570 to step 1580, which encodes the metadata.

[0184] In step 1580, when encoding metadata, entropy encoder 638, under the execution of processor 205, encodes feature map groups 512 and quantization ranges 516 (i.e., metadata 125) into bitstream 121. Metadata 125 can be encoded using SEI message 1413. The format of SEI message 1413 is described with reference to Appendix A. Control in processor 205 then proceeds from step 1580 to step 1590, which encodes the frame. At the first image (image sequence number equal to 0), the “layers_update,” “groups_update,” and “qr_update” flags in SEI message 1413 are set, and feature layers and dimensions, feature map group definitions, and associated quantization ranges are encoded into bitstream 121. The “qr_update” flag in SEI message 1413 can be set periodically to update the quantization range information accordingly. For random access configurations, each random access point or intra-frame image may include an updated quantization range. For low-latency configurations, the quantization range can be updated periodically for inter-frame images, for example, approximately one image per second. This corresponds to the periodicity of intra-frame images in a random access configuration. For instance, updating the quantization range for some inter-frame images allows for continuous adaptation of data independent of the bitstream structure (i.e., intra / inter-frame slice selection) when intra-frame images rarely appear in the bitstream.

[0185] In step 1590 of frame encoding, video encoder 120, under the execution of processor 205, encodes frame 119 into bitstream 121. When source device 110 is configured to encode feature maps, frame 119 is obtained from packaged feature map frames 117 via multiplexer 118. When source device 110 is configured to encode feature maps, video encoder 120 can use a subset of encoding tools available for different levels of video coding standards. A general constraint flag can be used to signal the subset of encoding tools. For example, the "Main 10" level can be signaled in the level-level syntax 1438 of bitstream 120, and the general constraint flag 1440 can signal that the following tools are not used in bitstream 120: LFNST (via gci_no_lfnst_constraint_flag), MIP (via gci_no_mip_constraint_flag), LMCS (via gci_no_lmcs_constraint_flag), ISP (via gci_no_isp_constraint_flag), affine (via gci_no_affine_motion_constraint_flag), GPM (via gci_no_gpm_constraint_flag), and MMVD (via gci_no_mmvd_constraint_flag). In addition to using the GCI flag, or an alternative to it, sub-profiles can be defined outside the VVC standard for feature map encoding, and can be identified within the bitstream using specific values ​​of the general_sub_profile_idc syntax element, which can be included in SPS1410. Disabling the deblocking filter achieves better compression efficiency and higher task performance when encoding feature maps. In the VVC encoding standard, for images in the reference bitstream 121 with the image parameter set pps_deblocking_filter_disabled_flag set to "1", the deblocking filter is disabled unless overridden at the slice or image level by encoding sh_deblocking_filter_disabled_flag with a value of "1" or ph_deblocking_filter_disabled_flag with a value of "1". Even though this disabling shows advantages, deblocking is not explicitly disabled using constraint flags in VVC standard version 1, therefore disabling the deblocking filter does not constitute part of a subset of tools that can be equivalent to the sub-profiles used for feature map encoding. Method 1500 completes, and processing in processor 205 proceeds to the next frame.

[0186] Figure 16A method 1600 for decoding feature maps from encoded data and performing the second part of a CNN is shown. Method 1600 can be implemented by a device such as a configured FPGA, ASIC, or ASSP. Alternatively, as described below, method 1600 can be implemented by a destination device 140 as one or more software code modules of application program 233 under the execution of processor 205. Method 1600 is repeated for each frame of video data encoded in bitstream 143. The software code module of application program 233 implementing method 1600 can be stored, for example, on hard disk drive 210 and / or memory 206. Method 1600 begins with step 1610 of decoding feature map groups. Method 1600 is configured to determine one or more parameters associated with quantization; and to inverse quantize data samples decoded from encoded data to derive feature maps according to one or more parameters. In one arrangement, method 1600 is configured to deinterleave feature maps corresponding to a set of feature maps after inverse quantization. As described in detail below, method 1600 can be used to determine feature maps based on an image of a first set of feature maps arranged in a first frame (or two-dimensional array) and a second set of feature maps arranged in a second frame (or two-dimensional array), wherein the first frame is different from the second frame.

[0187] In step 1610, which decodes the feature map groups, the entropy decoder 720, under the execution of processor 205, decodes from SEI message 1413 a structure indicating that each feature map of each layer is assigned to one or more groups of feature maps (i.e., feature map group 820). The decoded structure can be stored, for example, in memory 206 and / or hard disk drive 210. The syntax of the feature map groups in SEI message 1413 is described with reference to Appendix A. Control in processor 205 then proceeds from step 1610 to step 1620, which decodes the quantization range.

[0188] In step 1620, which decodes the quantization range, the entropy decoder 720, executed by the processor 205, decodes parameters in the form of the quantization range 822 for each feature map group 820 as determined in step 1610 from the SEI message 1413. The quantization range 822 is shared by the individual feature maps in the multiple feature maps within the feature map group. The quantization range 822 determined in step 1620 may be stored, for example, in memory 206 and / or hard disk drive 210. When symmetric quantization is used, a single value representing the maximum amplitude of floating-point data within the feature map belonging to the corresponding group is decoded in step 1620 for each feature map group. When asymmetric quantization is used in step 1620, a pair of values ​​representing the maximum and minimum values ​​of floating-point data within the feature map belonging to the corresponding group is decoded for each feature map group. The processor 205 may operate to perform step 1620 for each frame of video data, or the processor 205 may operate to perform step 1620 less frequently. Step 1620 can be performed either within an intraframe image or at a random access point in bitstream 143. When step 1620 is not performed for every frame, feature map groups and quantization range data are carried over to subsequent frames for reuse until a new set of feature map groups and / or quantization range data is decoded from bitstream 143. Control in processor 205 then proceeds from step 1620 to step 1630, which involves decoding the frame.

[0189] In step 1630, during frame decoding, entropy decoder 114, executed by processor 205, generates frame 145 by decoding the portion of bitstream 143 corresponding to access units such as AU 1414. Frame 145 may contain a packed feature map or may contain an image corresponding to, for example, a frame from video source 112. If frame 145 contains an image frame, i.e., does not contain a packed feature map, method 1600 terminates, decoding proceeds to the next frame. Frame 145 generated in step 1630 may be stored, for example, in memory 206 and / or hard disk 210. If frame 145 contains a packed feature map, processor 205 proceeds from step 1630 to step 1640, which determines feature map placement.

[0190] In step 1640, where feature map placement is determined, the unpacking module 810, under the execution of the processor 205, determines the position of each feature map in each layer of frame 145. Using the spatial size of each feature map, feature map grouping, and the number of feature maps in each layer, the unpacking module 810 determines the position of each feature map in each layer according to the method of step 1540 and as referenced. Figures 11 to 13 The placement information is determined. If the feature map size, number, and packing format remain unchanged compared to the previous frame, the feature map placement data from the previous frame is retained. Control in processor 205 then proceeds from step 1640 to step 1650, which involves unpacking the feature map.

[0191] In step 1650, when unpacking the feature map, the unpacking module 810, executed by the processor 205, extracts samples from frame 147 to generate an integer feature map 812 based on the feature map position determined in step 1640. The integer feature map 812 determined in step 1650 can be stored, for example, in memory 206 and / or hard disk drive 210. Control in the processor 205 then proceeds from step 1650 to step 1660, which involves inverse quantization of the feature map.

[0192] In step 1660, which involves inverse quantization of the feature maps, the inverse quantizer module 814, executed by processor 205, converts the integer feature map 812 into a floating-point feature map. These floating-point feature maps are combined into tensor 149 as input to the CNN head 150. The floating-point feature maps can be stored, for example, in memory 206 and / or hard disk 210. Integer samples are converted to floating-point precision, and the quant_offset and scale_f values ​​from step 1560 are used to shift the samples to a normalized range. For each feature map in the feature map group, the normalized range value is multiplied by the quantization range 822 of the feature map group 820 to create a floating-point feature map. The floating-point feature maps are combined into tensor 119 as a multidimensional array, typically with dimensions of (frame, channel, height, width). In the case of using FPN, this is combined to write the feature maps into a tensor in tensor set 119 corresponding to an FPN layer. Control in processor 205 then proceeds from step 1660 to step 1670, which performs the second part of the CNN.

[0193] In step 1670 of the second part of the CNN, the CNN head 150 performs the remaining stages of the CNN (i.e., task-specific stages) under the execution of processor 205. Decoded, unpacked, and inverse-quantized tensors 149 are input into the CNN head 150. Within the CNN head 150, a series of convolutions, normalization, fully connected layer operations, and activation stages are performed to obtain the CNN result 151. The CNN result 151 is stored, for example, in a task result buffer 152 configured within memory 206. Method 1600 terminates, and control in processor 205 advances to the next frame.

[0194] In the arrangement of method 1600, steps 1610 and 1620 are performed when the flag in SEI message 1413 indicates a specific flag. Step 1610 is performed when the "groups_update" flag decoded from SEI message 1413 indicates a specific flag, and step 1620 is performed when the "qr_update" flag decoded from SEI message 1413 also indicates a specific flag.

[0195] Figure 17A method for determining the grouping of feature maps is illustrated. Method 1700 can be implemented by a device such as a configured FPGA, ASIC, or ASSP. Alternatively, as described above, method 1700 can be implemented by source device 110 as one or more software code modules of application program 233 under the execution of processor 205. The software code module of application program 233 implementing method 1700 can be stored, for example, on hard disk drive 210 and / or memory 206. Method 1700 begins with step 1710 of the initialization list.

[0196] In step 1710, during the initialization of the list, group determiner 510, under the execution of processor 205, creates a set of groups such that each feature map in a given layer is assigned to a single group. A group is represented as an ordered list of feature maps, with adjacency within a group indicating the similarity of feature map pairs. The ordered list can be initialized and stored in memory 206 and / or hard disk 210. Control in processor 205 then proceeds from step 1710 to step 1720, which involves finding the most similar feature map pair.

[0197] In step 1720, the group determiner 510, under the execution of processor 205, determines the feature map pair with the greatest similarity from the similarity matrix of step 1520. Since the similarity matrix is ​​a measure of the difference between feature maps, the pair with the greatest similarity is identified by the position of the minimum value in the matrix. If the similarity matrix indicates that no other feature map pair has similarity (i.e., all entries are set to "Not a Number" (NaN)), that value is returned. Control in processor 205 then proceeds from step 1720 to step 1730 of the residual map test.

[0198] In step 1730 of the residual graph test, group determiner 510, under the execution of processor 205, determines whether all feature graph pairs were identified in step 1720. If step 1720 returns NaN, then groups that join each feature graph have been considered, and it is no longer necessary to join the groups together (i.e., to form a larger group from two smaller groups). If it is no longer necessary to join the groups together, method 1700 terminates with the set of groups as the result. Otherwise, if the feature graph pairs have the measured similarity (i.e., the result of the minimum operation is not NaN), then control in the processor then proceeds from step 1730 to step 1740, which involves finding the group index.

[0199] In step 1740, which determines the group index, the group determiner 510, executed by the processor 205, determines which group the corresponding feature map belongs to and the index within each group of the feature map. Control in the processor 205 then proceeds from step 1740 to step 1750, which allows for connection group testing.

[0200] In step 1750 of the connectable group test, the group determiner 510, executed by the processor 205, determines whether feature map pairs can be connected to form a larger group. If any feature map is in the middle of its corresponding group, it is impossible to connect these feature maps together because a node in the list may have only one predecessor and one successor node. The entry in the similarity matrix corresponding to the feature map pair is set to NaN, thus preventing further consideration of the feature map pair. Furthermore, if two feature maps belong to the same group, the entry in the similarity matrix corresponding to that feature map pair is set to NaN, thus preventing further consideration of connecting the two feature maps. If two feature maps are at the beginning or end of their respective groups, these feature maps can be connected to form a larger group from the two initial groups. In arrangements where the group size is limited to a specific number of feature maps, for groups that can be connected, if the resulting group size would exceed the group size limit, the entry in the similarity matrix corresponding to the feature map pair is set to NaN, and the groups are not connected. To reduce the iterations for determining feature map groups, if the group size is finite and the resulting group after concatenation equals the group size, the rows and columns in the similarity matrix corresponding to the endpoints of the newly formed group are set to NaN, thus preventing further consideration of concatenating these feature maps into larger groups. If groups are to be concatenated, control in processor 205 proceeds to step 1760 of concatenating groups.

[0201] In step 1760, where groups are joined, group determiner 510, executed by processor 205, joins two groups containing the pair of feature maps identified in step 1720. The groups are joined such that the pair is adjacent in the newly formed larger group. The joined groups determined in step 1760 can be stored, for example, in memory 206 and / or hard disk drive 210. When a feature map is located in the preceding group of two or more feature maps and is joined to another group, the feature map now occupies a position somewhere in the middle of the newly formed larger group. When a feature map becomes an intermediate node in a list or group, the row and column corresponding to that feature map in the similarity matrix are set to NaN, thereby preventing further consideration of joining the feature map to other groups. Processor 205 then proceeds from step 1760 to step 1720 to determine the next pair of feature maps to consider joining into a larger group.

[0202] In one arrangement, all feature maps from each layer are merged into a single group. When packed according to Packing Format 1100, the resulting feature map placement groups similar feature maps relatively close together. The VVC intra-block copy encoding tool can then be used to predict portions of a feature map from previous and adjacent feature maps, with some limitations on block selection generated by the IBC virtual buffer. Since the residuals of feature maps tend to be continuous and are encoded more efficiently using various transforms, the IBC search can use the Hadamard transform as a cost estimate in addition to or replacing the SAD cost estimate.

[0203] In another arrangement, the group size is limited to four. When the group size is limited to four, the feature maps of the "group of four" can be placed using a sample interleaving packing format 1200 to achieve compression efficiency based on a shared block structure and some degree of shared predictive signal between the four feature maps. In the implementation of method 1700, a similarity threshold can be applied so that only groups of four feature maps are determined when the four feature maps are highly similar. Other less similar feature maps can be assigned to a larger remaining group, which is packed in a raster scan format.

[0204] In another arrangement, groups can be defined across layers, and the size is limited to three, making it particularly suitable for three-layer FPNs. Using the packing arrangement 1300, inter-layer groups are packed in a juxtaposed manner, allowing the use of VVC cross-component prediction tools while improving compression efficiency. Combined grouping is possible, where inter-layer groups are used to pair feature maps across layers, while intra-layer groups are arranged based on the layers occupying the luminance channels of the frame.

[0205] In another arrangement, each layer has a group, and all feature maps of a layer reside within that group. Within this group, the order of the feature maps is encoded, allowing similar feature maps within the layer to be placed nearby, enabling tools such as IBC to predict a feature map from neighboring feature maps.

[0206] In another arrangement, each layer has a group, and within each group, feature maps are placed according to the channel index of the feature map's tensor. In this arrangement, a quantization range for each layer is encoded, achieving the low overhead of quantization range encoding as in SEI message 1413.

[0207] Since various grouping methods are possible, the “grouping_type” syntax element is included in SEI message 1413, and is further described with reference to Appendix A.

[0208] Figure 18A method is illustrated for selecting a set of encoding tools or functions of a video standard based on the type of frame data to be encoded. Method 1800 can be implemented by a device such as a configured FPGA, ASIC, or ASSP. Alternatively, as described below, method 1800 can be implemented by source device 110 as one or more software code modules of application program 233 under the execution of processor 205. The software code module of application program 233 implementing method 1800 can be stored, for example, on hard disk drive 210 and / or memory 206. Steps of method 1800 are configured to determine whether source device 120 generates encoded video data including encoded data comprising feature maps based on a convolutional neural network (CNN). Steps of method 1800 are also configured to generate encoded video data using multiple encoding tools or functions for encoding video data, provided that source device 120 generates encoded video data comprising feature maps. As also described, the steps of method 1800 are configured to generate the encoded data of the feature map using a first portion of multiple encoding tools or functions instead of a second portion of multiple encoding tools or functions, in the case where the source device 120 generates second encoded data including encoded data of the feature map.

[0209] Method 1800 begins with step 1810, which determines the frame type configuration.

[0210] In step 1810, when determining the frame type configuration, source device 110 is configured, under the execution of processor 205, to operate on video data or feature map data. This configuration may be the result of receiving commands via network 200 or 222 or via direct user control through a user interface (e.g., via keyboard 202, mouse 203). Control in processor 205 then proceeds from step 1810 to step 1820, which involves testing the frame containing feature map data.

[0211] In step 1820, under the execution of processor 205, source device 110 determines, based on a convolutional neural network (CNN), whether to generate encoded regular video frame data or encoded feature map data. The encoded data conforms to an encoding standard (e.g., the VVC standard). When source device 110 is configured for video frame data, control in processor 205 proceeds from step 1820 to step 1830, which selects the video data function. When source device 110 is configured for feature map transmission, control in processor 205 proceeds from step 1820 to step 1840, which selects the feature map function.

[0212] In step 1830, when selecting a video data function, multiplexer 118, under the execution of processor 205, directly routes frame data 113 to video encoder 120. A set of functions or encoding tools is selected for encoding frame data 119. The set of functions corresponds to the functions available in the profile of the video coding standard used to encode frame data 119. The set of functions corresponds to a first portion of the aforementioned multiple encoding tools or functions. For example, the set of functions defined for the "Main 10" profile of the VVC standard can be selected in step 1830. Control in processor 205 proceeds from step 1830 to step 1850, which involves encoding the frame data.

[0213] In step 1840, when selecting feature map functionality, multiplexer 118, under the execution of processor 205, routes the packaged feature map 117 as frame data 119 to video encoder 120. A subset of functions or sets of encoding tools, representing a standard level of encoding, are selected for use in encoding the frame data 119. A subset of encoding tools can be selected by activating a "constraint flag" to disable specific encoding tools or functions of the video coding standard being used to encode the frame data 119. The disabled encoding tools or functions represent a second portion of the aforementioned encoding tools or functions and can be at least one of the following: Low Frequency Inseparable Transform (LFNST), Matrix Intra-Frame Prediction (MIP), Linear Mode Chroma Scaling (LMCS), Affine Prediction Mode, Geometric Partition Mode (GPM), ISP, and Deblocking Filter. In this example, a constraint flag can be used to indicate the prohibition of the second portion of encoding tools or functions. For video coding standards other than VVC, encoding tools providing similar functionality can be similarly disabled. Control in processor 205 proceeds from step 1840 to step 1850, which involves encoding the frame data.

[0214] In step 1850, when encoding the frame data, the video encoder 120, under the execution of the processor 205, encodes the frame data 119 according to a set of functions or encoding tools. Method 1800 terminates, and the source device 110 advances to the next frame. As a result of method 1800, the bitstream 121 includes a clear indication of whether the contained data is regular video data or packed feature map data (e.g., in the form of a set of constraint flags presented earlier in the bitstream). Furthermore, while the bitstream 121 encodes the packed feature map data, an SEI message 1413 exists for at least one frame, thereby allowing the destination device 140 to further process the data after decoding the bitstream (e.g., processing the decoded frame data 145 using modules 148 and 150). If the destination device 140 intends to perform the task only according to the CNN header 150, the destination device does not need to decode the bitstream 143 when it is indicated that the bitstream 143 contains regular video data in addition to the initial profile and constraint flag syntax. A destination device that only outputs task result 151 to task result buffer 152 and does not output decoded video (e.g., to display device 160) does not need to implement encoding tools or functions that are indicated as disabled via constraint flags.

[0215] In the arrangement of method 1800, instead of indicating which tools are disabled for feature map encoding by setting constraint flags, tools are indicated by disabling an enable flag, for example, in the sequence parameter set or equivalent syntactic structure.

[0216] In the arrangements of methods 1500 and 1600, steps 1580 and 1610 encode and decode the feature map group size into a log2 value (i.e., the feature map group size needs to be a power of 2), wherein an offset of one is applied so that the encoded value of zero corresponds to a feature map group size of one (1). The syntax element “log2_group_size_minus1” is used to encode the feature map group size.

[0217] In another arrangement of methods 1500, 1600, and 1700, feature map groups are constrained to contain feature maps indexed in monotonically increasing order within a given layer. When feature maps exist within each group via monotonically increasing indexes, the group composition can be encoded using a bitmap to indicate the presence of a given feature map within the group. For subsequent groups, the encoded bitmap can be reduced in length to omit feature map indices already assigned to earlier groups.

[0218] In the CNN backbone 310 layout, the tensor dimensions and the resulting feature map sizes are chosen to align with the block size of the VVC standard. For typically rectangular videos and the default CTU size of 128×128, the width and height of the feature maps can be powers of 2; for example, the sizes of the three layers could be 128×64, 64×32, and 32×16. Having the feature map size as a power of 2 ensures better alignment of the packed features with the block sizes available in the VVC standard, resulting from quadtree, binary tree, or ternary tree splits, and reduces the likelihood of encoding artifacts in a single feature map caused by the content of adjacent feature maps.

[0219] In the arrangement of bitstream 1400, SPS1410 includes sps_deblocking_filter_enabled_flag for controlling the deblocking filter, as an additional syntax present when the SPS extension is activated via the flag "sps_extension_flag" being equal to one. When sps_deblocking_filter_enabled_flag is equal to zero, pps_deblocking_filter_control_present_flag in PPS1412 must be set to one to explicitly encode the deblocking filter control, pps_deblocking_filter_override_enabled_flag in PPS1412 must be set to zero to prevent slice header or image header overriding of the deblocking control set in PPS1412, and pps_deblocking_filter_disabled_flag in PPS1412 must be set to zero to disable in-loop filtering. When `sps_deblocking_filter_enabled_flag` is equal to one, these constraints on the flags `pps_deblocking_filter_control_present_flag`, `pps_deblocking_filter_override_enabled_flag`, and `pps_deblocking_filter_disabled_flag` do not apply. `gci_no_deblocking_filter_flag` exists in constraint flag 1440, and when set to one, `sps_deblocking_filter_enabled_flag` in SPS1410 must be set to zero. When `gci_no_deblocking_filter_flag` is set to zero, no constraints are applied to `sps_deblocking_filter_enabled_flag` in SPS1410. If sps_deblocking_filter_enabled_flag does not exist in SPS1410, then when gci_no_deblocking_filter_control_present_flag is set to one, constraints applicable to the pps_deblocking_filter_control_present_flag, pps_deblocking_filter_override_enabled_flag, and pps_deblocking_filter_disabled_flag flags are applied.The application of the deblocking filter is explicitly prohibited via a constraint flag, which allows for the definition of sub-levels for feature map encoding that exclude the application of the deblocking filter. The gci_no_deblocking_filter_flag can exist in the gci_reserved_zero_bits section of constraint flag 1440, which includes version 1 of the VVC standard. Deblocking may be unnecessary when the application of system 100 requires high quality (i.e., high bit rate) achieved using a low value of quantization parameter 692, and constraint flags can be used to completely omit deblocking, for example, for feature map encoding.

[0220] Industrial availability

[0221] The described arrangement is applicable to the computer and data processing industry, particularly to digital signal processing that encodes and decodes signals such as video and image signals, thereby achieving high compression efficiency. The provision of one or more of the aforementioned constraint flags allows selection of a given subset of tools (equivalent to "sub-grades"). The selection of this tool subset offers several benefits, such as implementation benefits for VVC vendors, as vendors can specify, for example, a subset of grades that excludes unnecessary or otherwise problematic encoding tools from a complexity perspective.

[0222] Arrangements for quantizing floating-point tensor data in groups of channels or feature maps and packing the resulting integer values ​​into planar frames are also disclosed. Grouping methods and trade-offs between very coarse and very fine-grained grouping are disclosed, wherein very coarse grouping has low overhead for quantized range data, and very fine-grained grouping has high overhead for quantized range data, with the intermediate granularity of the grouping providing task performance benefits.

[0223] The foregoing has only described some embodiments of the present invention, and modifications and / or changes may be made thereto without departing from the scope and spirit of the invention. These embodiments are illustrative and not restrictive.

[0224] Appendix A: The SEI message format and associated semantics representing metadata related to feature map packing and quantization in the bitstream are as follows:

[0225]

[0226]

[0227] Feature map packaging information semantics

[0228] The syntactic structure specifies the information needed to unpack the feature map plane frames and convert them into tensors for inference tasks.

[0229] A syntactic element with descriptor u(n) indicates that the syntactic element is encoded using n bits and interpreted as an unsigned integer value. A syntactic element with descriptor ue(v) indicates that the syntactic element is encoded as an exponential Golomb value and interpreted as an unsigned integer value.

[0230] The persistence of the Feature Map Information (SEI) message begins from the associated AU and continues until the next occurrence of the SEI message or the end of CLVS.

[0231] A frame_type of 0 indicates that the AU does not contain packed feature map data. A frame_type of 1 indicates that the AU does contain packed feature map data.

[0232] layers_update equal to 1 indicates that this instance of the feature map packing information SEI message defines the number of layers and the dimensions and number of feature maps in each layer.

[0233] The group_update value being 1 indicates that this instance of the feature map packing information SEI message defines the number and composition of feature map groups.

[0234] The value of qr_update equals 1, indicating that this instance of the feature map packing information SEI message signals an update to the quantization range of the feature map group.

[0235] `backbone_id` indicates the type of network backbone and extraction points, implicitly signaling the number of layers and tensor dimensions, and thus the feature map dimensions. The following table shows some predefined network backbones and their associated number of layers, feature map counts, and dimensions:

[0236]

[0237] layer_cnt specifies the number of layers present in a frame.

[0238] fm_cnt[layer_idx] specifies the number of feature maps that exist for layer_idx.

[0239] fm_width[layer_idx] specifies the width of the feature map for layer_idx.

[0240] fm_height[layer_idx] specifies the height of the feature map for layer_idx.

[0241] orig_source_width specifies the width of frame 112 in units of luminance samples before the resizing of the trunk operation (i.e. before the resizing module 304).

[0242] orig_source_height specifies the height of frame 112 in units of luminance samples before the resizing of the trunk operation (i.e. before the resizing module 304).

[0243] The `packing_format` parameter specifies the format in which feature map data is packed within a frame. The formats are enumerated according to the following table:

[0244]

[0245]

[0246] The grouping_type specifies the extent of the feature map group by setting the ExplicitGrouping, ExplicitGroupSize, and ExplicitLayerId flags.

[0247] The ExplicitGrouping flag equal to one indicates that feature map grouping is explicitly signaled in the bit stream, while the ExplicitGrouping flag equal to zero indicates that feature map grouping is implicitly determined based on grouping_type.

[0248] The ExplicitGroupSize flag being equal to one indicates that the size of each feature group is explicitly signaled in the bitstream, while the ExplicitGroupSize flag being equal to zero indicates that the size of each feature group is implicitly determined based on the grouping_type.

[0249] The ExplicitLayerId flag indicates that a group can contain feature maps from different layers, while an ExplicitLayerId flag of zero indicates that a group is implicitly limited to a single layer.

[0250] The following table shows the values ​​assigned to the flags ExplicitGrouping, ExplicitGroupSize, and ExplicitLayerId based on grouping_type, and describes the implicit behavior using implicit signal notification.

[0251]

[0252]

[0253] `group_cnt` exists when the `ExplicitGrouping` flag is equal to one, and signals the number of feature map groups. When `ExplicitGroupingFlag` is equal to zero, `group_cnt` is inferred based on `grouping_type` according to the table above.

[0254] The quant_type indicates the type of quantization operation according to the following table:

[0255]

[0256] qr_fraction_precision specifies the precision of encoding the fractional part of the floating-point quantization range in bits.

[0257] `group_size` exists when both the `ExplicitGrouping` and `ExplicitGroupSize` flags are equal to 1. `group_size` specifies the size of the group `grp_idx`. When `group_size` does not exist, it is inferred from the implicit rules described in the `grouping_type` table.

[0258] fm_idx[grp_idx][fm_idx] specifies the feature map index or channel index of position fm_idx within group grp_idx.

[0259] `layer_id[grp_idx][fm_idx]` (if present) specifies the layer index for the corresponding feature map identified in `fm_idx[grp_idx][fm_idx]`. If `layer_idx` does not exist, it is inferred. For `group_type` equal to 1, 2, or 3, feature maps in layer 0 are first assigned to one or more groups, and once all feature maps in layer 0 have been assigned to groups, feature maps in layer 1 are assigned to one or more groups, and so on. For `group_type` equal to 4, a group contains all feature maps from all layers.

[0260] qr_exp[grp_idx] specifies the exponential portion of the quantization range of group grp_idx.

[0261] qr_exp_sign[grp_idx] specifies the sign of the exponential portion of the quantization range of group grp_idx.

[0262] qr_fraction[grp_idx] specifies the fractional part of the quantization range of group grp_idx, where the bit width is specified by qr_precision.

[0263] second_qr_exp[grp_idx] (if present) specifies the exponential portion of the second quantization range of group grp_idx.

[0264] second_qr_exp_sign[grp_idx] specifies the sign of the exponential portion of the quantization range of group grp_idx.

[0265] second_qr_fraction[grp_idx] (if present) specifies the fractional part of the second quantization range of group grp_idx, where the bit width is specified by qr_precision.

[0266] When quant_type equals zero, the quantization range indicates the maximum magnitude of the value encountered in the feature map within the group to which the quantization range is applied.

[0267] When quant_type equals one, the quantization range indicates the maximum positive value encountered in the feature map within the group to which the quantization range is applied, while the second quantization range indicates the maximum negative value encountered in the feature map within the group to which the second quantization range is applied.

[0268] The quantization range and the second quantization (if present) range can also be adjusted, for example, by multiplying by a value slightly greater than 1.0 to allow for some margin. This margin allows the quantization range to be reused for frames following the frame associated with the feature map packing information SEI message, reducing the likelihood of needing to clip tensor values ​​at quantization module 518.

Claims

1. An encoding device, comprising: A determining unit is configured to determine whether to generate encoded data for a frame having multiple feature maps arranged thereon, the multiple feature maps being obtained at least based on processing an input image using a neural network. as well as An encoding unit is configured to generate encoded data for an input image using at least a plurality of functions, including a deblocking filter, when generating encoded data for an input image rather than encoded data for a frame arranged with the plurality of feature maps. When it is determined that coded data for a frame with the plurality of feature maps arranged is to be generated, the coding unit uses a first portion of the plurality of functions instead of a second portion of the plurality of functions, including a deblocking filter.

2. The encoding device according to claim 1, wherein, The second part of the multiple functions includes at least one of low-frequency inseparable quadratic transformation, matrix intra-frame prediction, linear model chroma scaling, and intra-frame sub-partitioning.

3. The encoding device according to claim 1, wherein, The second part of the multiple functions includes at least one of an affine prediction mode and a geometric partitioning mode.

4. The encoding device according to claim 1, wherein, The second part of the plurality of functions is constrained not to be used for the generation of encoded data for frames in which the plurality of feature maps are arranged.

5. The encoding device according to claim 1, wherein, The encoding unit is configured to encode information indicating that a second part of the plurality of functions is constrained to not be used for decoding encoded data of frames in which the plurality of feature maps are arranged.

6. The encoding device according to claim 1, wherein, The encoding device is used to generate first encoded data and second encoded data, wherein the first encoded data conforms to a first encoding standard and the second encoded data conforms to a second encoding standard.

7. The encoding device according to claim 1, wherein, The encoding unit is configured to encode the prohibition of use of the second part of the plurality of functions using constraint flags.

8. A decoding device, comprising: A determining unit is configured to determine whether to decode the encoded data of a frame having multiple feature maps arranged thereon, the multiple feature maps being obtained at least based on processing the input image using a neural network; as well as A decoding unit is configured to decode the encoded data of the input image using at least a plurality of functions, including a deblocking filter, when decoding the encoded data of the input image is required instead of decoding the encoded data of a frame having the plurality of feature maps arranged thereon. When it is determined that the encoded data of a frame having the plurality of feature maps arranged is to be decoded, the decoding unit uses a first portion of the plurality of functions instead of a second portion of the plurality of functions, including a deblocking filter.

9. The decoding device according to claim 8, wherein, The second part of the multiple functions includes at least one of low-frequency inseparable quadratic transformation, matrix intra-frame prediction, linear model chroma scaling, and intra-frame sub-partitioning.

10. The decoding device according to claim 8, wherein, The second part of the multiple functions includes at least one of an affine prediction mode and a geometric partitioning mode.

11. The decoding device according to claim 8, wherein, The second part of the plurality of functions is constrained to not be used for decoding encoded data of frames in which the plurality of feature maps are arranged.

12. The decoding device according to claim 8, wherein, The decoding unit is configured to decode information indicating that a second part of the plurality of functions is constrained to not be used for decoding encoded data of frames in which the plurality of feature maps are arranged.

13. The decoding device according to claim 8, wherein, The decoding device is used to decode the first encoded data and the second encoded data, wherein the first encoded data conforms to the first encoding standard and the second encoded data conforms to the second encoding standard.

14. The decoding device according to claim 8, wherein, The decoding unit is configured to decode the disabling of the second part of the plurality of functions using constraint flags.

15. An encoding method, comprising: Determine whether to generate encoded data for a frame with multiple feature maps arranged thereon, the multiple feature maps being obtained at least based on processing the input image using a neural network; In the case where the goal is to generate encoded data of an input image rather than encoded data of a frame with the plurality of feature maps arranged thereon, the encoded data of the input image is generated using a plurality of functions including at least a deblocking filter. as well as When it is determined that encoded data of a frame with the plurality of feature maps arranged is to be generated, the encoded data is generated using a first portion of the plurality of functions instead of a second portion of the plurality of functions, which includes a deblocking filter.

16. A decoding method, comprising: Determine whether to decode the encoded data of a frame with multiple feature maps arranged thereon, the multiple feature maps being obtained at least based on processing the input image using a neural network; In cases where the encoded data of an input image needs to be decoded instead of the encoded data of a frame with the plurality of feature maps arranged thereon, the encoded data of the input image is decoded using a plurality of functions including at least a deblocking filter. as well as When it is determined that the encoded data of a frame having the plurality of feature maps arranged is to be decoded, the encoded data is decoded using a first portion of the plurality of functions instead of a second portion of the plurality of functions, which includes a deblocking filter.

17. A non-transitory computer-readable storage medium storing a program that, when executed by a computer, implements the encoding method according to claim 15.

18. A non-transitory computer-readable storage medium storing a program that, when executed by a computer, implements the decoding method according to claim 16.

19. A computer program product comprising a program that, when executed by a computer, implements the encoding method according to claim 15.

20. A computer program product comprising a program that, when executed by a computer, implements the decoding method according to claim 16.

Images