Independent placement of auxiliary information in neural network-based picture processing.

Abstract

The present application provides a method and apparatus for processing picture data or picture feature data using a neural network with two or more layers. The present disclosure may be applied in the field of artificial intelligence (AI)-based video or picture compression technology, in particular in the field of neural network-based video compression technology. According to some embodiments, two types of data are combined during processing, including processing by the neural network. The two types of data are obtained from different stages of processing by the network. Some of the advantages may include higher scalability and more flexible design of the neural network architecture, which may further lead to better encoding / decoding performance.

Description

[Technical Field]

[0001] Embodiments of the present invention relate to the field of artificial intelligence (AI)-based video or picture compression techniques, and in particular to methods and apparatus for processing picture feature data to generate a bitstream or for processing picture feature data using a neural network to decode it into a bitstream. [Background technology]

[0002] Video coding (video encoding and decoding) is used in a wide range of digital video applications, such as broadcast digital TV, video transmission over the Internet and mobile networks, real-time conversation applications such as video chat and video conferencing, DVD and Blu-ray® discs, video content acquisition and editing systems, and camcorders for security applications.

[0003] The amount of video data required to depict even a relatively short video can be substantial, resulting in difficulties when the data is streamed or otherwise communicated over communication networks with limited bandwidth capacity. Therefore, video data is typically compressed before being communicated over modern telecommunications networks. When video is stored on a storage device, the size of the video can also be an issue, as memory resources may be limited. Video compression devices often use software and / or hardware at the source to code the video data before transmission or storage, thereby reducing the amount of data required to represent a digital video picture. The compressed data is then received at the destination by a video decompression device, which decodes the video data. Limited network resources and the ever-increasing demand for higher video quality have created a demand for improved compression and decompression techniques that improve compression ratios with little or no sacrifice in picture quality.

[0004] In recent years, deep learning has become increasingly popular in the field of picture and video encoding and decoding. Summary of the Invention [Means for solving the problem]

[0005] The present application provides methods and apparatus that can improve the configurability of neural networks, thereby achieving greater efficiency.

[0006] These and other objects are achieved by the subject matter of the independent claims. Further implementations are evident from the dependent claims, the description and the drawings.

[0007] Particular embodiments are outlined in the accompanying independent claims, and other embodiments are outlined in the dependent claims.

[0008] According to a first aspect, the present disclosure relates to a method for processing feature data of one or more pictures from a bitstream using a neural network including two or more stages, including a first stage and a second stage. The method includes the steps of obtaining first data based on the bitstream, processing the first data using the neural network, and outputting a result of the processing. The processing includes the steps of obtaining second data from the second stage of the neural network, the second data being based on data previously processed by the neural network, and using the first data together with the second data to generate inputs to the first stage of the neural network, wherein the first stage precedes the second stage in the neural network's processing of the feature data.

[0009] Acquiring two types of data that are used together in one stage and acquiring at least one type of data in another stage further increases the flexibility of the neural network architecture and may result in greater efficiency in terms of reduced complexity or latency, or rate, or higher quality.

[0010] In a possible implementation, the first data is obtained in a third stage of the neural network that precedes the first and second stages of feature data processing in the neural network or is equal to the first stage.

[0011] Acquiring the first and second data at different stages may further improve efficiency.

[0012] In a possible implementation, the first data represents a prediction error and the second data represents a prediction.

[0013] The present disclosure is easily applicable to prediction error and prediction signal. By obtaining at least one of them at a resolution different from the resolution at which they are combined, complexity, bitstream length, and latency can be saved. Furthermore, in picture coding, since there is high correlation in the spatial and temporal domains, prediction is auxiliary data that can improve the performance of decoding (and encoding) using neural networks. Furthermore, there are many techniques known in the art that can be very efficient in providing prediction.

[0014] In a possible implementation, the prediction is obtained by obtaining reference feature data, which is the feature data output by the neural network in the second stage, obtaining prediction information including motion information or spatial information regarding the reference feature data based on the bitstream, and generating a prediction based on the reference feature data and the prediction information.

[0015] This implementation is an example of obtaining predictions with the help of additional prediction information that may help to further improve prediction quality.

[0016] In a possible implementation, the prediction error is obtained by processing the current picture with a neural network, the prediction information is motion information, and the reference feature data is generated by neural network processing of picture data representing a picture preceding the current picture in decoding order.

[0017] Temporal prediction is typically more efficient than spatial prediction or lossless coding. Therefore, the above neural network architecture can further improve decoding (as well as encoding) performance.

[0018] In a possible implementation, using the first data together with the second data includes element-wise addition of the prediction or rescaled prediction and the prediction error or rescaled prediction error.

[0019] Such combinations may be particularly relevant to processing picture or picture feature data, predictions, and residuals.

[0020] In a possible implementation, the motion information includes motion vectors.

[0021] Motion vectors are an efficient means for indicating motion information, and there are many available techniques for their acquisition and signaling that can be readily applied to the embodiments described herein.

[0022] In a possible implementation, the second stage is the output of a neural network.

[0023] The output of the neural network is the reconstructed picture (on the decoding side). Obtaining second data from this stage can provide full resolution and improve quality.

[0024] In a possible implementation, the first stage is the input of a neural network.

[0025] The input of the neural network on the decoding side is the encoded bitstream. The embodiments of the present disclosure can be effectively applied to the decoding of bottleneck feature data.

[0026] In a possible implementation, the first stage is an input of a neural network, the first data is entropy-encoded data, the second stage is a stage different from the output stage of the neural network, and the second data is probability model data related to the feature data of the second stage, for example, the second data is probability model data for entropy decoding of the first data.

[0027] The collection of probability model data may enable further reduction in bitstream size. Obtaining probability model data from different stages can provide a flexible architecture for better tradeoff between performance and complexity.

[0028] In a possible implementation, the positions of the first stage and / or the second stage are configurable within the neural network, and the method includes a step of configuring the positions of the first stage and / or the second stage according to collection conditions based on one or more picture coding parameters.

[0029] The ability to configure the location of the first and second stages within the neural network provides additional flexibility, which may allow for dynamic neural network architecture changes. Such flexibility may result in better adaptation based on acquisition conditions, leading to more efficient encoding and / or decoding.

[0030] The collection condition is a condition or prerequisite that must be met for the second data and / or the first data to be input into a particular processing stage. The collection condition may include a comparison of some picture characteristics or picture feature characteristics with a threshold to determine whether to collect auxiliary information for a particular location. The picture characteristics or picture feature characteristics may be known to the encoder and decoder so that no additional signaling is required. Alternatively, or in addition, the collection condition may be configured by the encoding side by setting an indicator of whether auxiliary information should be collected for a preconfigured location. The indicator may be provided in a bitstream available to the decoder.

[0031] In particular, the method further comprises the step of parsing a stage selection indicator from the bitstream that specifies the first stage and / or the second stage, and the positions of the first stage and the second stage within the neural network are configured according to the parsed stage selection indicator.

[0032] According to a second aspect, the present invention relates to a method for processing at least one picture using a neural network including two or more stages, including a first stage and a second stage, to generate a bitstream, the method comprising the steps of: obtaining first data based on the at least one picture; obtaining second data in the second stage of processing, the second data being based on data previously processed by the neural network; and using the first data together with the second data to generate input for the first stage of the neural network, the first stage preceding the second stage in the neural network's processing of feature data. The method further comprises the step of inserting the feature data obtained by the processing into the bitstream.

[0033] The encoding portion of this disclosure may provide the same advantages as those described above for the decoding portion: the encoder prepares and provides a bitstream to the decoder so that the decoder can decode or reconstruct the data with the desired quality and application in mind.

[0034] To avoid redundancy, the advantages provided in the corresponding decoding claims apply to encoding in a similar manner.

[0035] In a possible implementation, the first data is obtained in a third stage of the neural network that precedes the first and second stages of feature data processing in the neural network or is equal to the first stage.

[0036] In a possible implementation, the first data represents a prediction error and the second data represents a prediction.

[0037] In a possible implementation, the prediction is obtained by obtaining reference feature data, which is the feature data output by the neural network in the second stage, obtaining prediction information including motion information or spatial information regarding the reference feature data based on at least one picture, generating a prediction based on the reference feature data and the prediction information, and inserting the obtained prediction information into the bitstream.

[0038] In a possible implementation, the prediction error is obtained by processing the current picture using a neural network, the prediction information is motion information, and the reference feature data is generated by neural network processing of picture data representing a picture preceding the current picture in decoding order.

[0039] In a possible implementation, using the first data together with the second data may include using the first data or is again It involves element-wise subtraction of the prediction or rescaled prediction from the scaled first data.

[0040] In a possible implementation, the motion information includes motion vectors.

[0041] In a possible implementation, the second stage is the output of a decoding neural network representing the reconstructed picture data.

[0042] In a possible implementation, the first stage is the output of a neural network.

[0043] In a possible implementation, the first stage is the output of a neural network, the first data is processed data to be entropy encoded, the second stage is a stage different from the input stage of the neural network, and the second data is probabilistic model data related to the feature data of the second stage.

[0044] In a possible implementation, the second data is probability model data for entropy encoding of the first data.

[0045] In a possible implementation, the positions of the first stage and / or the second stage are configurable within a neural network, and the method includes a step of configuring the positions of the first stage and / or the second stage according to collection conditions based on one or more picture coding parameters.

[0046] In a possible implementation, the method further includes determining and including in the bitstream a stage selection indicator that designates the first stage and / or the second stage, and the positions of the first stage and the second stage within the neural network are configured according to the determined stage selection indicator.

[0047] In a possible implementation, the determination of the stage selection indicator is based on an optimization procedure performed according to a cost function including one or more of rate, distortion, latency, accuracy, and complexity.

[0048] Determining the stage based on a cost function can improve the fit of the neural network and the results to the desired requirements. Thus, such optimization can improve performance. In combination with indicating the stage positions obtained in this way, flexibility is further improved.

[0049] According to a third aspect, the present disclosure relates to an apparatus for processing feature data of one or more pictures from a bitstream using a neural network including two or more stages, including a first stage and a second stage, the apparatus including a processing circuit configured to obtain first data based on the bitstream, process the first data using the neural network, and output a result of the processing, the processing including obtaining second data from the second stage of the neural network based on data previously processed by the neural network, and using the first data together with the second data to generate input to the first stage of the neural network, wherein the first stage precedes the second stage in the neural network's processing of the feature data.

[0050] For advantageous effects of the present disclosure, please refer to the description of the first aspect. Details will not be described again here. The decoding device has a function of implementing the actions in the example method of the first aspect. The functions may be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. In a possible implementation form, the decoding device includes a bitstream decoding module for obtaining first data based on a bitstream, and a neural network module configured to perform the above-mentioned processing and output. These modules may perform corresponding functions in the example method of the first aspect. For details, please refer to the detailed description in the example method. Details will not be described again here.

[0051] According to a fourth aspect, the present disclosure relates to an apparatus for processing at least one picture using a neural network including two or more stages, including a first stage and a second stage, to generate a bitstream, the apparatus including a processing circuit. The processing circuit is configured to process the at least one picture using the neural network. The processing includes an operation of obtaining first data based on the at least one picture, an operation of obtaining second data in the second stage of processing, where the second data is based on data previously processed by the neural network, and an operation of using the first data together with the second data to generate input for the first stage of the neural network, where the first stage precedes the second stage in the neural network's processing of feature data. The method further includes an operation of including feature data obtained by the processing in the bitstream.

[0052] For advantageous effects of the present invention, please refer to the description of the second aspect. Details will not be described again here. The encoding device has functions for implementing the actions in the example method of the second aspect. The functions may be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. In a possible implementation form, the encoding device includes a neural network module configured to perform the above-mentioned processing, and a bitstream output module configured to process the picture using the neural network, insert the obtained data into a bitstream, and output the bitstream. These modules may perform corresponding functions in the example method of the second aspect. For details, please refer to the detailed description in the example method. Details will not be described again here.

[0053] The method according to the first aspect of the invention may be performed by an apparatus according to the third aspect of the invention. Other features and implementations of the method according to the first aspect of the invention depend directly on the functionality and implementation of the apparatus according to the third aspect of the invention.

[0054] The method according to the second aspect of the invention may be performed by an apparatus according to the fourth aspect of the invention. Other features and implementations of the method according to the second aspect of the invention depend directly on the functionality and implementation of the apparatus according to the fourth aspect of the invention.

[0055] According to a fifth aspect, the present invention relates to a bitstream decoding device comprising a processor and a memory, the memory storing instructions for causing the processor to perform the method according to the first aspect.

[0056] According to a sixth aspect, the present invention relates to a bitstream encoding device comprising a processor and a memory, the memory storing instructions for causing the processor to perform the method according to the second aspect.

[0057] According to a seventh aspect, there is proposed a computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to encode video data, the instructions causing the one or more processors to perform a method according to the first or second aspect or any possible embodiment of the first or second aspect.

[0058] According to an eighth aspect, the present invention relates to a computer program product comprising program code for performing the method according to the first or second aspect or any possible embodiment of the first or second aspect when the program code is executed on a computer.

[0059] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, drawings, and claims.

[0060] In the following, embodiments of the invention will be described in more detail with reference to the accompanying figures and drawings. [Brief explanation of the drawings]

[0061] [Figure 1] FIG. 1 is a block diagram illustrating an exemplary network architecture on the encoder and decoder sides including a hyper-prior model. [Figure 2] FIG. 1 is a block diagram illustrating a general network architecture on the encoder side including a hyper-prior model. [Figure 3] FIG. 1 is a block diagram showing a general network architecture on the decoder side including a hyper-prior model. [Figure 4] 1 is a schematic diagram illustrating the general scheme of a neural network based encoder and decoder; [Figure 5] FIG. 2 is a block diagram illustrating an example autovariant encoder and decoder. [Figure 6] FIG. 1 is a general block diagram illustrating the application of auxiliary information at predefined stages of encoding and decoding neural networks. [Figure 7] 10A and 10B are visualizations of the predicted signal in the signal domain after processing with one processing layer including downsampling and after processing with N layers including downsampling. [Figure 8] FIG. 1 is a block diagram illustrating an example video coding pipeline that combines a traditional block-based codec with a deep learning-based approach. [Figure 9] FIG. 1 is a block diagram illustrating an example configurable conditional autoencoder with multiple selectable positions for collecting auxiliary information. [Figure 10] FIG. 1 is a block diagram illustrating an example configurable conditional autoencoder with one selectable position for collecting auxiliary information. [Figure 11]FIG. 1 is a block diagram illustrating an exemplary neural network process including multiple collection opportunities (positions) for collecting prediction information and residual information as auxiliary information. [Figure 12] FIG. 1 is a schematic diagram illustrating a comparison of performance for downsampling and reconstructing two example pictures with different characteristics. [Figure 13] FIG. 10 is a block diagram illustrating the application of a configurable collection unit for the purpose of inputting indicators of a probabilistic model as auxiliary information. [Figure 14] FIG. 1 is a block diagram illustrating an encoding-side and decoding-side neural network with an exemplary numbering of the processing stages (or layers). [Figure 15] FIG. 1 is a block diagram showing an encoding and decoding neural network with both a prediction and a residual in position 0 of the pipeline. [Figure 16] FIG. 1 is a block diagram showing an encoding and decoding neural network with both a prediction and a residual in the first position of the pipeline. [Figure 17] FIG. 1 is a block diagram showing an encoding and decoding neural network with a prediction at the i-th stage of the pipeline and a residual at the (i+r)-th stage. [Figure 18] FIG. 1 is a block diagram showing an encoding and decoding neural network with prediction at stage 0 of the pipeline and residual at stage K. [Figure 19] FIG. 1 is a block diagram showing an encoding and decoding neural network with predictions in the first stage of the pipeline and residuals in the Kth stage. [Figure 20] FIG. 1 is a block diagram showing an encoding and decoding neural network with predictions in the first stage of the pipeline and residuals in the (K−1)th stage. [Figure 21] FIG. 10 is a block diagram illustrating a possible handling of residuals and predictions according to a first exemplary implementation. [Figure 22]FIG. 10 is a block diagram illustrating a possible handling of residuals and predictions according to a second exemplary implementation. [Figure 23] FIG. 1 is a block diagram illustrating an exemplary overview architecture for handling prediction and residual in the signal domain. [Figure 24] FIG. 1 is a block diagram illustrating an exemplary overview architecture for handling predictions and residuals in latent space. [Figure 25] FIG. 1 is a block diagram illustrating an exemplary general architecture for handling predictions and residuals in a latent space, where the latent space of the prediction is from processing a previous picture. [Figure 26] FIG. 1 is a block diagram illustrating an example of a video coding system configured to implement embodiments of the present invention, where the system uses a neural network to encode or decode video pictures. [Figure 27] FIG. 10 is a block diagram illustrating another example of a video coding system configured to implement embodiments of the present invention, where a video encoder and / or video decoder uses a neural network to encode or decode video pictures. [Figure 28] 1 is a block diagram illustrating a video coding device configured to implement an embodiment of the present invention. [Figure 29] 1 is a block diagram illustrating a video coding device configured to implement an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0062] The embodiments of the present application provide an AI-based video picture compression technique, and in particular, a neural network-based video compression technique.

[0063] Video coding typically refers to the processing of a sequence of pictures, which form a video or video sequence. In the field of video coding, the terms "picture," "frame," and "image" are sometimes used synonymously. Video coding (or coding in general) includes two parts: video encoding and video decoding. Video encoding occurs at the source side and typically involves processing the original video picture (e.g., by compression) to reduce the amount of data needed to represent the video picture (for more efficient storage and / or transmission). Video decoding occurs at the destination side and typically involves the reverse process compared to the encoder's process to reconstruct the video picture. Embodiments referring to "coding" a video picture (or pictures in general) should be understood to relate to "encoding" or "decoding" the video picture or respective video sequence. The combination of the encoding and decoding parts is also called a CODEC (encoding and decoding).

[0064] In the case of lossless video coding, the original video picture can be reconstructed. In other words, the reconstructed video picture has the same quality as the original video picture (assuming no transmission loss or other data loss occurs during storage or transmission). In the case of lossy video coding, further compression is performed, for example by quantization, to reduce the amount of data required to represent the video picture, and the video picture cannot be completely reconstructed at the decoder side. In other words, the quality of the reconstructed video picture is lower or inferior to the quality of the original video picture.

[0065] Some H.26x video coding standards (e.g., H.261, H.263, H.264, H.265, H.266) are used for "lossy hybrid video coding" (i.e., spatial and predictive prediction in the sample domain are combined with 2D transform coding to apply quantization in the transform domain). Each picture of a video sequence is typically divided into a set of non-overlapping blocks, and coding is typically performed at the block level. Specifically, at the encoder side, video is usually processed, i.e., encoded, at the block (video block) level. For example, a predictive block is generated through spatial (intra-picture) prediction and temporal (inter-picture) prediction, and the predictive block is subtracted from a current block (the block being processed or the block to be processed) to obtain a residual block. The residual block is transformed and quantized (compressed) in the transform domain to reduce the amount of data to be transmitted. At the decoder side, the inverse processing portion to the encoder is applied to the encoded or compressed block to reconstruct the current block for representation. Additionally, the encoder replicates the decoder processing loop so that both generate the same prediction (eg, intra-prediction and inter-prediction) and / or reconstruction for processing, i.e., coding, subsequent blocks.

[0066] The present disclosure relates to processing picture data using neural networks for the purposes of encoding and decoding the picture data, which may still refer to or include some components known from the framework of the above-mentioned standards.

[0067] The following provides a brief introduction to some of the terms used in this specification.

[0068] Picture Size: Refers to the width, or height, or width and height pair of a picture. The width and height of an image are usually measured in number of luminance samples.

[0069] Downsampling: Downsampling is a process in which the sampling rate of a discrete input signal is reduced. For example, if the input signal is an image with sizes h and w, the output of downsampling is h and w, where at least one of the following is true: h2 <h w2 <w

[0070] In one exemplary implementation, downsampling may be implemented to retain only every mth sample and discard the remainder of the input signal (e.g., an image). However, downsampling may be performed by other means, such as convolution or other filtering.

[0071] Upsampling: Upsampling is a process in which the sampling rate of a discrete input signal is increased. For example, if the input image has sizes h and w, the output of downsampling is h and w, where at least one of the following is true: h <h2 w <w2

[0072] Resampling: The downsampling and upsampling processes are both examples of resampling. Resampling is a process in which the sampling rate (sampling interval) of an input signal is changed.

[0073] Sampling ratio: When the picture size is changed during the resampling process (upsampling or downsampling), the ratio between the output picture size and the input picture size is called the sampling ratio. The sampling ratio may differ in the horizontal and vertical dimensions.

[0074] Interpolation filtering: During the upsampling or downsampling process, filtering may be applied to improve the accuracy of the resampled signal and reduce the effects of aliasing. An interpolation filter typically involves a weighted combination of sample values at sampling positions surrounding the resampling position. This f(x r ,y r )=Σs(x,y)C(k) where f() is the resampled signal and (x r ,y r ) are coordinates in the resampled image, C(k) are the interpolation filter coefficients, and s(x,y) are samples of the input signal. The summation operation is r ,y r ) for (x,y) in the neighborhood of

[0075] Cropping: Trimming the outer edges of a digital image. Cropping can be used to make an image smaller (in number of samples) and / or to change the aspect ratio (length vs. width) of an image.

[0076] Padding: Padding refers to increasing the size of an image by generating new samples (usually at the borders of the image), for example, by using predefined sample values or by using sample values of positions within the image.

[0077] Resizing: Resizing is a general term where an input image is resized. This may be done using one of the methods of padding or cropping, or it may be done by resampling.

[0078] Integer division: Integer division is division in which the fractional part (remainder) is discarded.

[0079] Convolution: Convolution is given by the following general formula: where f() may be defined as the input signal and g() may be defined as the filter.

[0080]

number

[0081] This convolution is a discrete convolution, where n and m are sampling indices and therefore integers. The range of indices n and m may depend on the signal size (dimension) and filter size. In general, they can theoretically range from minus infinity to infinity.

[0082] NN module: A neural network module is a component of a neural network. It can be a layer or a sub-network within a neural network. A neural network is a sequence of one or more NN modules.

[0083] Position in Pipeline (Stage): Specifies a position in a processing pipeline within a processing network that may be or include a neural network. A position in a network may be specified, for example, by the number of NN modules (or layers) preceding that position. Applying a transformation at the i-th position in a pipeline means applying the transformation to the output of the i-th NN module and using the result of the transformation as the input of the (i+1)-th NN module. Applying a transformation at the 0-th position in a pipeline is interpreted as applying the transformation to the input of a NN. Applying a transformation at the K-th position in a pipeline is interpreted as applying the transformation to the output of a NN.

[0084] Latent Space: An intermediate step in neural network processing, the latent space representation contains the outputs of the input or hidden layers, which are usually not intended to be displayed.

[0085] Lossy NN Module: Information processed by a lossy NN module results in information loss, and the lossy module cannot restore the processed information.

[0086] Lossless NN module: Information processed by a lossless NN module does not result in information loss, and lossless processing allows the processed information to be restored.

[0087] Bottleneck: A latent space tensor that can be input to a lossless coding module. The term bottleneck relates to the fact that the channel size is usually smaller than the previous stage.

[0088] NN layer: A processing step in a neural network that performs one processing operation on input data.

[0089] Variational Autoencoder (VAE) Exemplary deep learning-based image and video compression algorithms follow the variational autoencoder (VAE) framework, e.g., Z. Cui, J. Wang, B. Bai, T. Guo, and Y. Feng, "G-VAE: A Continuously Variable Rate Deep Image Compression Framework," arXiv preprint arXiv:2003.02012, 2020.

[0090] Figure 1 illustrates a VAE framework. The VAE framework can be thought of as a nonlinear transform coding model. On the encoder side of the network, an encoder 1 maps an image x to a latent representation via a function y = f(x). The encoder includes or consists of a neural network. A quantizer 2 converts the latent representation into a discrete value y_hat = Q(y) of a desired bit length and / or precision. The quantized signal (latent space) y_hat is included in a bitstream (bitstream 1) using arithmetic coding, denoted AE, which stands for Arithmetic Encoder 5.

[0091] On the decoder side of the network, the encoded latent space is decoded from the bitstream by an arithmetic decoder AD6. Decoder 4 converts the quantized latent representation output by AD6 into a decoded image x_hat=g(y_hat). Decoder 4 includes or consists of a neural network.

[0092] In Figure 1, two sub-networks are connected to each other. The first network includes the above-mentioned processing units 1 (encoder 1), 2 (quantizer), 4 (decoder), 5 (AE), and 6 (AD). At least units 1, 2, and 4 are called an autoencoder / decoder, or simply an encoder / decoder network.

[0093] The second sub-network, which includes at least units 3 and 7, is called a hyperencoder / decoder or context modeler. Specifically, the second sub-network models the probability model (context) for AE5 and AD6. The entropy model, or in this case, the hyperencoder 3, estimates the distribution z of the quantized signal y_hat that approaches the minimum rate achievable with lossless entropy source coding. The estimated distribution is quantized by the quantizer 8 to obtain the quantized probability model z_hat, which represents side information that can be transmitted to the decoder in the bitstream. To do so, the arithmetic encoder AE9 may encode the probability model into bitstream 2. The bitstream 2 may be transmitted to the decoder side together with bitstream 1 and also provided to the encoder. Specifically, to be provided to AE5 and AD6, the quantized probability model z_hat is arithmetically decoded by AD10, then decoded using the hyperdecoder 7, and inserted into AD6 and AE5.

[0094] Figure 1 shows the encoder and decoder in a single diagram. On the other hand, since the encoder and decoder may operate separately, Figures 2 and 3 show the encoder and decoder separately. In other words, the encoder may generate bitstream 1 and bitstream 2. The decoder may receive such a bitstream from storage, over a channel, etc., and decode it without further communication with the encoder. The above description of the encoder and decoder elements also applies to Figures 2 and 3.

[0095] Many deep learning-based image / video compression systems reduce the dimensionality of a signal before converting it into binary digits (bits).

[0096] In the VAE framework, for example, an encoder, which is a nonlinear transform, maps an input image x to y, where y has a smaller width and height than x. Because y has a smaller width and height and therefore a smaller size, the dimension of the signal is reduced, and therefore it becomes easier to compress the signal y.

[0097] The general principle of compression is illustrated in Figure 4. The input signal x corresponds to the input data, which is the input of the encoder. The transformed signal y corresponds to the latent space, which has smaller dimensionality than the input signal and is therefore also called the bottleneck. Typically, the dimensionality of the channel is minimized at this processing position in the encoder-decoder pipeline. Each row of circles in Figure 4 represents a layer in the encoder or decoder processing chain. The number of circles in each layer indicates the size or dimensionality of that layer. The latent space, which is the output of the encoder and the input of the decoder, represents the compressed data y. At the decoder side, the latent space signal y (encoded image) is processed by the decoder neural network, which leads to expanding the dimensionality of the channel until we obtain reconstructed data x_hat, which may have the same dimensionality as the input data x but may differ from the input data x, especially if lossy processing is applied. The dimensionality of the channel processed by the decoder layer is typically higher than the y-dimension of the bottleneck data. In other words, the encoding operation usually corresponds to a reduction in the size of the input signal, and the decoding operation corresponds to a reconstruction of the image in its original size, hence the name bottleneck.

[0098] As mentioned above, signal size reduction can be achieved by downsampling or rescaling. The reduction in signal size typically occurs incrementally along a chain of processing layers, rather than all at once. For example, if an input image x has dimensions h and w (denoting height and width) and a latent space y has dimensions h / 16 and w / 16, size reduction may occur in four layers during encoding, where each layer reduces the size of the signal by a factor of two in each dimension.

[0099] Known deep learning-based video / image compression methods typically use multiple downsampling layers. An exemplary VAE is shown in FIG. 5 and utilizes six downsampling layers, marked 51 through 56. Layers that include downsampling are indicated by downward arrows in the layer description. The layer description "Conv N×5×5 / 2↓" means that the layer is a convolutional layer ("Conv") with N channels and the convolution kernel is 5×5 samples in size. The "2↓" indicates that downsampling by a factor of 2 occurs in this layer. Downsampling by a factor of 2 results in one of the dimensions of the input signal being reduced by half at the output. In FIG. 5, the 2↓ indicates that both the width and height of the input image are reduced by a factor of 2. There are six downsampling layers that halve the size in one dimension, so if the width and height of the input image x (514) are given by w and h, then the output signal z_hat (513) has a width and height equal to w / 64 and h / 64, respectively.

[0100] In the first sub-network, several convolutions are followed by a generalized division normalization (GDN) on the encoder side and an inverse GDN (IGDN) on the decoder side. In the second sub-network, the activation function applied is a rectified linear unit (ReLU). Note that the present disclosure is not limited to such an implementation, and in general, other activation functions may be used instead of GDN or ReLU.

[0101] The network architecture in Figure 5 includes a hyper-prior model. a , g s ) shows the image autoencoder architecture, and the right side (h a , h s ) corresponds to an autoencoder that implements a hyperprior as explained above with reference to Figures 1 to 3. The model is based on the analysis and synthesis transformation g a and g sAs in Figures 1 to 3 described above, Q represents quantization, and AE and AD represent arithmetic encoder and decoder, respectively. The encoder applies g to the input image x. a , resulting in a latent representation y with a spatially varying standard deviation.

[0102] The result is h a is fed to summarize the distribution of standard deviations in z. z is then quantized, compressed, and transmitted as side information. The encoder calculates the spatial distribution of standard deviations, which is used to obtain probability values (or frequency values) for arithmetic coding (AE).

[0103]

number

[0104] The decoder uses the quantized vector z_hat 513 to estimate y_hat, and uses that estimate to compress and transmit the quantized latent representation y_hat 515. The decoder first recovers z_hat from the compressed signal. The decoder then uses h to obtain y_hat. s provides a correct probability estimate for successfully recovering y_hat. The decoder then uses g to obtain the reconstructed image. s Supply to.

[0105] The decoder includes upsampling layers 57-59 and 510-512. A further layer 420, implemented as a convolutional layer but which does not provide upsampling to the received input, is provided between upsampling layers 411 and 410 in the input processing order. A corresponding convolutional layer "conv Mx3x3 / 1" is also shown for the decoder. Such layers do not change the size of the input, but are provided in the NN to perform operations on the input that change certain characteristics. However, such layers need not be provided.

[0106] When viewed in the processing order of bitstream 2 through the decoder, the upsampling layers proceed in reverse order, i.e., from upsampling layer 512 to upsampling layer 57. Each upsampling layer is shown here as providing upsampling with an upsampling ratio of 2, indicated by a ↑. Of course, not all upsampling layers have the same sampling ratio; other upsampling ratios, such as 3, 4, or 8, may also be used. Layers 57 through 512 are implemented as convolutional layers (conv). Specifically, because upsampling layers are intended to provide the inverse operation on the input to that of the encoder, upsampling layers may apply a deconvolution operation to the received input such that the size of the received input is increased by a factor corresponding to the upsampling ratio. However, this disclosure is not limited to deconvolution; upsampling may be performed in any other manner, such as bilinear interpolation between two adjacent samples or nearest neighbor sample copying.

[0107] FIG. 6 shows a general block diagram of a deep learning-based picture (still or video) codec with auxiliary information applied after the first layer. The deep learning architecture can be any architecture, including a conditional autoencoder and a hyperprior. A conditional VAE means that the network is conditioned by collecting additional information, referred to herein as auxiliary information. Such auxiliary information, along with feature data, is processed by one or more layers of the NN. For example, the input 602 of a picture encoder is an image, which may be provided in multiple channels. The exemplary representation in FIG. 6 shows three color channels (e.g., YUV color space) with luminance and two subsampled chrominances, as is one common approach in picture coding. In general, there may be more channels (e.g., additional colors such as white, or depth or other features, such as motion) or fewer channels (e.g., only one grayscale channel, black and white, or a color palette). The input picture channels 602 are arranged into an input tensor x and input to the encoder neural network 600A. The encoder neural network 600A may include multiple layers or layer stages, which may include downsampling layers that may be implemented, for example, by convolutions with or without strides, nonlinear activations such as sigmoid, ReLU, or GDN, several masked convolutions that do not change the dimensionality of the tensors, etc. Note that the layers may also include entropy encoding or other processing layers.

[0108] The input signal X may be processed by one or more layers, including convolution ("conv") and downsampling ("↓"), along with some nonlinear operations (NonLinAct) and masked convolution (MaskConv). The encoder may include two or more layers and write the compressed information in the latent space to one or more bitstreams ("----") 620 and / or 630. For example, one of the bitstreams (e.g., bitstream 620) may correspond to bitstream 1, and the other (e.g., bitstream 630) may correspond to bitstream 2 shown in Figures 1 through 3. In Figure 6, the stages or layers are shown as downsizing (narrower in the processing direction). However, this is for illustrative purposes, and in general, the network may also include layers that do not change the tensor size and / or layers that increase the tensor size. Layer 1 may be followed by gathering 610, or auxiliary information gathering. The gatherer 610 may collect auxiliary information and combine it with the output data of Layer 1. The combined information is then input into Layer 2 for further processing.

[0109] Correspondingly, the decoder portion 600B of the network analyzes one or more bitstreams 620 and / or 630 and reconstructs the signal using one or more layers of (de)convolution, upsampling ("↑"), nonlinear operations, and masked convolution. This exemplary decoder network 600B is symmetrical to the encoder network 600A (although this is not generally necessary). A collector 650 collects auxiliary information and applies a combination of the collected auxiliary information and the output of the penultimate layer to the final decoding layer (symmetrical to layer 1 of the encoder). After processing by the final layer, an output tensor is obtained, in this example a reconstructed picture 692.

[0110] Auxiliary information in the encoding / decoding pipeline In picture codecs, auxiliary information may be used to facilitate the encoder and decoder. The auxiliary information may be applied at one or more specific positions in the coding (encoding and / or decoding) pipeline along with the encoded or decoded tensors. In the example of FIG. 6, the auxiliary information is used to assist in encoding and decoding the processed signal after the first layer. Because the network architecture is primarily predefined and fixed for a specific application (picture encoding and decoding), the application of the auxiliary information is typically tied to a specific, fixed position in the processing pipeline, as shown by collection blocks 610 and 650 in FIG. 6. The parameters (e.g., weights) applied in the neural network are usually pre-trained (e.g., provided before encoding and / or decoding). Therefore, because the weights and positions for applying the auxiliary information are predetermined, the architecture of the neural network is predetermined. This may limit the flexibility of the network and its adaptation to various parameters and characteristics of the encoded or decoded content.

[0111] A possible example of auxiliary information is inter-predicted frames (predictions) applied in a region of the input signal x. Inter-prediction exploits temporal redundancy between adjacent frames in a video and the current frame, since the content of adjacent frames usually does not change completely. Instead, the content usually moves only slightly between frames. Therefore, inter-prediction requires two pieces of information: one or more adjacent frames as reference pictures, and motion information (e.g., motion vectors). Inter-prediction takes these two as inputs and applies motion compensation to generate a prediction for the current frame.

[0112] Another form of prediction is intra-prediction within a frame (picture). Intra-prediction exploits spatial redundancy within a frame. Given an area within a frame, sample values in the area are usually correlated. Therefore, intra-prediction uses reconstructed neighboring samples of a frame to predict the value of a sample at a current position within the same frame.

[0113] In Figure 7, we provide a visual example of prediction as supplementary information. The image on the top left is a predicted frame 710 from the common test sequence BasketballDrill. The prediction is made in the same domain as the input signal x, i.e., in this example, the NN input picture domain. The original width and height of the luma frame are denoted as W and H, respectively. The smaller frame 720 located in the bottom left corner is the same predicted frame, but after one convolutional layer conv(1,↓2) with one channel and downsampling by 2. Therefore, both the width and height are half the original resolution, i.e., W / 2 and H / 2. On the right, an example of prediction in the latent space domain after N convolutional layers, with a downsampling factor of 2 for each convolutional layer, is shown. Here, the width and height of the prediction are therefore 2 N The figure shows a grid of downsampled frames, where the grid size is determined by the number of channels 740, K, where there are a total of K = 13 * 11 = 143 channels. In other words, the mosaic picture 730 shows the side information (here, predictions) in the latent space domain after n convolutional layers with K channels, N * conv(K, ↓2).

[0114] Figure 8 shows a combined approach using both traditional block-based video codec (HEVC) and deep learning to capture various frame characteristics, according to Lu, Z. Ma, L. Xu, and D. Wang, "Tests on Decomposition, Compression, Synthesis (DCS)-based Technology," JVET-U0096, Joint Video Experts Team (JVET) 21st Meeting by teleconference, January 6-15, 2021. Auxiliary information, such as prediction, is used at multiple locations within the coding pipeline. The coding pipeline in Figure 8 can be divided into three stages: decomposition, compression, and synthesis. In the figure, ↓ represents downsampling, ↑ represents upsampling, and E and D represent the HEVC video encoder and decoder, respectively. STF represents a spatial texture frame, which is an intra-predicted frame within a group of pictures (GOP). TMF represents a temporal texture frame, which is an inter-predicted frame within a GOP. The prediction and residual coding of the HEVC encoder and decoder are located at static (fixed) locations within the HEVC video codec pipeline.

[0115] In Figure 8, the input GOP is first classified into two categories, STF and TMF, in the decomposition stage. The motion information in TMF can be captured at a coarse granularity and is therefore downsampled before being encoded.

[0116]

number

[0117] (i.e., the low-resolution input for the TMF). In comparison, the STF is

[0118]

number

[0119] The STF and downsampled TMF are encoded in the original high resolution with the notation ∂ ...

[0120]

number

[0121] The STF and TMF encoded information are written into the bitstream. They are then parsed and used by the HEVC decoder to reconstruct the STF and TMF. Again, when decoding the TMF, its reference frame is the STF.

[0122]

number

[0123] The outputs of the compression stage are, respectively,

[0124]

number

[0125] and

[0126]

number

[0127] , and the reconstructed signals of STF and TMF are denoted as

[0128] In the synthesis stage, the reconstructed STF and TMF are used to generate high-fidelity video at its native resolution. First, the temporally smooth and spatially fine-grained motion representation of the current TMF is obtained by aggregating information across neighboring TMFs.

[0129]

number

[0130] A motion compensation network (denoted as motion compensation) is utilized to generate the temporal motion features. Then, the temporal motion features generated by the motion compensation network are used to generate the decoded and upsampled TMF

[0131]

number

[0132] The decoded STF, along with its resampled version, is fed to a non-local texture transfer network to learn and transfer cross-resolution information for high-fidelity frame reconstruction with both spatial detail and temporal smoothness. The hybrid coding pipeline uses auxiliary information at various locations within the codec, such as temporal motion features generated by a motion compensation network, and prediction and residual information used in the HEVC encoder and decoder (not shown in Figure 8, but used in blocks E and D). However, the auxiliary information is applied at fixed locations within the coding pipeline.

[0133] Such implementations may not be able to effectively adapt to the characteristics of images and video sequences, and therefore produce suboptimal coding results.

[0134] A residual signal (also called a prediction error signal or residual signal) obtained by subtracting a prediction signal from an original signal may also be used as side information. In some implementations of the present disclosure, the prediction and residual signals may occur not only in the same domain as the input signal x, but also in a domain one or more layers later in the processing order of the encoder portion 600A or one or more layers earlier in the processing order of the decoder portion 600B. Conventionally, the location in the encoder and / or decoder where this side information is utilized along with the encoded and / or decoded tensors is fixed.

[0135] However, depending on the content, and in particular the characteristics of the input picture, the auxiliary information may not work well at a pre-designed fixed position in the coding pipeline. Therefore, utilizing the auxiliary information at a fixed position in the coding pipeline may lead to suboptimal coding results. Therefore, an adaptive solution that conditionally applies this auxiliary information at different positions based on the input characteristics may be desirable.

[0136] Selection of auxiliary information input location based on collection conditions According to one embodiment, a method is provided for processing picture feature data from a bitstream using a neural network. The neural network includes multiple (i.e., two or more) neural network layers, hereinafter also referred to as layers. Such a neural network 900B is illustrated in FIG. 9. The term picture feature data refers to any latent space data obtained by picture processing on the encoder side. In other words, the picture feature data may be the output of the encoding process.

[0137] The method includes obtaining picture feature data based on the bitstream 940. For example, the picture feature data may be directly parsed or parsed and decoded from the bitstream. Furthermore, the picture feature data may be obtained by some processing based on the decoded data. The method further includes processing the picture feature data using the neural network 900B. For each of one or more pre-configured positions in the neural network, the processing includes: a) determining whether to collect (960_1, 960_2, 960_3) auxiliary data 950 for processing by one of a plurality of neural network layers at a preconfigured position (following the preconfigured position in the feature data processing direction) based on collection conditions 910_1, 910_2, 910_3; b) if it is determined that auxiliary data 950 should be collected, processing by the layer at said preconfigured location is based on the auxiliary data 950; Includes:

[0138] A preconfigured location is a location in a neural network where the same auxiliary information can be collected. There may be one such location, or two or more (or all) such locations. In FIG. 9, there are three preconfigured locations: collection 960_1, 960_2, and 960_3. As mentioned above, a location in the network may be specified, for example, by the number of NN modules (here, layers) that precede that location in the order of processing picture feature data. A preconfigured location may be fixed as part of the neural network architecture. Nevertheless, flexibility may be improved by allowing or prohibiting auxiliary data to be input into one or more layers. However, this disclosure is not limited to fixed locations. Alternatively, a preconfigured location may be configurable by side information directly or indirectly included in the bitstream. Here, direct configuration may signal the actual location, while indirect configuration may be based on the same rules known in the encoder and decoder, for example, based on the values of encoding parameters (e.g., from the bitstream).

[0139] If it is determined that auxiliary data is not to be collected for processing by a particular layer, the processing by that layer is not based on the auxiliary data. In the exemplary architecture in FIG. 6, the location where the auxiliary data is input in the example of FIG. 6 is fixed and located after the first layer. In contrast, in FIG. 9, the auxiliary information can be inserted at multiple potential locations within the neural network. This is indicated by collection units 960_1, 960_2, and 960_3, which are controlled by conditions 910_1, 910_2, and 910_3 (labeled as question mark circles) throughout the neural network.

[0140] Similar processing occurs on the encoder side. An encoder-side method is provided for processing a picture 902 using a neural network 900A including multiple neural network layers to generate a bitstream 940. The method includes processing the picture 902 using the neural network 900A. The processing includes, for each of one or more preconfigured positions within the neural network, determining, based on collection conditions 910_1, 910_2, and 910_3, whether to collect auxiliary data 950 for processing by the layer at the preconfigured position (e.g., by respective collectors 920_1, 920_2, and 920_3). If it is determined that auxiliary data should be collected, processing by the layer at the preconfigured position is based on the auxiliary data 950. The method further includes inserting data obtained by processing the picture by the neural network into the bitstream 940. As seen in FIG. 9 , there may be an additional bitstream (or bitstream portion) 945 that may be provided when a hyper-apriori sub-network is applied to provide lossless encoding of probability model parameters.

[0141] In this way, the location at which the auxiliary information is applied can be dynamically changed depending on the content and / or characteristics of the images or frames in the video sequence. Thus, the configurable neural network has a dynamic architecture, since the application of the auxiliary information and the corresponding collection parameters can occur at different locations.

[0142] The term collection condition in this specification refers to, for example, a requirement that, when met, determines whether a collector collects auxiliary information 950. The requirement may be different for each collection unit 960_1, 960_2, and 960_3 (and each encoder-side collection unit 920_1, 920_2, and 920_3). Such a determination has the same effect as selecting a location where auxiliary information is input. Depending on the design of the requirement, auxiliary information may be selected for only one of the preconfigured locations. This can be achieved, for example, by providing mutually exclusive requirements (collection conditions) for each collector. In other words, as a result of applying the collection condition in the determination step, auxiliary data will be collected for a single location among one or more preconfigured locations.

[0143] However, the above examples do not limit the present disclosure. As shown in FIG. 9 , there is one auxiliary information 950 or one type of auxiliary information 950 that can be collected at a preconfigured location. In some exemplary implementations, applying the collection condition in the determining step results in the same auxiliary data being collected for two or more of the preconfigured locations. The above-described techniques for collecting auxiliary information can be applied to two or more types of auxiliary information. Then, for each auxiliary information type, the collection location can be determined from a set of preconfigured locations. Note that the set of preconfigured locations can be different for different types of auxiliary information. However, the sets can overlap or be the same for several types of auxiliary information. For example, the auxiliary information can be a prediction signal, a prediction error signal, a probability model, etc. Some of them can be applicable to the same or overlapping stages at corresponding locations, e.g., a prediction signal and a prediction error signal can be used together. Some of the auxiliary information types can be applicable to different stages, e.g., a probability model for entropy encoding a latent spatial signal.

[0144] When we refer to signals in this specification, we primarily mean discrete signals, such as input pictures to an encoding network or input picture feature data to a decoding network, or data representing the latent space at any stage, or auxiliary information.

[0145] As described above, different layers of a neural network may process feature tensors with different sizes. In picture processing, feature tensors typically have three dimensions: width, height, and number of channels, as discussed with reference to FIG. 7. A feature picture with width and height can be considered to correspond to a channel. Therefore, some resizing or rescaling may be performed to support the (selectable) collection of the same auxiliary information at multiple stages of the pipeline. For example, in an exemplary implementation, two or more of the preconfigured locations are preconfigured. The auxiliary data is scalable in size to match the dimensions of the input channels processed by the layers at two or more of the preconfigured locations. Therefore, in this exemplary implementation, as a result of applying the collection conditions in the determining step, auxiliary information (data) may be (i) collected or (ii) collected and scaled for a single location among the preconfigured locations. This is equivalent to selecting a single location from among the preconfigured locations where auxiliary information is collected and adapting the auxiliary information (if necessary) to the resolution processed by the layer to which the auxiliary information is to be input. However, the present disclosure is not limited to always applying auxiliary information at at least one position. It is also possible to design the collection conditions so that, as a result of applying the collection conditions, auxiliary information is not collected in any of the pre-configured information for several pictures. In other words, (iii) a third possibility of not collecting auxiliary information can be implemented.

[0146] The embodiments described herein focus on the selectable collection of auxiliary information. However, apart from such selectable collection, the neural network architecture may provide one or more collection stages in which certain types of auxiliary conditions are always collected (without relying on preconditions such as the satisfaction of collection conditions), as in traditional conditional VAE frameworks.

[0147] This embodiment allows adaptively changing the position where the auxiliary information is applied according to the collection conditions.

[0148] It should be noted that the present disclosure is not limited to a particular NN framework such as VAE. Furthermore, the present disclosure is not limited to image or video compression, but may also be applied to object detection, object recognition, or classification systems, generally any picture processing system, including the human visual system, where pictures are reconstructed for human visual purposes, or machine vision systems, where pictures are not necessarily reconstructed, but picture features suitable for deriving a desired output, such as a segmentation map, depth map, object classification, object detection, etc.

[0149] 10 illustrates an example embodiment in which the collection conditions are based on picture characteristics or picture feature data characteristics obtained from the bitstream. The neural network in FIG. 10 processes an input picture 1002 and outputs a processed picture 1092.

[0150] For example, the picture characteristics or picture feature data characteristics include resolution. The collection conditions include comparing the resolution with a preconfigured resolution threshold. For example, if the picture resolution exceeds the preconfigured threshold, the auxiliary data should be collected for a first location among the preconfigured locations. On the other hand, if the picture resolution does not exceed the preconfigured threshold, the auxiliary data should be collected for a second location different from the first location among the preconfigured locations. The threshold may be empirically defined as part of the design of the neural network architecture, or may be trained and / or configurable by including corresponding parameters in the bitstream.

[0151] In addition to or alternatively to the resolution-based condition, in an exemplary embodiment, the picture is a video picture, and the picture characteristics include a picture type. The collection condition includes determining whether the picture type is a temporally predicted picture type or a spatially predicted picture type. A spatially predicted picture is predicted using only samples within its own picture. In this case, samples from other pictures are not used. A temporally predicted picture must include some areas that are temporally predicted based on samples on other pictures. However, a temporally predicted picture may also include some areas that use spatial prediction, i.e., areas that use samples within its own picture.

[0152] Both the picture type and the picture resolution or other picture characteristics are known to both the encoder and the decoder so that both sides can operate in the same way without needing any additional side information.

[0153] 10, the first processing layer is a downsampling layer 1010 in the encoder 1010A and an upsampling layer 1070 in the decoder 1010B. The downsampling 1010 and upsampling 1070 in this example are a factor of 2 in both the width and height of the frame. Auxiliary information, such as prediction, can potentially be applied at two or more locations before or after the first layer 1010 in the encoder and the corresponding layer 1070 in the decoder.

[0154] On the encoder side, in this example, the collection unit 1030 is a weighted sum of two input signals: one of the full size of the original frame to be encoded, and the other of a quarter of the original frame size (full size) corresponding to a downsampled version of the original frame.

[0155] For example, a weight pair of (1, 0) indicates encoding using auxiliary information at full size (indicated by a weight factor of 1) rather than quarter size (indicated by a weight factor of 0), while a weight pair of (0, 1) indicates encoding using auxiliary information at quarter size rather than full size. In this way, the location at which auxiliary information is applied can be made configurable within the neural network. The conditions controlling where auxiliary information is used can be, for example, the frame size (resolution) and / or frame type of the original frame.

[0156] For example, the condition may be whether the frame size is 4K resolution or greater and whether the frame type is interframe. In an exemplary implementation, if the condition is true, a weight pair of (0, 1) is used, so that these input frames are encoded at quarter size using the auxiliary information. Other frames are encoded at their original resolution using the auxiliary information. The position at which the auxiliary information is applied may be dynamically changed at the frame level based on the frame type and / or frame size.

[0157] In other words, FIG. 10 illustrates an example in which the collection unit 1030 (and, accordingly, the decoder 1060) is simply a weighted sum. The weights can have values (1, 0) or (0, 1), as described above. A weight (1, 0) means that a full-size picture is encoded, while a weight (0, 1) means that coding is performed on a quarter-size signal corresponding to a signal downsampled by 2. The collection condition 1050 in this example is whether the video has a resolution of 4K or higher, and at the same time, whether the target frame is a temporally predicted frame. However, as mentioned above, this disclosure is not limited to such combined collection conditions. Rather, the thresholded resolution may only be whether the resolution exceeds a certain value or belongs to a certain set of resolutions. Alternatively, the frame type alone may be the evaluation condition, without considering the resolution. Further examples are possible, including other conditions used alone or in combination.

[0158] For example, coding parameters such as prediction mode, motion information (e.g., for deriving the intensity or type of motion), etc., may be applied in the collection conditions. The configurability of the neural network in the exemplary embodiments described above is achieved by switching auxiliary information on / off at specific locations based on content adaptation conditions. The content may include direct characteristics of an image, such as frame size and frame type, as shown in the example of FIG. 10. However, the characteristics may also be indirect characteristics of the content, such as a histogram of sample values in a given area. The granularity of the content may also be variable. For example, the conditions determining the location of auxiliary information may be updated at the frame level, block level, or the level of a group of pictures or any amount of pictures. In other words, in some embodiments, the neural network is trained to process video pictures. The determination of whether to collect auxiliary data for processing by a layer at the preconfigured locations is performed every predetermined number of video pictures, where the predetermined number of video pictures is one or more.

[0159] The condition may also have different forms. As explained below, the condition may be determined from an analyzed flag at the decoder. At the encoder, the flag value may be determined based on some RDO calculation or based on other cost functions. It is also possible that the condition is not explicitly signaled but is determined by other information of the image / frame in the video sequence.

[0160] One advantage of flexible locations for applying the auxiliary information is that pictures or frames can be adaptively encoded and decoded, which may result in better performance, which may include a lower rate for the resulting bitstream 1055 at the same quality, or vice versa (higher quality at the same rate).

[0161] According to one embodiment, the method comprises, at the decoder side, obtaining from the bitstream an indication specifying whether to collect auxiliary data for one or more preconfigured positions, the collection condition for each of the one or more preconfigured positions being: - the determination is positive if the instructions specify that auxiliary data should be collected for said preconfigured location; If the instructions specify that no auxiliary data should be collected for said preconfigured location, the decision is negative.

[0162] The indication may be a flag supported by the bitstream syntax that directly indicates whether to collect specific auxiliary information at a specific location. Alternatively, the indication may be included directly in the bitstream according to the syntax and specify a single location among preconfigured locations where auxiliary information should be collected. The indicator may also be designed to take a value indicating that auxiliary information should not be collected at any of the preconfigured locations. In this embodiment, the collection condition is based on evaluating the indicator.

[0163] The above-mentioned embodiments have been described with reference to auxiliary information. Generally, the auxiliary information (or auxiliary data) provides some (supplementary) information about the picture feature data that is processed by the neural network to generate the output. In a particular example also shown in Figure 11, the auxiliary data includes prediction data that is a prediction of the picture (e.g., in an input picture domain for the encoder) or a prediction of the picture feature data after processing by one or more of the layers of the neural network (e.g., other than the input picture domain for the encoder).

[0164] As shown in FIG. 11, in some exemplary implementations, the auxiliary data is a combined pair of predicted data and auxiliary data combined with the predicted data. The term "combined" here refers to the fact that the predicted and supplementary data are obtained from the same pipeline position (stage). The auxiliary data may be, for example, prediction error data (data representing a prediction error), which in certain cases may be a prediction residual (the difference between a predicted picture and a prediction of the picture). The predicted signal and residual signal may be obtained after zero, one, or several layers of the neural network, as shown in FIG. 11. Applying auxiliary information at the i-th position means using the auxiliary information for a transformation at the i-th position in the pipeline. The transformation here may be a general transformation performed by a layer, or an actual frequency-domain transformation performed as a processing step (layer) of the network, such as an orthogonal transform such as a Fourier transform (e.g., a fast Fourier transform (FFT) or a discrete cosine transform (DCT) or their integer versions). As mentioned above, the auxiliary information is generally a tensor that may have the same size as the encoder-side input signal or the latent space representation in the coding pipeline, depending on the collection position.

[0165] In general, the number of positions to which the prediction signal and the residual signal can be applied is N+1. This corresponds to position 0 at the decoder side, which represents the input signal (input to the NN), and corresponds to the output signal and N further positions 1 to N at the encoder. Note that pre-configured positions are not necessarily provided after each layer. Figure 11 shows an example in which the auxiliary information is the prediction signal and the residual signal. They can be collected at several positions, as shown, by providing multiple collection units 1150 in the encoder part 1100A and the decoder part 1100B of the neural network.

[0166] In the example of Figure 11, a single one of these locations applies the prediction signal and the residual signal during encoding and decoding. In other words, the locations that apply the side information do not overlap, but rather change within the coding pipeline. The collection stage 1150 collects the prediction signal (pred) as side information. i ) 1110 and subtracting the prediction from the picture feature data of the ith position (calculated in the decoder processing direction) to obtain the residual (residual) i ) 1120. The residuals are then encoded into a bitstream 1140. Thus, the residuals may be collected in the decoder 1100B by adding the residuals 1160 to a prediction obtained by a process that may include, for example, a motion compensation process or an intra-prediction process.

[0167] On the encoder side, the method may further include selecting whether to collect auxiliary data for one or more preconfigured positions based on optimizing a cost function including at least one of rate, distortion, accuracy, latency, or complexity. In other words, the positions may be determined by RDO or a similar technique. For example, on the encoder side, the decision regarding the stage in the pipeline where the auxiliary information is applied may be made by multi-pass encoding. In one example, applying the auxiliary information is allowed only at one position, as described above. In this case, the encoder attempts to apply the auxiliary information at the i-th position in the pipeline for preconfigured positions i=0, 1, 2, . . ., N, and obtains the distortion and rate for each variant. The optimal variant (position) for the defined rate / distortion ratio is then selected.

[0168] In another example, if it is possible to apply auxiliary information at any one or more positions in the pipeline, then for each Boolean vector of length (N+1), the encoder attempts to apply the auxiliary information at a position corresponding to the position in the Boolean vector, so that 2 N+1variants are tried. The search space can be reduced if some a priori knowledge is taken into account. A Boolean vector indicates, for each position i, whether auxiliary information is collected at that position.

[0169] However, the present disclosure is not limited to predictions and prediction residuals as auxiliary information. Rather, auxiliary data is applied as a kind of guidance data that enables directionality in the decoding process. In other words, the auxiliary data conditions the process with some supplementary information. In general, the auxiliary data can be labels or other data to aid the generative network.

[0170] It should be noted that in this disclosure, the meaning of prediction and residual signal is somewhat generalized, i.e., unlike conventional codecs such as HEVC where the prediction and residual signal are generated in the same region of the input signal x, the prediction and residual signal herein may refer to the signal after one or more layers in a neural network, as shown in Figure 11. In the embodiment of Figure 11, the prediction and residual signal are combined, i.e., the co-located prediction signal pred i and the residual signal resi i is enabled or disabled as soon as the collection conditions are met.

[0171] However, there are embodiments in which the predicted data and the supplementary data have dimensions that are processed by layers at different locations in the network. In other words, they are collected from different stages. The (selected) collection location of the prediction and the (selected) collection location of the residual may be different. Some embodiments of such approaches are described below in the section "Combining Data from Different Processing Stages of a Neural Network," and these embodiments may be combined with changing the collection location as described herein.

[0172] Figure 12 shows some experimental data. In experiments, the network configuration can be selected based on the analysis of the input signal. This analysis is performed in the frequency domain. An example is shown in Figure 12, where the transform FFT2 (see https: / / www.mathworks.com / help / matlab / ref / fft2.html), corresponding to the two-dimensional fast Fourier transform, is used as the transform tool to perform the spectral analysis. For a given image, it is possible to predict (or measure) the spectral changes introduced by the FFT2 transform before and after a network layer using 2x downsampling. If the main signal energy is preserved, it is advantageous to apply an auxiliary signal to a lower-resolution representation, since the error between the original and reconstructed picture is likely negligible. Figure 12 shows the following analysis steps: Step (a) shows the input (original) picture. The experiment here was performed on two different pictures (see the top and bottom of (a)). Step (b) shows the FFT2 result when FFT2 is applied to the original image. Step (c) shows the FFT2 result when FFT2 is applied to the original picture after downsampling by 2. Step (d) shows the reconstructed image based on the subsampled signal. Step (e) shows FFT2 applied to the reconstructed image. Step (f) shows the difference between the reconstructed image and the original image. Step (g) shows FFT2 of the difference image. As can be seen, the top image in Figure 12 has fewer high-frequency components. The reconstructed frame is obtained by first applying an auxiliary signal at the downsampled resolution and then upsampling. Such resampling has little effect on the signal content. In contrast, the loss of high-frequency energy is evident in both picture details and frequency content for the bottom image, where more high-frequency components are observed. This observation likely applies to other transforms as well. Overall, the network is adaptively configured so that the auxiliary signal is applied to the convolutional layer at an appropriate resolution and the spectral energy of the input signal is sufficiently preserved. The acquisition condition could be based on FFT2 or another transform that can provide some spectral information.For example, some statistical parameter of the spectral distribution may be thresholded to define at which locations auxiliary information should be collected.

[0173] It should be noted that the present disclosure is not limited to providing the prediction signal and the residual signal as side information. Another exemplary possibility is the application of side information in lossless compression.

[0174] Thus, potential locations where auxiliary information may be applied include locations where the signal may be processed losslessly. As shown in Figure 13, the neural network may include a subnetwork where the signal y_hat may be recovered losslessly from the decoder, as described above with reference to Figures 1 to 3. According to one embodiment, after obtaining the signal y_hat there is a collection unit 1310 where the auxiliary information may be applied later in the neural network.

[0175] In other words, in an embodiment, the neural network includes a subnetwork for lossless decoding having at least one layer, and auxiliary data is input to the subnetwork for lossless decoding. Thus, the collection condition 1320 may control whether a probability model for entropy coding (encoding and decoding as described above for the VAE framework) is provided as auxiliary information from the lossless coding subnetwork. If auxiliary information is not provided, the probability model may be determined, for example, by a default probability distribution or by bitstream signaling. The generation condition may be based on optimization based on a cost function or the like, taking into account the complexity and latency caused by the subnetwork and the subnetwork's contribution to reducing the resulting bitstream size. Alternatively, the collection condition may be based on the statistics of previously coded data, for example, their variance. If the data is nearly stationary or slowly changing, applying auxiliary information from the subnetwork may not be necessary.

[0176] In general, the neural network may be trained to perform at least one of still picture decoding, video picture decoding, still picture filtering, video picture filtering, and machine vision processing including object detection, object recognition, or object classification.

[0177] The encoding and / or decoding described above may be performed on each of a plurality of auxiliary data, including first auxiliary data and second auxiliary data, where the first auxiliary data is associated with a first set of one or more preconfigured positions and the second auxiliary data is associated with a second set of one or more preconfigured positions. In other words, the present disclosure may be applied to different types of auxiliary data within the same network. For example, the first auxiliary data is a prediction and residual signal, and the second auxiliary data is a probability model for lossless coding. In some embodiments, the first set of one or more preconfigured positions and the second set of one or more preconfigured positions share at least one preconfigured position. However, this is merely an example, and in some cases, the sets do not necessarily share any preconfigured positions. This may be the case, for example, in the example of prediction / residual as the first data and a probability model as the second data.

[0178] As shown in Figure 9, a configurable conditional autoencoder neural network is provided as an exemplary embodiment. Within this architecture, the location of applying auxiliary information, such as predictions and / or residuals, can be adaptively determined based on conditions. Thus, the architecture of the neural network can be dynamically modified depending on the characteristics of images or frames in a video sequence.

[0179] In general, embodiments of the present disclosure provide a technique for dynamically configurable neural network-based picture codecs, where the position within the coding pipeline at which side information is applied depends on conditions. The side information may be a combined pair of a prediction signal and a residual signal. The paired prediction and residual signals may be located in the same region as the input signal x, or in a region after processing by one or more layers of the neural network. The position at which side information is applied may be variable, which may be achieved by exclusively using side information at certain positions among potential positions within the neural network based on collection conditions. Such collection conditions control the position at which side information is applied in a content-adaptive manner, i.e., the conditions may be determined based on one or more characteristics of the input signal (picture). Furthermore, the conditions controlling the position at which side information is applied may be updated on the fly, for example, when encoding a video sequence; the conditions may be updated at the block level (within a frame), the frame level, or the group level of frames. One of the potential positions at which side information is applied is the position at which the signal is losslessly coded, as shown in FIG. 13. The position change may be applied to two or more independent auxiliary signals. Given two independent auxiliary signals A1 and A2, there must be two corresponding sets of potential positions to apply A1 and A2, respectively. Assuming these two sets of potential positions are S1 and S2, the positions within S1 and S2 are not restricted; i.e., they may or may not overlap. The same applies to three or more independent auxiliary signals.

[0180] The embodiments have been described primarily with respect to methods. However, the present disclosure is not limited thereto. Rather, the present invention also relates to an apparatus for processing picture feature data from a bitstream using a neural network including multiple neural network layers. The apparatus includes a processing circuit configured to obtain picture feature data from the bitstream and process the picture feature data using the neural network, and for each of one or more preconfigured locations in the neural network, the processing includes determining, based on a collection condition, whether to collect auxiliary data for processing by one of the multiple neural network layers at the preconfigured location. If it is determined that auxiliary data should be collected, processing by the layer at the preconfigured location is based on the auxiliary data. The processing circuit may be, for example, one or more processors configured with corresponding software, as described in detail with reference to Figures 26 to 29.

[0181] Similarly, the present invention relates to an apparatus for processing pictures using a neural network including multiple neural network layers to generate a bitstream, the apparatus comprising: a processing circuit configured to process pictures using the neural network, the processing including determining, for each of one or more preconfigured locations in the neural network, based on a collection condition, whether to collect auxiliary data for processing by the layer at said preconfigured location, if it is determined that auxiliary data should be collected, the processing by the layer at said preconfigured location is based on the auxiliary data, and the processing circuit configured to insert data obtained by processing the picture by the neural network into the bitstream. The processing circuit may be, for example, one or more processors configured by corresponding software, as will be described in detail with reference to Figures 26 to 29.

[0182] Combining data from different processing stages of a neural network FIG. 14 is a block diagram illustrating the architecture of a neural network for processing picture data, specifically the layer numbering applied on the encoder and decoder sides. An input picture 1402 is stacked into an input data tensor and input to an encoder-side neural network 1400A. As can be seen in the figure, the encoder-side network 1400A includes multiple cascaded processing layers (or blocks of layers) numbered from 1 to K. The stages here indicate stages of processed feature tensors. The input data tensor corresponds to the 0th stage, the feature tensor after processing the input data tensor using layer 1 corresponds to the 1st stage, and so on, up to the final (Kth) stage, which corresponds to the output of the neural network at the bottleneck. The output here corresponds to a bottleneck feature tensor (also referred to as an output or latent space tensor or output feature data) that can be encoded (e.g., by an arithmetic encoder as illustrated above) to form a bitstream 1450.

[0183] The decode-side network 1400B includes multiple cascaded processing layers (or layer blocks), numbered K through 1. In this example, the processing layers (or layer blocks) here correspond to the respective processing layers (or layer blocks) on the encoder side for simplicity of explanation. Note that in general, the encoder-side network and the decoder-side network do not necessarily need to be strictly symmetrical, meaning that they do not necessarily have the same number of layers, but can still have the same number of layers. The specific architecture may depend on the task to which they are deployed. Even for the same task (e.g., video / image coding), the structure of the NN may differ. Nevertheless, the number of downsampling layers on the encoder side is almost always equal to the number of upsampling layers on the decoder side.

[0184] The output of the decode-side network, in this example, is a reconstructed picture 1492. However, in general, the decode-side output does not have to be a reconstructed picture for human viewing. It can be feature data for, or the result of, computer processing, such as computer vision processing.

[0185] FIG. 15 shows in more detail the processing and data flow within a neural network that utilizes predictions and prediction residuals for encoding and decoding picture data at the same stage of the pipeline. In particular, these predictions and residuals are used at the same stage of the pipeline, i.e., stage 0. This may be a fixed arrangement or the result of the auxiliary information input location selection described above, as explained above. In this example, the network described with reference to FIG. 11 is shown in more detail. At the bottom, FIG. 15 shows encoding and decoding networks 1586 that process the current picture, i.e., the frame at time instance t+1. Networks 1586 essentially correspond to networks 1400A and 1400B described with reference to FIG. 14. However, at stage 0 on the encoder and decoder sides, auxiliary information is collected (1560, 1565) (similar to that shown in FIG. 11).

[0186] The information collected at the encoder side is a prediction signal. The prediction signal is combined with the input picture data (input tensor) at the 0th stage in section 1560 (corresponding to the collector). That is, the element-by-element difference between the input tensor and the prediction is calculated to obtain a residual, which is further encoded, i.e., processed on the encoding side of network 1586, and inserted into bitstream 1572. Note that element-by-element difference is only one exemplary possibility for implementing a common use or combination of data. In general, the combination may be performed in different ways. For example, if the two types of combined data have different resolutions or dimensions, the combination may further include rescaling. Depending on the neural network architecture and the training of the network, other types of combination, such as concatenation, may generally be used. Therefore, even though some examples showing addition or subtraction are provided in the figures, the present disclosure is not limited to such operations. Rather, they may represent a general combination of data, as will be further described with reference to Figures 21 and 22.

[0187] The decoder side processes latent space data from the bitstream 1572 using layers K through 1. The resulting data at the 0th stage (the output of the first layer or layer block at the decoder side) are decoded residuals. They are combined with predictions collected in a unit denoted as 1565. The collecting here includes combining the predictions with the decoded residuals. The combining here is an element-wise sum of the decoded residual tensor and the prediction. Note that the prediction here has the same size (dimension) as the residual signal for element-wise processing in the collecting units 1560 and 1565. As mentioned above, the element-wise sum shown here is only an exemplary combination of two signals (types of data). In general, the combining can be done differently and may mean tensor concatenation or stacking, rescaling, etc.

[0188] The upper part 1506 of Figure 15 shows how the prediction collected for processing the current picture is obtained. In particular, the prediction is obtained based on a reference frame (reference data) generated by the same network for the previous picture (at time instant t), and possibly based on further data such as motion information that may be provided in the bitstream 1590 of the current picture (at time instant t+1). The neural network processing of the tth (previous) picture by the encoding part of the network 1506 involves processing similar to that described above for the current picture. For t=0, there are typically no previously coded frames. Therefore, only the decoded samples of the current (t=0) frame can be used as prediction. For example, intra prediction can be applied. Alternatively, such a frame can be encoded in a different manner, for example, by lossless coding only.

[0189] The neural network processing of the tth (previous) picture by the encoding part of the network 1506 includes providing a reference frame (generated in the 0th stage by adding the decoded residual to the previously obtained reference frame) for further processing, namely motion estimation 1510 at the encoder side and motion compensation 1530 at the decoder side.

[0190] In motion estimation 1510, the motion between a reference picture (decoded at time instant t) and a current picture (at t+1) is estimated. The motion is then represented by motion information and inserted into a bitstream 1590. The estimated motion is then used in motion compensation 1520. Motion compensation is performed according to any well-known technique. The motion information may include the entire optical flow or a subsampled optical flow (and possibly further encoded by lossless or lossy coding) for a specific portion of the picture. In general, the motion information enables generating a prediction from a reference picture. The motion information specifies which sample values from the reference picture contribute to each sample value or predicted picture (and how they contribute). Motion compensation 1520 generates a predicted picture, which is then collected in collection unit 1560 to generate a residual signal (prediction error).

[0191] On the decoder side, the motion information is decoded from the bitstream 1590. During the decoding of the current picture, motion compensation 1530 generates a predicted picture based on the reference picture from the 0th stage of decoding the previous picture and based on the motion information, in the same way as described for motion compensation 1520 on the encoder side.

[0192] Note that Figure 15 mainly shows a signal flow. In practice, buffering may occur at several stages; for example, reference pictures may be stored in reference picture buffers at the encoder and decoder sides. A predicted picture may be generated using one reference picture as described above. However, this disclosure is not limited to such processing, and a predicted picture may be generated using two or more predicted pictures. The state-of-the-art provides various techniques using two reference pictures (bi-prediction), for example, a picture preceding the current picture in display order and a picture following the current picture in display order (bi-directional prediction). It should further be noted that time instances t and t+1 are measured in units of processed pictures. However, they are also merely exemplary. This disclosure is not limited to using the immediately preceding picture as a reference. In particular, for temporal scalability, a reference picture preceding or following the current picture in display order (but preceding the current picture in encoding and decoding order) may be used. Bitstreams 1590 and 1572 may be multiplexed into one bitstream.

[0193] FIG. 16 is similar to FIG. 15 in that it illustrates the collection of predictions in the same stage as the residuals, but the collection of predictions and residuals occurs in Stage 1. Thus, predictions do not need to be performed in the input signal domain. In this case, predictions are performed in the latent space domain in the first stage, which is the stage after processing the current input picture 1601 (at time instant t+1) using the first layer or block or layer 1605. In particular, the first layer 1605 may include downsampling, in which case handling predictions in the latent space may provide improved efficiency, e.g., in terms of complexity and latency. As seen in FIG. 16, the residuals are obtained in a collector unit 1660, e.g., by subtracting (generally combining) the prediction tensor from the output of the first layer 1605. The output of the first layer 1605 in this case may be a subsampled (and possibly further processed, e.g., filtered) input picture tensor of the same size as the prediction tensor, which is obtained based on a reference frame obtained from the first stage of the decoder pipeline (network 1606) of the previous picture. Corresponding motion information may then result in less data being conveyed in a bitstream 1690, which may be multiplexed with the residual bitstream 1672. On the decoder side, collection 1665 is also performed in a corresponding stage (here the first stage, e.g., a stage preceding processing by the first layer or layer block in the decoding processing order). Collection involves combining (in this case, element-wise summing) the prediction signal and the decoded residual in the latent space of the first stage, as in FIG. 15.

[0194] As will be apparent to those skilled in the art, following FIGS. 15 and 16, prediction signal collection can be performed at any stage from 2 to K. In particular, if a layer (or a block of a layer) includes subsampling, the higher the stage, the lower the prediction accuracy. On the other hand, as the stage increases, the rate of the generated bitstream may decrease, and the complexity and latency may also decrease. Thus, for low bitrate / quality coding, it may be advantageous to have the above-mentioned stages later (corresponding to a larger stage index), while for high bitrate / quality, it may be advantageous to have the above-mentioned stages earlier (corresponding to a smaller stage index). The possibility of independently selecting the prediction / residual collection stage may improve network operation. Furthermore, the stage selection may be adaptively performed depending on content characteristics (e.g., spectrum), as described in the above embodiments and examples described with reference to FIGS. 1 to 13.

[0195] 15 and 16 show examples where the prediction and residual are obtained from the same stage of processing (0th or 1st in these examples). However, according to one embodiment of the present disclosure, this is not necessarily the case. It may be advantageous to obtain the prediction and residual from different stages.

[0196] FIG. 17 shows an exemplary embodiment in which prediction is obtained within the i-th stage and residual coding is performed in the (i+r)-th stage of the NN pipeline.

[0197] According to such an embodiment, a method is provided for processing feature data of one or more pictures from bitstream 1772 (and possibly 1790) using a neural network. Neural network 1786 includes two or more stages (K through 1), including a first stage and a second stage, where the "first" stage and the "second" stage are any of stages K through 1, and the terms "first" and "second" are used as placeholders for the stage number.

[0198] The method includes obtaining first data 1760 based on a bitstream and processing the first data using a neural network. For example, the first data in Figure 17 is a residual decoded from a bitstream 1772 and possibly also processed by one or more layers. In Figure 17, the first data is a residual decoded from the bitstream 1772 and processed by layers K through i+r+1. In other words, the residual is used in the (i+r)th stage of the decoding pipeline.

[0199] Note that in general, the first data may be entropy coded feature data directly from the bitstream, without necessarily being processed by one or more layers. As discussed below, this disclosure may also apply to entropy encoding / decoding.

[0200] The process using neural networks is - obtaining second data 1765 from a second stage of the neural network 1706, the second data 1765 being based on data previously processed by the neural network; - using the first data together with the second data to generate inputs to a first stage of a neural network, the first stage preceding the second stage in the neural network's feature data processing (in the example of Figure 17, the order of processing in the decoding portion of the network, i.e., from stage K to stage 0); Includes:

[0201] The term "previous" here means a previous frame or a previous block (intra), etc. The method includes a step of outputting a result of process 1702 (e.g., a reconstructed picture or picture feature data). In general, this embodiment is applicable whenever the first stage and the second stage are different. However, as shown with reference to the embodiments described with reference to Figures 1 to 16, the first and second stages can also be the same stage.

[0202] The second data in this example may be a predicted frame. Obtaining the second data may include obtaining a reference frame and motion information. However, in general, the reference frame, or generally the reference data, may be directly obtained as a prediction (e.g., when motion is not signaled, or when intra-prediction or another type of prediction is used). Note that Figure 17 shows processing at the picture (frame) level. However, the present disclosure is not limited thereto, and in general, a picture portion may be encoded / decoded.

[0203] In the example of FIG. 17, the first stage is the (i+r)th stage of the decoding pipeline, and the second stage is the ith stage of the decoding pipeline.

[0204] In an exemplary implementation, the first data (e.g., residual) is obtained in a third stage of the neural network, which precedes or is equal to the first and second stages of the neural network's feature data processing. In other words, the first data may be obtained in a stage different from the stage where the first data is combined with the second data. This may be advantageous for some neural network architectures and may correspond to a type of skip connection. The Kth to (i+r+1)th layers may be used to preprocess the decoded residual signal and transform it into the same feature space as the second data processed using the (i+1) to (i+r) layers. Essentially, layers (i+1) to (i+r) and K to (i+r+1) are used to transform the first data and the second data into the same feature space (the same domain) in the exemplary implementation shown in FIG. 17. Once the processed first data and the processed second data are in the same domain, they can be combined as shown in FIGS. 21-22.

[0205] 17, the first data represents the prediction error (prediction residual) and the second data represents the prediction. In particular, the prediction is obtained by: - Obtaining reference feature data, which are the feature data output by the neural network in the second stage. In the general case of the i-th stage, the reference data is not in the picture domain but in the latent space domain, and is therefore generally referred to herein as "reference feature data." As already discussed with reference to Figure 15, the reference feature of the 0-th stage is the reference picture. - Obtaining prediction information, including motion or spatial information relative to reference feature data (at the same or a different stage), based on the bitstream. The relationship here is given by the fact that the prediction information is determined with respect to the reference feature data in the encoding part of the network. In FIG. 17, the motion information included in the bitstream 1790 is obtained by performing motion estimation for the motion between the reference feature data and the feature data processed by layers 1 to i on the encoder side. Other exemplary implementations are possible, such as using input picture data 1701 stacked into a tensor to generate the motion information and then downsampling it. As will be apparent to those skilled in the art, rescaling or resizing may be applied, if necessary. - generating a prediction based on the reference feature data and the prediction information. In Figure 17, the generation of the prediction is performed by motion compensation, as described above with reference to Figure 15. Thus, the second data is a prediction 1765 corresponding to the output of the motion compensation.

[0206] 17, the prediction error is obtained by processing the current picture using a neural network. The prediction information is motion information, and the reference feature data is generated by neural network processing of picture data representing pictures preceding the current picture in decoding order, and possibly further applying motion estimation. As also mentioned above, decoding order is not necessarily display order, and the prediction may be based on more than one reference.

[0207] Using the first data 1760 with the second data 1765 described above involves element-wise addition of a prediction or rescaled prediction and a prediction error or rescaled prediction error. Such rescaling of the prediction data is shown in FIG. 17 and is included, for example, in the processing by layers i+1 to i+r (which may be, for example, convolutional layers). As mentioned above, the element-wise addition may be replaced by a different type of combination, such as concatenation or other operation or sequence of operations.

[0208] Note that the present disclosure is not limited to auxiliary data being predictions. As described below, in some embodiments, auxiliary (second) data may be obtained from a different (sub)network as a hyper-prior and used together with the data encoded in the 0th stage to generate latent space feature data at the input of the decoding side of the neural network. In such cases, using together refers to applying probability model information, for example, to arithmetic decoding of picture feature data.

[0209] 17, the motion information includes motion vectors. However, this disclosure is not limited to any particular method of motion estimation and compensation. Thus, in general, the motion information is not limited to motion vectors. Rather, the motion information may include affine transformation parameters. In addition to motion vectors, the motion information may include positions relative to the signaled motion vectors (in the case of non-regular motion field sampling).

[0210] Figure 18 is a specific case of the more general embodiment of Figure 17. In particular, in Figure 18, the second stage is the output of the (decoder) neural network, i.e., the 0th stage. In other words, the reference frame is obtained from the 0th stage of the decoding process, as already shown in Figure 15. Note that the 0th stage in this case is the stage of processing the previous picture (time instance t). However, in this example, it is assumed that the network architecture does not change for processing the current picture or picture feature data (time instant t+1) and the previous picture or picture feature data (time instant t).

[0211] In Figure 18, the first stage (equivalent to the third stage described above) is K, i.e., the residual is calculated (in the encoder) and used in the Kth stage of the pipeline (in the decoder). However, the present disclosure is not limited thereto, and the first stage may be the input of a neural network on the decoder side, especially in the case of embodiments relating to lossless coding, which will be described in more detail below.

[0212] Figure 19 shows another example in which the reference picture (and therefore also the prediction) is obtained at stage 1 of the pipeline, i.e., the second stage is stage 1 of the decoding pipeline. This corresponds to Figure 16. The first stage for calculating and using the residual remains stage K of the encoding and decoding pipeline, as in Figure 18 described above.

[0213] Another example is shown in FIG. 20. In FIG. 20, the second stage for obtaining a reference frame is stage 1 as in FIG. 16 and FIG. 19. However, the first stage for calculating and using a residual is stage (K-1) of the encoding and decoding pipeline. In other words, on the encoding side, after obtaining the residual (after combining the first data and the second data), the combined data may be processed by one or more layers 2020 of a neural network before inserting the processed data into a bitstream 2050. In this exemplary processing pipeline, as well as in the examples of FIG. 17 to FIG. 19, a prediction (corresponding to the second data) may also be processed by one or more layers 2010 of a neural network on the encoding side. Correspondingly, on the decoding side, before combining the further processed prediction in stage K-1, data from the bitstream may be processed by one (layer K in FIG. 20) or more layers 2022. Also, the prediction may be processed by one or more layers 2012 of a neural network on the decoding side. As mentioned above, processing the first data and the second data before combining them may include, but is not limited to, adapting them to a common size (resolution).

[0214] The above example shows a symmetric encoding and decoding pipeline and a symmetric collection of predictions at corresponding stages in the encoding and decoding pipeline. Note that this is not necessarily the case. Due to the possibility of rescaling and resizing, asymmetric solutions are possible.

[0215] 15 to 20 described specific exemplary embodiments in which prediction is applied to obtain a residual or to obtain picture (feature) data. Illustratively, at the encoding side, prediction data (second data) was subtracted from the picture (feature) data input to the neural network (and possibly processed by several layers) to obtain a residual. The subtraction was element-wise. Similarly, at the decoding side, the residual was combined with the prediction element-wise. Such an approach is shown in FIG. 21. In FIG. 21, the difference between the prediction block and the original block was determined element-wise and included in the bitstream. Then, the difference (also referred to as the residual or prediction error in this disclosure) is obtained from the bitstream at the decoder side and added to the prediction block to obtain a reconstructed block. This example is merely schematic, and the blocks may directly represent picture data or picture feature data.

[0216] However, in some embodiments, direct combining of original blocks and predicted blocks, and / or direct combining of residual blocks and predicted blocks, is not always possible for each block. For example, as discussed above, the prediction, input (original) data, and residual may have different sizes (resolutions). Accordingly, Figure 22 illustrates generalized combining in an encoder and a decoder. For example, in the encoder, obtaining the residual may include upsampling or downsampling of at least one of the original data and predicted data. Similarly, in the decoder, applying the residual to the prediction may include rescaling / resizing of the residual and / or prediction, etc. In one example, the input (original) data is stacked with the predicted data, so that the number of channels becomes the sum of the number of original data channels and the number of predicted data channels. Such combined data is further processed by one or more convolutional layers (possibly by downsampling or strided convolution), and the result is the residual. At the decoder side, the application of the residual is performed in the following manner: the residual is stacked with the prediction data and then processed by one or more convolutional layers (possibly by downsampling or strided convolution); the result of such processing is reconstructed data (which may have two or more channels since it is a latent space). At the encoder side, if the original data and the prediction data have different sizes (resolutions), rescaling / resizing may be applied before stacking the channels. At the decoder side, if the residual and the prediction data have different sizes (resolutions), rescaling / resizing may be applied before stacking the channels. Also, several convolutional layers may be used to preprocess the prediction and / or prediction error before combining. In general, such types of preprocessing may be different for prediction and prediction error.

[0217] In the following, embodiments are provided for applications in lossless encoding and decoding. Generally, the first data is obtained after the encoding part of the autoencoder and is further quantized before arithmetic coding. In this case, such an embodiment using the second data as an entropy model can still function for lossy coding that precedes lossless coding.

[0218] Thus, the first stage is the input of a neural network. The first data is entropy encoded data. The second data is probability model data related to the feature data of the second stage. For example, the second data is probability model data for entropy decoding of the first data.

[0219] In this exemplary implementation, the second data is used to predict the probability of symbols in the first data, which can substantially improve arithmetic coding. The technical effect is comparable to subtracting the prediction from the signal and encoding the difference (residual), for example, close to zero. Lossy coding is implemented when the symbols encoded by arithmetic coding are the quantized output of the encoding part of the autoencoder (which is one advantageous specific practical embodiment). Lossless coding is implemented when the symbols are raw image samples (which are already integers and can be encoded by arithmetic coding).

[0220] This embodiment is an extension of the embodiment described with reference to Figure 14. In Figure 14, the probability model is determined based on feature data in the 0th stage and applied in the 0th stage of encoding and decoding (for arithmetic encoding and arithmetic decoding). However, this does not generally have to be the case. Rather, second data corresponding to the probability model information may be obtained, for example, by applying a hyper-prior subnetwork based on picture feature data in the latent space, for example, from stages greater than 0.

[0221] In general, in any of the above examples or embodiments, the positions of the first and / or second stages may be configurable. For example, the positions of the first and / or second stages may be configurable within a neural network, and the processing method at the encoding side and / or decoding side may include configuring the positions of the first and / or second stages according to collection conditions based on one or more picture coding parameters. In other words, a technique for selecting collection positions based on collection conditions may be applied to the first and / or second stages. The collection conditions may be based on parameters (information) available to both the encoder and the decoder.

[0222] Alternatively, or in addition, the collection condition may depend on a selection made in the encoder and signaled to the decoder. For example, a decoding method may include parsing a stage selection indicator from a bitstream, the stage selection indicator specifying the first stage and / or the second stage. The positions of the first stage and the second stage in the neural network are configured according to the parsed stage selection indicator. At the encoding side, a stage selection instruction is generated and inserted into the bitstream. As described above, the encoder may decide to select a particular stage based on some optimization based on a cost function including rate, distortion, complexity, latency, accuracy, etc. Alternatively, other techniques, such as the amount of lost information (e.g., the FFT2 described above), may be applied.

[0223] In summary, this disclosure provides some embodiments that may make neural network-based image / video codecs, or general picture processing, dynamically configurable, where the position in the coding pipeline at which side information is applied may depend on conditions. Further or alternative improvements may result from an architecture in which side information obtained in one pipeline stage for a reference frame may be used in other pipeline stages for the current frame. An example of such a scheme for prediction and residual side information is discussed with reference to FIG. 17. For example, a prediction obtained from the i-th stage of the pipeline on the reference frame side may be further processed by r subsequent NN modules and used within the (i+r)-th stage of the pipeline for the current frame to calculate residuals. In this case, the residuals are calculated between the predictions processed by NN modules i+1, i+2, .., i+r and the current frame processed by NN modules 1, .., i+r. The residuals are then further processed by NN modules (i+r+1), .., K. The output of the Kth NN module (at the encoder side) is the bottleneck and is signaled in the bitstream. At the decoder side, the same scheme is applied: predictions are processed by NN modules i+1, i+2, .. i+r, and the residual obtained from the bitstream is processed by module K, .., (i+r+1). The combined signal obtained from the predictions and residuals is then processed by modules (i+r), .., 1.

[0224] As explained above for one-stage selection, at the encoder side, the decision regarding the stage in the pipeline where the side information is applied can be made by multi-pass encoding. In one example, applying side information is allowed only at one position. In this case, the encoder for k=0, 1, 2, .., K attempts to apply side information at the kth position in the pipeline and obtains the distortion and rate for each variant. The optimal variant for the defined rate / distortion ratio is then selected. In another example, if applying side information is possible at all positions in the pipeline, the encoder for each Boolean vector of length (K+1) attempts to apply side information at the position corresponding to the position in this Boolean vector, so 2 K+1 variants are tried.

[0225] In the context that more than one stage may be selected, all combinations of i and r may be tried. The following condition must be met: 0≦i≦i+r≦K. To avoid trying all possible positions for applying the side information and thus reducing the complexity of the encoder, heuristic methods may be used instead of an exhaustive search. For example, the spectral analysis described above (e.g., using FFT2) may be used.

[0226] In the exemplary implementations shown in FIGS. 15-20, the prediction is calculated only once, and the residual is applied only once. Generally, a coding (encoding and / or decoding) scheme may include two or more prediction / residue acquisition blocks at different stages (e.g., for processing by different NN layers). In summary, in the above-described embodiments, the side information may include a prediction signal and / or a residual signal. The prediction signal is obtained from the i-th pipeline position of a previously coded frame, and the residual signal is applied at the k-th pipeline position during current frame processing. In some embodiments, i and k are different from each other. The values i and k may be predetermined, e.g., fixed within the neural network architecture, or configurable within the neural network architecture. For example, the values i and k depend on conditions derived at the encoder and decoder sides. Alternatively, or in addition, the values i and k are determined based on syntax elements parsed from the bitstream.

[0227] In some embodiments, k is equal to i+r, where r is greater than or equal to zero. Such an example is shown in the general scheme in FIG. 17. FIG. 20 illustrates the scheme for i=1, i+r=K-1, where K is the number of NN modules in the encoder pipeline. A module may be shown as a block and may include one or more layers. In a possible implementation, i is equal to 0, i.e., prediction is performed in a single region (the 0th position in the pipeline). In some other possible implementations, i+r is equal to K, where K is the number of NN modules in the encoder pipeline. Residual coding is performed within the bottleneck, i.e., at the Kth position in the pipeline. FIG. 18 illustrates the scheme for i=0, i+r=K. FIG. 19 illustrates the scheme for i=1 and i+r=K.

[0228] In any of the above examples, the prediction signal may be calculated based on information from the pth pipeline position of the previously coded frame and additionally signaled information. For example, the additionally signaled information may include motion information. For example, the residual signal may be calculated by a module having two inputs: the prediction signal and the output of the kth NN module in the pipeline. In some embodiments, the residual signal is calculated as the difference between the prediction signal and the output of the kth NN module in the pipeline. Furthermore, the application of the residual signal is performed by a module having two inputs: the residual signal and the prediction signal. Finally, the application of the residual signal is performed by adding the residual signal to the prediction signal.

[0229] Figures 23 to 25 summarize some possible architectures of the processing pipeline.

[0230] FIG. 23 shows an example in which an original picture, a prediction, and a residual are in the signal domain, i.e., the 0th stage of processing by an encoding neural network. The original picture and prediction are combined into the obtained residual. The residual is encoded using a VAE with a hyper-prior. The result is a bitstream containing the coded residual and possibly some side information. The decoder side receives the bitstream. The decoder here is a VAE with a hyper-prior that outputs a reconstructed residual. The reconstructed residual can be combined with the prediction signal to obtain a reconstructed picture. The reconstructed picture can be further compared to the original picture, for example, by applying a distance measure, e.g., an objective or perceptual quality metric.

[0231] FIG. 24 shows an example in which the original picture, prediction, and residual are in the latent space domain, i.e., the first stage of processing by the encoding neural network. In this case, the original picture and prediction are first processed by one or more layers of a VAE encoder and then combined to obtain a residual in the latent space domain. These are encoded by a hyper-ahead encoder, thereby generating a bitstream. The bitstream can be decoded by a hyper-ahead decoder to obtain a reconstructed latent space residual. The reconstructed latent space residual is combined with the prediction (also in the latent space domain) in the latent space domain, thus obtaining a reconstructed picture feature tensor that is input to the VAE decoder to obtain a reconstructed picture. The reconstructed picture can be further compared to the original picture, for example, by applying a distance measure, e.g., an objective or perceptual quality metric.

[0232] FIG. 25 shows that an exemplary original picture is encoded by a VAE encoder, and then a residual is obtained in the latent space by subtracting (or combining) the encoded picture with a prediction obtained from the latent space of the previous frame. The residual is encoded into a bitstream using a hyper-ahead encoder. The bitstream can be decoded by a hyper-ahead decoder, thus obtaining a reconstructed residual in the residual space. These are combined with the prediction obtained from the latent space of the previous frame, thus obtaining a reconstructed picture feature tensor in the latent space. The obtained reconstructed picture feature tensor is transformed by a VAE decoder to obtain a reconstructed picture. The reconstructed picture can be further compared with the original picture, for example, by applying a distance measure, such as an objective or perceptual quality metric.

[0233] Any of these three architectures can be the basis for using the embodiments and exemplary implementations described above. These architectures included a VAE encoder and decoder. However, in general, this disclosure is not limited to a VAE with a hyperprior. Other approaches to symbol probability estimation for arithmetic coding can be applied as well. For example, a context-based approach can be used. In this case, information about previously decoded samples is additionally used by the symbol probability estimation neural network. An example of a probability estimation neural network using a context approach is shown in Figure 13.

[0234] Overview of Some Embodiments The present application provides a method and apparatus for processing picture data or picture feature data using a neural network with two or more layers. The present disclosure may be applied to the field of artificial intelligence (AI)-based video or picture compression technology, particularly to the field of neural network-based video compression technology. According to some embodiments, a location within the neural network where auxiliary information can be input for processing can be selected based on a collection condition. The collection condition may evaluate whether certain preconditions are met. Some of the advantages may include better performance in terms of rate and / or resolution due to increased flexibility in neural network configuration.

[0235] According to the first aspect , BookThe disclosure relates to a method for processing picture feature data from a bitstream using a neural network including multiple neural network layers, the method including the steps of obtaining picture feature data from the bitstream and processing the picture feature data using the neural network, wherein for each of one or more preconfigured locations in the neural network, the processing includes determining whether to collect auxiliary data for processing by one of multiple neural network layers at said preconfigured location based on a collection condition, and if it is determined that auxiliary data should be collected, the processing by the layer at said preconfigured location is based on the auxiliary data.

[0236] A preconfigured location is a location within a neural network where auxiliary information can be collected. Whether auxiliary information is actually collected at a particular preconfigured location is determined in the decision step described above. The term "location" refers to a node within a neural network. A node corresponds to a feature tensor input to and / or output from a layer (or a block of layers or modules, depending on the architecture of the neural network). In particular, a location may be specified by the number of neural network layers that precede the location in the processing direction.

[0237] Auxiliary information is any information about picture data or picture feature data that is added to a neural network. Such information may be input into the neural network to further improve processing. Some specific examples of auxiliary information are provided in the following exemplary implementations.

[0238] A collection condition is a condition or prerequisite that must be met for auxiliary data to be input at a particular preconfigured location. The collection condition may include a comparison of some picture characteristics or picture feature characteristics with a threshold to determine whether to collect auxiliary information for a particular location. The picture characteristics or picture feature characteristics may be known to the encoder and decoder so that no additional signaling is required. Alternatively, or in addition, the collection condition may be configured by the encoding side by setting an indicator of whether auxiliary information should be collected for a preconfigured location. The indicator may be provided in a bitstream available to the decoder.

[0239] Configuring the location of inputting auxiliary information into the neural network has the effect of providing greater flexibility, allowing for dynamic neural network architecture changes. Such flexibility may result in better adaptation based on acquisition conditions, leading to more efficient encoding and / or decoding.

[0240] In a possible implementation, applying the collection conditions in the determination results in the auxiliary data being collected for a single one of one or more pre-configured locations.

[0241] Such an implementation provides a similar effect to selecting, for a particular auxiliary information, the location within the neural network where the auxiliary information should be collected, which allows providing the auxiliary information at the location that may be most appropriate according to some criteria, such as processing time or complexity and / or coding efficiency, which may include rate or distortion.

[0242] In a possible alternative implementation, applying a collection condition in a determination results in the auxiliary data being collected for two or more of the pre-configured locations.

[0243] Such an alternative implementation to the one described above may ensure that auxiliary information is available at any layer that may benefit from it, which may further increase the configurability and therefore flexibility of the network.

[0244] In a possible implementation, there are two or more of the preconfigured locations (the processing is performed for two or more preconfigured locations), the auxiliary data is scalable in size to match the dimensions of the input channels processed by the layer at two or more of the preconfigured locations, and as a result of applying the collection conditions in the determination, the auxiliary data is (i) collected, or (ii) collected and scaled for a single one of the preconfigured locations.

[0245] Therefore, the auxiliary data may be appropriately scaled to enable its combination with the feature data. Such scaling allows for the provision of a wide variety of auxiliary information that may come from different stages or sources.

[0246] In a possible implementation, the collection conditions are based on picture characteristics or picture feature data characteristics obtained from the bitstream.

[0247] This implementation allows for content adaptation of the neural network and may improve the performance of encoding or decoding or other processing performed by the neural network.

[0248] In a possible implementation, the picture characteristics or picture feature data characteristics include resolution, and the collection conditions include a comparison of the resolution with a preconfigured resolution threshold.

[0249] Resolution is a relevant decision criterion because it affects the level of detailed features, which may degrade in quality when further downsampled or otherwise processed in a lossy manner.

[0250] In a possible implementation, the picture is a video picture, the picture characteristics include a picture type, and the collection conditions include determining whether the picture type is a temporally predicted picture type or a spatially predicted picture type.

[0251] Picture type is a relevant decision criterion since it affects the quality of the prediction. Furthermore, encoding intra pictures with high quality may be desirable since this may also affect inter pictures, which typically use intra-coded pictures as a reference for temporal prediction.

[0252] According to an exemplary implementation, the method further includes obtaining, from the bitstream, an instruction specifying whether to collect auxiliary data for one or more preconfigured locations, wherein the collection condition for each of the one or more preconfigured locations is as follows: if the instruction specifies that auxiliary data should be collected for the preconfigured location, the determination is positive; and if the instruction specifies that auxiliary data should not be collected for the preconfigured location, the determination is negative.

[0253] Explicit signaling of collection locations provides additional configurability and allows for closer adaptation to the picture's content, even if the predefined collection conditions may not capture the picture's content.

[0254] In a possible implementation, the auxiliary data provides information about the picture feature data that is processed by the neural network to generate the output.

[0255] Providing additional information regarding picture feature data may improve reconstruction and / or other types of processing. Depending on the processing type, different auxiliary information may be relevant and applicable.

[0256] In a possible implementation, the auxiliary data includes prediction data that is a prediction of the picture or a prediction of picture feature data after processing by one or more of the layers of the neural network.

[0257] Moving picture (video) coding efficiency can depend heavily on removing temporal correlation from adjacent pictures, and therefore providing prediction or prediction error data can improve neural network processing.

[0258] In a possible implementation, the auxiliary data is a combined pair of predicted data and supplemental data combined with the predicted data.

[0259] Collecting predictions and prediction errors in the same region can provide a relatively simple and effective way of collecting step selections.

[0260] In a possible implementation, the prediction data and the supplementary data have the dimensions of data processed by layers at different positions in the neural network.

[0261] In other words, predictions may be provided at a different stage than the stage at which prediction residuals or supplementary information are provided, thus achieving efficiency by better adapting to the content.

[0262] In a possible implementation, the neural network comprises a sub-network for lossless decoding having at least one layer, and the auxiliary data is input to said sub-network for lossless decoding.

[0263] Recently, variable autoencoders with hyperpriors have been adopted and may be efficiently supported by conditional collection (based on collection conditions) of the hyperprior. In some cases, it may be beneficial (leading to a more compact bitstream) if a sub-network is used to derive the probability model parameters. In other cases, it may be too computationally expensive if a default or context-based probability model already works well.

[0264] In a possible implementation, the neural network is trained to perform at least one of still picture decoding, video picture decoding, still picture filtering, video picture filtering, and machine vision processing including object detection, object recognition, or object classification.

[0265] This implementation allows the deployment of the methods described herein to many important applications that may benefit from flexible location (and possibly also resolution) of auxiliary information.

[0266] In one embodiment, the method is performed for each of a plurality of auxiliary data including first auxiliary data and second auxiliary data, where the first auxiliary data is associated with a first set of one or more preconfigured locations and the second auxiliary data is associated with a second set of one or more preconfigured locations.

[0267] This allows the neural network to efficiently collect two or more types of auxiliary information, even if they are quite different, e.g., prediction-related auxiliary information and lossless coding-related auxiliary information, which are naturally used in different positions.

[0268] In a possible implementation, the first set of one or more preconfigured locations and the second set of one or more preconfigured locations share at least one preconfigured location.

[0269] This is an example implementation that may be suitable for some applications. For example, in the case of prediction and prediction error, all pre-configured positions may be shared.

[0270] In a possible implementation, the neural network is trained to process video pictures, and the decision of whether to collect auxiliary data for processing by the layer at the preconfigured location is made for every predetermined number of video pictures, where the predetermined number of video pictures is one or more.

[0271] Having some granularity for collection location adaptation may contribute to efficiency as it may involve less complexity and, in some embodiments, less signaling overhead.

[0272] According to a second aspect, the present invention relates to a method for processing pictures using a neural network including multiple neural network layers to generate a bitstream, the method comprising: processing a picture using the neural network, the processing comprising, for each of one or more preconfigured locations in the neural network, determining based on a collection condition whether to collect auxiliary data for processing by the layer at said preconfigured location, and if it is determined that auxiliary data should be collected, processing by the layer at said preconfigured location is based on the auxiliary data. The method further comprises inserting data obtained by processing the picture by the neural network into the bitstream.

[0273] The encoding portion of this disclosure may provide the same advantages as those described above for the decoding portion: the encoder prepares and provides a bitstream to the decoder so that the decoder can decode or reconstruct the data with the desired quality and application in mind.

[0274] To avoid redundancy, the advantages provided for the corresponding decoding process claims apply to encoding in an analogous manner.

[0275] In a possible implementation, applying the collection conditions in the determination results in the auxiliary data being collected for a single one of one or more pre-configured locations.

[0276] In a possible implementation, applying a collection condition in a determination results in the auxiliary data being collected for two or more of the pre-configured locations.

[0277] In a possible implementation, there are two or more of the preconfigured locations (the processing is performed for two or more preconfigured locations), the auxiliary data is scalable in size to match the dimensions of the input channels processed by the layer at two or more of the preconfigured locations, and as a result of applying the collection conditions in the determination, the auxiliary data is (i) collected, or (ii) collected and scaled for a single one of the preconfigured locations.

[0278] In a possible implementation, the collection conditions are based on picture characteristics or picture feature data characteristics included in the bitstream.

[0279] In a possible implementation, the picture characteristics or picture feature data characteristics include resolution, and the collection conditions include a comparison of the resolution with a preconfigured resolution threshold.

[0280] In a possible implementation, the picture is a video picture, the picture characteristics include a picture type, and the collection conditions include determining whether the picture type is a temporally predicted picture type or a spatially predicted picture type.

[0281] In some embodiments, the method further includes generating an indication specifying whether to collect auxiliary data for one or more preconfigured locations, and including the indication in the bitstream.

[0282] According to an example implementation, the method further includes selecting, for one or more preconfigured locations, whether to collect auxiliary data based on optimizing a cost function including at least one of rate, distortion, accuracy, speed, or complexity.

[0283] Determining the location for collection based on a cost function can improve the adaptation of the neural network and the results to the desired requirements. Thus, such optimization can improve performance. In combination with indicating the collection location thus obtained, flexibility is further improved.

[0284] In a possible implementation, the auxiliary data provides information about the picture feature data that is processed by the neural network to generate the output.

[0285] In a possible implementation, the auxiliary data includes prediction data that is a prediction of the picture or a prediction of picture feature data after processing by one or more of the layers of the neural network.

[0286] In a possible implementation, the auxiliary data is a combined pair of predicted data and supplemental data to be combined with the predicted data.

[0287] In a possible implementation, the prediction data and the supplementary data have the dimensions of data processed by layers at different positions in the neural network.

[0288] In a possible implementation, the neural network comprises a sub-network for lossless decoding having at least one layer, and the auxiliary data is input to said sub-network for lossless decoding.

[0289] In a possible implementation, the neural network is trained to perform at least one of still picture encoding, video picture encoding, still picture filtering, video picture filtering, and machine vision processing including object detection, object recognition, or object classification.

[0290] According to an example implementation, the method is performed for each of a plurality of auxiliary data including first auxiliary data and second auxiliary data, where the first auxiliary data is associated with a first set of one or more preconfigured locations and the second auxiliary data is associated with a second set of one or more preconfigured locations.

[0291] In a possible implementation, the first set of one or more preconfigured locations and the second set of one or more preconfigured locations share at least one preconfigured location.

[0292] In a possible implementation, the neural network is trained to process video pictures, and the decision of whether to collect auxiliary data for processing by the layer at the preconfigured location is made for every predetermined number of video pictures, where the predetermined number of video pictures is one or more.

[0293] According to a third aspect, the present invention relates to an apparatus for processing picture feature data from a bitstream using a neural network including multiple neural network layers. The apparatus includes a processing circuit configured to obtain picture feature data from the bitstream and, for each of one or more preconfigured locations in the neural network, process the picture feature data using the neural network. The processing includes determining, based on a collection condition, whether to collect auxiliary data for processing by one of the multiple neural network layers at the preconfigured locations, and if it is determined that auxiliary data should be collected, processing by the layer at the preconfigured location is based on the auxiliary data.

[0294] For advantageous effects of the present disclosure, please refer to the description of the first aspect. Details will not be described again here. The decoding device has functions for implementing the actions in the example method of the first aspect. The functions may be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. In a possible implementation form, the decoding device includes a feature data input module configured to acquire picture feature data from a bitstream, and a neural network module configured to perform the above-mentioned processing. These modules may perform corresponding functions in the example method of the first aspect. For details, please refer to the detailed description in the example method. Details will not be described again here.

[0295] According to a fourth aspect, the present invention relates to an apparatus for processing pictures using a neural network comprising a plurality of neural network layers to generate a bitstream, the apparatus comprising: a processing circuit configured to process pictures using a neural network, the processing comprising: for each of one or more preconfigured locations in the neural network, determining based on a collection condition whether to collect auxiliary data for processing by the layer at said preconfigured location, and if it is determined that auxiliary data should be collected, processing by the layer at said preconfigured location inserts data obtained by processing the picture by the neural network based on the auxiliary data into the bitstream.

[0296] For advantageous effects of the present invention, please refer to the description of the second aspect. Details will not be described again here. The encoding device has functions for implementing the actions in the example method of the second aspect. The functions may be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. In a possible implementation form, the encoding device includes a neural network module configured to perform the above-mentioned processing, and a bitstream output module configured to process the picture using the neural network, insert the obtained data into a bitstream, and output the bitstream. These modules may perform corresponding functions in the example method of the second aspect. For details, please refer to the detailed description in the example method. Details will not be described again here.

[0297] The method according to the first aspect of the invention may be performed by an apparatus according to the third aspect of the invention. Other features and implementations of the method according to the first aspect of the invention depend directly on the functionality and implementation of the apparatus according to the third aspect of the invention.

[0298] The method according to the second aspect of the invention may be performed by an apparatus according to the fourth aspect of the invention. Other features and implementations of the method according to the second aspect of the invention depend directly on the functionality and implementation of the apparatus according to the fourth aspect of the invention.

[0299] According to a fifth aspect, the present invention relates to a video stream decoding device comprising a processor and a memory, the memory storing instructions for causing the processor to perform the method according to the first aspect.

[0300] According to a sixth aspect, the present invention relates to a video stream encoding device comprising a processor and a memory, the memory storing instructions for causing the processor to perform the method according to the second aspect.

[0301] According to a seventh aspect, there is proposed a computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to encode video data, the instructions causing the one or more processors to perform a method according to the first or second aspect, or any possible embodiment of the first or second aspect.

[0302] According to an eighth aspect, the present invention relates to a computer program product comprising program code for performing the method according to the first or second aspect or any possible embodiment of the first or second aspect, when the program code is executed on a computer.

[0303] Software and hardware implementations In the following, embodiments of the video coding system 10, the video encoder 20, and the video decoder 30 will be described with reference to FIGS.

[0304] 26 is a schematic block diagram illustrating an example coding system 10, e.g., video coding system 10 (coding system 10 for short), that may utilize techniques of the present application. Video encoder 20 (or encoder 20 for short) and video decoder 30 (or decoder 30 for short) of video coding system 10 represent examples of devices that may be configured to perform techniques according to various examples described in the present application. Video encoder 20 and video decoder 30 may be implemented using (including or as) a neural network, as described in the above embodiments and examples. The remainder of system 10 may be, but need not be, implemented by a neural network.

[0305] As shown in FIG. 26, coding system 10 includes a source device 12 configured to provide encoded picture data 21 to, for example, a destination device 14 for decoding encoded picture data 13.

[0306] Source device 12 includes an encoder 20 and may additionally, i.e., optionally, include a picture source 16, a preprocessor (or preprocessing unit) 18, such as a picture preprocessor 18, and a communication interface or unit 22.

[0307] Picture source 16 may include or be any type of picture capture device, e.g., a camera for capturing real-world pictures, and / or any type of picture generation device, e.g., a computer graphics processor for generating computer-animated pictures, or any type of other device for obtaining and / or providing real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). Picture source may be any type of memory or storage that stores any of the foregoing pictures.

[0308] To distinguish from the preprocessor 18 and the processing performed by the preprocessing unit 18, the picture or picture data 17 may also be referred to as a raw picture or raw picture data 17.

[0309] The pre-processor 18 is configured to receive (raw) picture data 17 and perform pre-processing on the picture data 17 to obtain a pre-processed picture 19 or pre-processed picture data 19. The pre-processing performed by the pre-processor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal. It may be understood that the pre-processing unit 18 may be an optional component. It may be understood that the pre-processing unit 18 may be an optional component.

[0310] Video encoder 20 is configured to receive pre-processed picture data 19 and provide encoded picture data 21 (as described in further detail in 1 to 25 above).

[0311] The communications interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and transmit the encoded picture data 21 (or any further processed version thereof) via the communications channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0312] The destination device 14 includes a decoder 30 (e.g., a video decoder 30), and may additionally, i.e. optionally, include a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34.

[0313] The communications interface 28 of the destination device 14 is configured to receive the encoded picture data 21 (or any further processed version thereof), for example, directly from the source device 12 or any other source, for example, a storage device, for example, an encoded picture data storage device, and to provide the encoded picture data 21 to the decoder 30.

[0314] The communication interface 22 and the communication interface 28 may be configured to transmit or receive the encoded picture data 21 or the encoded data 13 via a direct communication link between the source device 12 and the destination device 14, e.g., a direct wired or wireless connection, or via any type of network, e.g., a wired or wireless network or any combination thereof, or any type of private and public network or any combination thereof.

[0315] The communications interface 22 may be configured, for example, to package the encoded picture data 21 into an appropriate format, e.g., packets, and / or to process the encoded picture data using any type of transmission encoding or processing for transmission over a communications link or network.

[0316] The communications interface 28, which forms a counterpart of the communications interface 22, may be configured to receive the transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or depackaging, for example, to obtain the encoded picture data 21.

[0317] Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrow pointing from source device 12 to destination device 14 for communication channel 13 in FIG. 26, or as bidirectional communication interfaces, and may be configured to, for example, send and receive messages to set up a connection, confirm and exchange a communication link and / or any other information related to data transmission, e.g., transmission of encoded picture data.

[0318] The video decoder (or decoder) 30 is configured to receive the encoded picture data 21 and provide decoded picture data 31 or decoded pictures 31 (further details are described above, for example, with reference to Figures 1 to 25).

[0319] Post-processor 32 of destination device 14 is configured to post-process decoded picture data 31 (also called reconstructed picture data), e.g., decoded picture 31, to obtain post-processed picture data 33, e.g., post-processed picture 33. The post-processing performed by post-processing unit 32 may include, e.g., color format conversion (e.g., from YCbCr to RGB), color correction, cropping or resampling, or any other processing, e.g., to prepare decoded picture data 31, e.g., for display, e.g., by display device 34.

[0320] Display device 34 of destination device 14 is configured to receive post-processed picture data 33, for example, to display the picture to a user or viewer. Display device 34 may be or include any type of display for displaying the reconstructed picture, e.g., an integrated or external display or monitor. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a microLED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any type of other display.

[0321] Coding system 10 further includes a training engine 25. Training engine 25 is configured to train encoder 20 (particularly the neural network module within encoder 20) or decoder 30 (particularly the neural network module within decoder 30) to process input pictures, picture regions, or picture blocks. Note that the embodiments and examples described above describe neural networks that process pictures (sometimes called frames). However, the present disclosure is also applicable to any other granularity, and portions (regions) of complete pictures may be treated as pictures during encoding and decoding.

[0322] It should be noted that a neural network according to an embodiment may use a pre-trained network, so a training engine need not be provided within the system of the present disclosure.

[0323] The training data may be stored in a database (not shown). The training engine 25 performs training based on the training data to obtain a target model (e.g., the target model may be a neural network used for object recognition, object classification, picture segmentation, or picture encoding and reconstruction, etc.). It should be noted that the source of the training data is not limited in this embodiment of the present application. For example, the training data may be obtained from the cloud or another location to perform model training.

[0324] The target model obtained through training by training engine 25 may be applied to coding systems 10 and 40, for example, to source device 12 (e.g., encoder 20) or destination device 14 (e.g., decoder 30) shown in FIG. 26. Training engine 25 may obtain the target model through training in the cloud, and coding system 10 downloads the target model from the cloud and uses the target model. Alternatively, training engine 25 may obtain the target model through training in the cloud and use the target model, and coding system 10 obtains the processing results directly from the cloud.

[0325] 26 depicts source device 12 and destination device 14 as separate devices, an embodiment of the devices may also include both or both functionality: source device 12 or corresponding functionality and destination device 14 or corresponding functionality. In such an embodiment, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, or by separate hardware and / or software, or by any other combination thereof.

[0326] As will be clear to those skilled in the art based on the description, the presence and (exact) division of functions within the source device 12 and / or destination device 14 as shown in FIG. 26 may vary depending on the actual device and application.

[0327] Encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., decoder 30), or both encoder 20 and decoder 30, may be implemented via processing circuitry such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated to video coding, or any combination thereof, as shown in FIG. 27 . Encoder 20 may be implemented via processing circuitry 46 to embody various modules as discussed with respect to encoder 20 and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented via processing circuitry 46 to embody various modules as discussed with respect to decoder 30 and / or any other decoder system or subsystem described herein. The processing circuitry may be configured to perform various operations as discussed below. 29, if the techniques are implemented partially in software, a device may store instructions for the software in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques of the present invention. Either video encoder 20 and video decoder 30 may be integrated as part of a combined encoder / decoder (encoder / decoder, codec) in a single device, for example, as shown in FIG.

[0328] Source device 12 and destination device 14 may include any of a wide variety of devices, including any type of handheld or stationary device, e.g., a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or content distribution server), a broadcast receiver device, a broadcast transmitter device, etc., and may use no operating system or any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0329] In some cases, the video coding system 10 shown in FIG. 26 is merely an example, and the techniques of the present application may be applied to video coding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other examples, data may be retrieved from local memory or streamed over a network. A video encoding device may encode data and store the encoded data in memory, and / or a video decoding device may retrieve data from memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data to memory and / or retrieve data from memory and decode it.

[0330] 27 is an illustration of an example of a video coding system 40 including an encoder 20 and / or a decoder 30, according to an example embodiment. The video coding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by processing circuitry 46), an antenna 42, one or more processors 43, one or more memories 44, and / or a display device 45.

[0331] 27, imaging device 41, antenna 42, processing circuitry 46, video encoder 20, video decoder 30, processor 43, memory 44, and / or display device 45 may be in communication with one another. Video coding system 40 may include only video encoder 20 or only video decoder 30, in different examples.

[0332] In some examples, antenna 42 may be configured to transmit or receive an encoded bitstream of video data. Additionally, in some examples, display device 45 may be configured to present the video data. Processing circuitry 46 may include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. Video coding system 40 may also include optional processor 43. Optional processor 43 may also include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. Additionally, memory 44 may be any type of memory, such as volatile memory (e.g., static random access memory (SRAM) or dynamic random access memory (DRAM)), or non-volatile memory (e.g., flash memory). In a non-limiting example, memory 44 may be implemented by a cache memory. In other examples, logic circuitry 47 and / or processing circuitry 46 may include memory (e.g., a cache) for implementing a picture buffer.

[0333] In some examples, video encoder 20 implemented using logic circuitry may include a picture buffer (e.g., implemented by processing circuitry 46 or memory 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the picture buffer. The graphics processing unit may include video encoder 20 implemented via logic circuitry 46 to embody various functional modules described herein. The logic circuitry may be configured to perform various operations described herein.

[0334] In some examples, decoder 30 may be implemented via logic circuitry 46 in a similar manner to implement the various modules described with reference to decoder 30 and / or any other decoder system or subsystem described herein. In some examples, decoder 30 implemented using logic circuitry may include a picture buffer (implemented by processing circuitry 44 or memory 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the picture buffer. The graphics processing unit may include video encoder 30 implemented via logic circuitry 46 to embody the various modules described herein.

[0335] In some examples, antenna 42 may be configured to receive an encoded bitstream of video data. As described, the encoded bitstream may include data, indicators, index values, mode selection data, etc., related to the video frame coding described herein, such as data related to a coding partition (e.g., transform coefficients or quantized transform coefficients, optional indicators (as described), and / or data defining the coding partition). Video coding system 40 may further include a decoder 30 coupled to antenna 42 and configured to decode the encoded bitstream. Display device 45 is configured to present the video frames.

[0336] In this embodiment of the present application, with respect to the examples described with reference to encoder 20, it should be understood that decoder 30 may be configured to perform the reverse process. With respect to signaling syntax elements, decoder 30 may be configured to receive and parse such syntax elements and correspondingly decode associated video data. In some examples, encoder 20 may entropy encode syntax elements into an encoded video bitstream. In such examples, video decoder 30 may parse such syntax elements and correspondingly decode associated video data.

[0337] Figure 28 is a schematic diagram of a video coding device 400 according to one embodiment of the present disclosure. The video coding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video coding device 400 may be a decoder, such as the video decoder 30 of Figure 26, or an encoder, such as the video encoder 20 of Figure 26. As mentioned above, when referring to a picture (video) encoder and decoder herein, it is not necessarily an encoder and decoder that provides encoding / decoding of picture data, including reconstruction for a human viewer. Rather, the encoder may provide a bitstream, and the decoder may decode picture data or picture feature data to be used in a computer vision task.

[0338] Video coding device 400 includes an ingress port 410 (or input port 410) and receiver unit (Rx) 420 for receiving data, a processor, logic unit, or central processing unit (CPU) 430 for processing the data, a transmitter unit (Tx) 440 for sending the data, and an egress port 450 (or output port 450) for transmitting the data. Video coding device 400 may also include optical-to-electrical (OE) and electrical-to-optical (EO) components coupled to ingress port 410, receiver unit 420, transmitter unit 440, and egress port 450 for the entry and exit of optical or electrical signals.

[0339] The processor 430 is implemented by hardware and software. The processor 430 may be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the ingress port 410, the receiver unit 420, the transmitter unit 440, the egress port 450, and the memory 460. The processor 430 includes a coding module 470 (e.g., a neural network (NN)-based coding module 470). The coding module 470 implements the disclosed embodiments described above. For example, the coding module 470 implements, processes, prepares, or provides various coding operations. Thus, the inclusion of the encoding / decoding module 470 provides substantial improvements to the functionality of the video coding device 400 and affects the switching of the video coding device 400 to different states. Alternatively, the coding module 470 is implemented as instructions stored in the memory 460 and executed by the processor 430.

[0340] Memory 460 may include one or more disks, tape drives, and solid-state drives, and may be used as an overflow data storage device to store programs when such programs are selected for execution and to store instructions and data read during program execution. Memory 460 may be volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0341] FIG. 29 is a simplified block diagram of an apparatus 500 that may be used as either or both of source device 12 and destination device 14 in FIG. 26, according to an example embodiment.

[0342] Processor 502 in device 500 may be a central processing unit. Alternatively, processor 502 may be any other type of device or devices capable of manipulating or processing information now existing or later developed. While the disclosed implementations may be practiced using a single processor as shown, e.g., processor 502, advantages in speed and efficiency may be achieved using two or more processors.

[0343] The memory 504 in the apparatus 500, in one implementation, may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 that is accessed by the processor 502 using a bus 512. The memory 504 may further include an operating system 508 and application programs 510, which include at least one program that enables the processor 502 to perform the methods described herein. For example, the application programs 510 may include applications 1 through N, which further include a video coding application that performs the methods described herein.

[0344] The apparatus 500 may also include one or more output devices, such as a display 518. The display 518, in one example, may be a touch-sensitive display that combines a display with a touch-sensing element operable to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0345] Although shown here as a single bus, bus 512 of device 500 may be comprised of multiple buses. Additionally, secondary storage 514 may be directly coupled to other components of device 500 or may be accessed over a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Accordingly, device 500 may be implemented in a wide variety of configurations.

[0346] The embodiments of the present application relate to the application of neural networks. For ease of understanding, the following will first explain some nouns or terms used in the embodiments of the present application. The nouns or terms are also used as part of the content of the present invention.

[0347] (1) Neural Networks A neural network (NN) is a machine learning model. A neural network may contain neurons. A neuron is a set of neurons, x s and an intercept of 1 as input, and the output of the computation unit may be:

[0348]

number

[0349] where s=1, 2, ..., or n, where n is a natural number greater than 1, and W s is x s where is the weight of the neuron, and b is the bias of the neuron. f is the activation function of the neuron. The activation function is used to introduce nonlinear features into the neural network to convert the input signal in the neuron into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many single neurons. Specifically, the output of a neuron can be the input of another neuron. The input of each neuron can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be a region containing several neurons.

[0350] (2) Deep Neural Networks A deep neural network (DNN), also called a multi-layer neural network, can be understood as a neural network with many hidden layers. The "many" here has no special metric. DNNs are divided based on the location of different layers, and the neural network within a DNN can be divided into three types: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the intermediate layer is the hidden layer. The layers are fully connected. Specifically, any neuron in the i-th layer is surely connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, DNNs are actually not complex in terms of the work of each layer, and are based on the following linear relationship:

[0351]

number

[0352] It is simply expressed as, where:

[0353]

number

[0354] is the input vector,

[0355]

number

[0356] is the output vector,

[0357]

number

[0358] is the bias vector, W is the weight matrix (also called coefficients), and α() is the activation function.

[0359]

number

[0360] is the input vector

[0361]

number

[0362] Since there are many layers in a DNN, there are many coefficients W and bias vectors

[0363]

number

[0364] The definition of these parameters in a DNN is as follows: The coefficient W is used as an example. In a three-layer DNN, the linear coefficient from the fourth neuron in the second layer to the second neuron in the third layer is

[0365]

number

[0366] The superscript 3 denotes the layer in which the coefficient W is located, and the subscript corresponds to the index 2 of the third layer for the output and the index 4 of the second layer for the input. In conclusion, the coefficient from the kth neuron in the (L-1)th layer to the jth neuron in the Lth layer is

[0367]

number

[0368] It is defined as follows. It should be noted that there is no parameter W in the input layer. In a deep neural network, the more hidden layers, the more capable the network is of describing complex cases in the real world. Theoretically, a model with more parameters indicates higher complexity and greater "capacity," indicating that the model can complete more complex learning tasks. Training a deep neural network is the process of learning weight matrices, and the ultimate goal of training is to obtain the weight matrices of all layers of the trained deep neural network (weight matrices formed by the vector W in many layers).

[0369] (3) Convolutional Neural Networks A convolutional neural network (CNN) is a deep neural network with a convolutional structure and a deep learning architecture. In deep learning architectures, multi-layer learning is performed at various levels of abstraction using machine learning algorithms. As a deep learning architecture, a CNN is a feed-forward artificial neural network. Neurons in a feed-forward artificial neural network may respond to pictures input to the neurons. A convolutional neural network includes a feature extractor composed of convolutional layers and pooling layers. The feature extractor can be thought of as a filter. The convolution process can be thought of as using a trainable filter to perform convolution on an input image or a convolutional feature plane (feature map).

[0370] A convolution layer is a neuron layer in a convolutional neural network where a convolution process is performed on an input signal. The convolution layer 221 may include multiple convolution operators. A convolution operator is also called a kernel. In image processing, a convolution operator functions as a filter that extracts specific information from an input image matrix. A convolution operator may essentially be a weight matrix, which is usually predefined. In the process of performing a convolution operation on an image, the weight matrix usually processes pixels at a granularity level of one pixel (or two pixels, depending on the stride value) horizontally in the input image to extract specific features from the image. The size of the weight matrix should be related to the size of the picture. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input picture. During the convolution operation, the weight matrix spans the entire depth of the input picture. Therefore, a convolution output of a single depth dimension is generated through convolution with a single weight matrix. However, in most cases, a single weight matrix is not used, but multiple weight matrices of the same size (rows by columns), i.e., multiple matrices of the same type, are applied. The outputs of the weight matrices are stacked to form the depth dimension of the convolved picture. The dimension here can be understood as being determined based on the aforementioned "multiple." Different weight matrices can be used to extract different features from the picture. For example, one weight matrix is used to extract edge information of the picture, another weight matrix is used to extract a specific color of the picture, and yet another weight matrix is used to blur unwanted noise in the picture. The multiple weight matrices have the same size (rows by columns). The feature maps extracted from the multiple weight matrices with the same size also have the same size, and the extracted feature maps with the same size are combined to form the output of the convolution operation. The weight values in these weight matrices need to be obtained through extensive training in practical applications.Each weight matrix formed by using the weight values obtained through training can be used to extract information from the input image to enable the convolutional neural network to make correct predictions. When a convolutional neural network has multiple convolutional layers, a relatively large number of general features are usually extracted in the initial convolutional layers. The general features are sometimes called low-level features. As the depth of the convolutional neural network increases, the features extracted in subsequent convolutional layers, such as high-level semantic features, become more complex. Features with higher-level semantics are more applicable to the problem to be solved.

[0371] The amount of training parameters often needs to be reduced. Therefore, a pooling layer often needs to be periodically introduced after a convolutional layer. One convolutional layer may be followed by one pooling layer, or multiple convolutional layers may be followed by one or more pooling layers. During picture processing, the pooling layer is used only to reduce the spatial size of the picture. The pooling layer may include an average pooling operator and / or a max pooling operator to sample the input picture to obtain a picture with a relatively small size. The average pooling operator may be used to calculate pixel values within a specific range in the picture to generate an average value. The average value is used as the average pooling result. The max pooling operator may be used to select a pixel with the maximum value within a specific range as the max pooling result. In addition, just as the size of the weight matrix in the convolutional layer needs to be related to the size of the picture, the operators in the pooling layer also need to be related to the size of the picture. The size of the processed picture output from the pooling layer may be smaller than the size of the picture input to the pooling layer. Each pixel in the picture output from the pooling layer represents the average or maximum value of the corresponding sub-region of the picture input to the pooling layer.

[0372] After the processing performed in the convolutional / pooling layer, as described above, the convolutional neural network is not ready to output the required output information because only features are extracted and the resulting parameters are reduced from the input image in the convolutional / pooling layer. However, to generate the final output information (required class information or other related information), the convolutional neural network needs to use neural network layers to generate the output of one required class or a group of required classes. Therefore, the convolutional neural network layer may include multiple hidden layers. The parameters included in the multiple hidden layers may be obtained through pre-training based on related training data for a specific task type. For example, the task type may include image recognition, image classification, and super-resolution image reconstruction.

[0373] Optionally, in the neural network layer, multiple hidden layers are followed by an output layer of the entire convolutional neural network. The output layer has a loss function similar to multi-class cross entropy, which is used to calculate prediction errors in particular. Once forward propagation through the entire convolutional neural network is completed, backward propagation begins to update the weight values and deviations of each of the above-mentioned layers in order to reduce the loss of the convolutional neural network and the error between the results output by the convolutional neural network and the ideal results by using the output layer.

[0374] (4) Recurrent Neural Networks Recurrent neural networks (RNNs) are used to process sequence data. In traditional neural network models, layers are fully connected, from the input layer to the hidden layer and then to the output layer, and nodes in each layer are not connected. While such general neural networks solve many difficult problems, they still cannot solve many other problems. For example, when predicting a word in a sentence, adjacent words in the sentence are not independent, so the previous word usually needs to be used. RNNs are called recurrent neural networks because the current output of a sequence is related to the previous output of the sequence. In a specific representation, the network memorizes previous information and applies it to the calculation of the current output. Specifically, nodes in the hidden layer are connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. In theory, RNNs can process sequence data of any length. Training an RNN is the same as training a traditional CNN or DNN. The backpropagation algorithm is also used, but there is a difference: when an RNN is extended, parameters such as W of the RNN are shared. This is different from the traditional neural network described in the previous example. In addition, when using the gradient descent algorithm, the output at each step depends not only on the network state at the current step, but also on the network state at several previous steps. The learning algorithm is called the backward propagation through time (BPTT) algorithm.

[0375] Now that convolutional neural networks exist, why do we need recurrent neural networks? The reason is simple. In convolutional neural networks, elements are assumed to be independent of each other, and inputs and outputs are as independent as cats and dogs. However, in the real world, many elements are interrelated. For example, stock prices change over time. In another example, a person says, "I love traveling, and my favorite place is Yunnan." I will go there if I have the chance." Here, people should know that the person is going to "Yunnan." This is because people can infer the answer based on the context. However, how can a machine know this? Enter RNNs. RNNs are intended to enable machines to memorize like humans. Therefore, the output of an RNN must depend on both the current input information and previously memorized information.

[0376] (5) Recursive residual convolutional neural network (RR-CNN)

[0377] (5) Artificial neural network (ANN)

[0378] (6) Loss function In the process of training a deep neural network, since it is expected that the output of the deep neural network will be as close as possible to the actual expected predicted value, the current network's predicted value can be compared with the actual expected target value. Then, the weight vectors of each layer of the neural network are updated based on the difference between the predicted value and the target value. (Indeed, there is usually an initialization process before the first update, specifically, parameters are pre-configured for all layers of the deep neural network.) For example, if the network's predicted value is large, the weight vectors are adjusted to reduce the predicted value, and adjustments are continuously made until the deep neural network can predict the actual expected target value or a value very close to the actual expected target value. Therefore, "how to obtain the difference between the predicted value and the target value through comparison needs to be defined in advance. This is the loss function or objective function. Loss functions and objective functions are important formulas used to measure the difference between the predicted value and the target value. The loss function is used as an example. A higher output value (loss) of the loss function indicates a larger difference. Therefore, training a deep neural network is a process of minimizing the loss as much as possible.

[0379] (7) Backpropagation Algorithm A convolutional neural network may correct parameter values in an initial super-resolution model during the training process according to a backpropagation (BP) algorithm so that the error loss in reconstructing the super-resolution model is smaller. Specifically, the input signal is forward-transferred until an error loss occurs at the output, and the parameters in the initial super-resolution model are updated based on the backpropagation error loss information to converge the error loss. The backpropagation algorithm is an error-loss-centered backpropagation movement intended to obtain parameters such as the weight matrix of the optimal super-resolution model.

[0380] (8) Generative Adversarial Networks Generative adversarial networks (GANs) are deep learning models. They include at least two models: a generative model and a discriminative model. The two models learn from each other to generate better outputs. Both the generative model and the discriminative model can be neural networks, specifically, deep neural networks or convolutional neural networks. The basic principle of GANs is as follows. Using GANs to generate pictures as an example, assume there are two networks: G (generator) and D (discriminator). G is a network for generating pictures. G receives random noise z and generates pictures by using the noise, where the picture is denoted as G(z). D is a discriminative network used to determine whether a picture is "real." The input parameter of D is x, where x represents a picture, and the output D(x) represents the probability that x is a real picture. A value of 1 for D(x) indicates that the picture is 100% authentic. A value of 0 for D(x) indicates that the picture cannot be authentic. In the process of training a generative adversarial network, the goal of the generative network G is to generate a picture that is as authentic as possible to deceive the discriminant network D, and the goal of the discriminant network D is to distinguish the picture generated by G from an authentic picture as much as possible. In this way, there is a dynamic "gaming" process between G and D, specifically, the "adversary" in the "generative adversarial network." The final gaming result, in an ideal state, is that G can generate a picture G(z) that is difficult to distinguish from an authentic picture, and D has difficulty determining whether the picture generated by G is authentic, specifically, D(G(z))=0.5. In this way, an excellent generative model G can be obtained and used to generate pictures.

[0381] Those skilled in the art will appreciate that the functions described with reference to the various illustrative logical blocks, modules, and algorithm steps disclosed and described herein may be implemented by hardware, software, firmware, or any combination thereof. If implemented by software, the functions described with reference to the illustrative logical blocks, modules, and steps may be recorded in or transmitted via a computer-readable medium as one or more instructions or code and performed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium, such as a data storage medium, or any communication medium that facilitates the transfer of a computer program from one place to another (according to a communication protocol). As such, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium, such as a signal or carrier. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this application. A computer program product may include a computer-readable medium.

[0382] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other compact disk storage, magnetic disk storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Additionally, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or another remote source over coaxial cable, fiber optic, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, or microwave, the coaxial cable, fiber optic, twisted pair, DSL, or wireless technologies such as infrared, radio, or microwave are included within the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals, or other transitory media, and actually refer to non-transitory, tangible storage media. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically by using lasers. Combinations of the foregoing items should also be included within the scope of computer-readable media.

[0383] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor," as used herein, may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described with reference to the exemplary logic blocks, modules, and steps described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or may be incorporated into a combined codec. Additionally, the techniques may be implemented entirely within one or more circuits or logic elements.

[0384] The technology in this application may be implemented in a variety of apparatuses or devices, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). To highlight functional aspects of an apparatus configured to implement the disclosed technology, various components, modules, or units are described herein, but are not necessarily implemented by different hardware units. In practice, as described above, the various units may be combined into a codec hardware unit in combination with appropriate software and / or firmware, or may be provided by interoperable hardware units (including one or more processors as described above).

[0385] The above description is merely an example of a specific implementation form of the present application and is not intended to limit the protection scope of the present application. Any variations or replacements that can be easily conceived by those skilled in the art within the technical scope disclosed in the present application shall fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

[0386] In summary, the present disclosure provides a method and apparatus for processing picture data or picture feature data using a neural network with two or more layers. The present disclosure may be applied to the field of artificial intelligence (AI)-based video or picture compression technology, particularly to the field of neural network-based video compression technology. According to some embodiments, two types of data are combined during processing, including processing by a neural network. The two types of data are obtained from different stages of processing by the network. Some of the advantages may include higher scalability and more flexible design of the neural network architecture, which may further lead to better encoding / decoding performance. [Explanation of symbols]

[0387] 1 Encoder, Processing Unit, Unit 2 Quantizer, Unit 3 units, HyperEncoder 4 decoders, units 5 Arithmetic Encoder, AE 6 Arithmetic decoder AD, AD 7 units, Hyper Decoder 8 Quantizer 9 Arithmetic Encoder, AE 10 AD, Video Coding System, Coding System, System 12 Source Devices 13 Encoded picture data, communication channel, encoded data 14 Destination Device 16 Picture Source 17 Picture or Picture Data, Raw Picture or Raw Picture Data 18 Preprocessor, preprocessing unit, picture preprocessor 19 Preprocessed Picture, Preprocessed Picture Data 20 Video Encoder, Encoder 21 Encoded Picture Data 22 Communication interface or communication unit, communication interface 28 Communication interface or communication unit, communication interface 30 Video decoder, decoder 32 Post-processor, post-processing unit 34 Display Devices 40 Video Coding System 41 Imaging Device 42 Antenna 43 processors 44 memory 45 Display Devices 46 Processing Circuit 47 Logic Circuits 57-59 Upsampling Layers 57~512 layers 221 Convolutional Layer 400 Video Coding Device 410 upsampling layer, inlet port, input port 411 upsampling layers 420 layer, receiver unit 430 Processor, Logic Unit, or Central Processing Unit 440 Transmitter Unit 450 outlet port 460 memory 470 coding modules, neural network (NN)-based coding modules, encoding / decoding modules 500 devices 502 processor 504 memory 506 Code and Data 508 Operating Systems 510 Application Program 510~512 upsampling layers 512 Bus 514 Secondary Storage 518 Display 600A Encoder Neural Network, Encoder Network, Encoder Part 600B decoder part, decoder network 602 channels, input picture channels 610 Collection, Collector 620 bitstream 630 bitstream 692 Reconstructed Pictures 710 predicted frames 720 frames 730 Example, Mosaic Picture 740 channels 900A Neural Network 900B Neural Network 902 Pictures 910_1 Collection conditions, conditions 910_2 Collection conditions, conditions 910_3 Collection conditions, conditions 920_1 Collection Unit 920_2 Collection Unit 920_3 Collection Unit 940 bitstream 945 bitstream 950 auxiliary data 960_1 Collection, collection unit, collector 960_2 Collection, collection unit, collector 960_3 Collection, collection unit, collector 1002 Input Picture 1010 downsampling layers 1010A Encoder 1010B decoder 1030 Collection Unit 1050 Collection Conditions 1060 decoder 1070 upsampling layer 1092 processed pictures 1100A Encoder part 1100B decoder part, decoder 1110 Prediction signal (predi) 1120 Residual (resii) 1140 bitstream 1150 Collection Unit, Collection Stage 1160 Residual 1310 Collection Unit 1320 Collection Conditions 1400A Neural Network, Encoder Side Network, Network 1400B Decoder Network, Network 1402 Input Picture 1450 bitstream 1492 Reconstructed Pictures 1506 Top, Network 1510 Motion Estimation 1520 Motion Compensation 1530 Motion Compensation 1560 parts, collection units 1561 Collection Unit 1572 bitstream 1586 encoding and decoding network, network 1590 bitstream 1601 Input Picture 1605 first layer or block or layer, first layer 1606 Network 1660 Collector Unit 1665 Collection 1672 Residual Bitstream 1690 bitstream 1701 Input picture data 1702 Processing 1706 Neural Networks 1760 First Data 1765 Second data, prediction 1772 bitstream 1786 Neural Networks 1790 bitstream 2010 layer 2012 layer 2020 tier 2022 layers 2050 bitstream

Claims

1. 1. A method for processing feature data of one or more pictures from a bitstream (1772) using a neural network (1786) including two or more stages (K-0) including a first stage (i+r) and a second stage (i), the method comprising: obtaining first data (1760) based on the bitstream (1772); processing the first data (1760) using the neural network (1786), obtaining second data (1765) from the second stage of the neural network, the second data being based on data previously processed by the neural network (1786); using the first data (1760) together with the second data (1765) to generate inputs to the first stage of the neural network, wherein the first stage (i+r) precedes the second stage (i) in the processing of feature data by the neural network (1786); and outputting a result (1702) of the processing performed by the step of processing the first data (1760) using the neural network (1786); Including, method.

2. 2. The method of claim 1, wherein the first data is obtained in a third stage of the neural network that precedes or is equal to the first stage and the second stage of the feature data processing of the neural network.

3. 3. The method of claim 2, wherein the first data (1760) represents a prediction error and the second data (1765) represents a prediction.

4. The prediction is Obtaining reference feature data, which is feature data output by the neural network (1786) in the second step; obtaining prediction information including motion information or spatial information regarding the reference feature data based on the bitstream (1772); generating the prediction based on the reference feature data and the prediction information; obtained by The method of claim 3.

5. the prediction error is obtained by processing a current picture using the neural network (1786); the prediction information is motion information, and the reference feature data is generated by the neural network (1786) processing picture data representing a picture preceding the current picture in decoding order. The method of claim 4.

6. 6. The method of claim 3, wherein using the first data with the second data (1765) comprises element-wise addition of the prediction or rescaled prediction and the prediction error or rescaled prediction error.

7. The method of claim 5 or 6, wherein the motion information comprises motion vectors.

8. 8. The method of claim 1, wherein the second stage is the output of the neural network (1786).

9. 9. The method of claim 2, wherein the first stage is an input to the neural network (1786).

10. the first stage is the input of the neural network (1786); the first data is entropy encoded data; the second stage is a stage different from the step of outputting the neural network (1786); the second data is probabilistic model data relating to the feature data of the second stage; 3. The method according to claim 1 or 2.

11. The method of claim 10 , wherein the second data is probability model data for entropy decoding of the first data.

12. the locations of the first stage and / or the second stage are configurable within the neural network (1786), and the method includes configuring the locations of the first stage and / or the second stage according to collection conditions based on one or more picture coding parameters.

12. The method according to any one of claims 1 to 11.

13. further comprising parsing a stage selection indicator from the bitstream (1772) that specifies the first stage and / or the second stage; the positions of the first stage and the second stage within the neural network (1786) are configured according to the analyzed stage selection indicator.

13. The method according to any one of claims 1 to 12.

14. 1. A method for processing at least one picture using a neural network (1706) including two or more stages, including a first stage and a second stage, to generate a bitstream (1772), the method comprising: processing the at least one picture using the neural network (1706), obtaining first data (1760) based on the at least one picture; acquiring second data (1765) in the second stage, the second data (1765) being based on data previously processed by the neural network (1706); using the first data (1760) together with the second data (1765) to generate inputs for the first stage of the neural network (1706), the first stage preceding the second stage in the processing of feature data in the neural network (1706); and including in the bitstream (1772) feature data obtained by processing the at least one picture using the neural network (1706); Including, method.

15. 15. The method of claim 14, wherein the first data (1760) is obtained in a third stage of the neural network (1706) that precedes or is equal to the first stage and the second stage in the feature data processing of the neural network (1706).

16. 16. The method of claim 15, wherein the first data (1760) represents a prediction error and the second data (1765) represents a prediction.

17. The prediction is Obtaining reference feature data, which is feature data output by the neural network (1706) in the second stage; obtaining prediction information, including motion information or spatial information, regarding the reference feature data based on the at least one picture; generating the prediction based on the reference feature data and the prediction information; Inserting the obtained prediction information into the bitstream (1772). is obtained by 17. The method of claim 16.

18. the prediction error is obtained by processing a current picture using the neural network (1706); the prediction information is motion information, and the reference feature data is generated by the neural network (1706) processing picture data representing a picture preceding the current picture in decoding order.

18. The method of claim 17.

19. 19. The method of claim 15, wherein the step of using the first data (1760) with the second data (1765) further comprises a step of including in the bitstream (1772) an element-wise subtraction result of a prediction or a rescaled prediction from the first data (1760) or rescaled first data (1760).

20. 20. The method of claim 18 or 19, wherein the motion information comprises motion vectors.

21. 21. The method of any one of claims 14 to 20, wherein the second stage is an output of a decoding neural network (1706) representing reconstructed picture data.

22. 22. The method of any one of claims 15 to 21, wherein the first stage is the output of the neural network (1706).

23. the first stage is the output of the neural network (1706); the first data is processed data to be entropy encoded; the second stage is a stage different from the input stage of the neural network (1706); the second data is probabilistic model data relating to the feature data of the second stage; 16. The method of claim 14 or 15.

24. 24. The method of claim 23, wherein the second data is probability model data for entropy encoding of the first data.

25. the location of the first stage and / or the second stage is configurable within the neural network (1706) or the neural network (1786); the method comprising configuring the positions of the first stage and / or the second stage according to a collection condition based on one or more picture coding parameters; 25. The method of any one of claims 1 to 13 or any one of claims 14 to 24.

26. determining and including in the bitstream (1772) a stage selection indicator that designates the first stage and / or the second stage; the positions of the first stage and the second stage within the neural network (1706) or the neural network (1786) are configured according to the determined stage selection indicator; 26. The method of any one of claims 1 to 13 or any one of claims 14 to 25.

27. 27. The method of claim 26, wherein the determining the stage selection indicator is based on an optimization procedure performed according to a cost function including one or more of rate, distortion, latency, accuracy, and complexity.

28. A computer program (510) stored on a computer-readable storage medium (540) and comprising code instructions that, when executed on one or more processors, cause said one or more processors to perform the method of any one of claims 1 to 27.

29. 1. An apparatus for processing feature data of one or more pictures from a bitstream (1772) using a neural network (1786) including two or more stages, including a first stage and a second stage, the apparatus comprising: obtaining first data (1760) based on the bitstream (1772); processing the first data (1760) using the neural network (1786); obtaining second data (1765) from the second stage of the neural network (1786) based on data previously processed by the neural network (1786); using the first data (1760) together with the second data (1765) to generate inputs to the first stage of the neural network (1786), the first stage preceding the second stage in processing feature data of the neural network (1786); and outputting the results of the processing; a processing circuit configured to perform Device.

30. 1. An apparatus for processing at least one picture using a neural network (1706) including two or more stages, including a first stage and a second stage, to generate a bitstream (1772), the apparatus comprising: processing the at least one picture using the neural network (1706), obtaining first data (1760) based on the at least one picture; obtaining second data (1765) in the second stage of the processing, the second data (1765) being based on data previously processed by the neural network (1706); using the first data (1760) together with the second data (1765) to generate inputs for the first stage of the neural network (1706), the first stage preceding the second stage of feature data processing in the neural network (1706); and including in the bitstream (1772) feature data obtained by processing the at least one picture using the neural network (1706); a processing circuit configured to perform Device.

Images