Configurable location of auxiliary information input to a picture data processing neural network - Patents.com

Abstract

The present application provides a method and apparatus for processing picture data or picture feature data using a neural network having two or more layers. The present disclosure may be applied in the field of artificial intelligence (AI)-based video or picture compression technology, in particular in the field of neural network-based video compression technology. According to some embodiments, a location in the neural network where auxiliary information may be input for processing is selectable based on a collection condition. The collection condition may evaluate whether some preconditions are met. Some of the advantages may include better performance in terms of rate and / or disclosure due to the effect of increased flexibility in neural network configuration.

Description

[Technical field]

[0001] Embodiments of the present invention relate to the field of artificial intelligence (AI) based video or picture compression techniques, and in particular to methods and apparatus for processing picture feature data to generate a bitstream or for processing picture feature data using a neural network to decode it into a bitstream. [Background technology]

[0002] Video coding (video encoding and decoding) is used in a wide range of digital video applications, such as broadcast digital TV, video transmission over the Internet and mobile networks, real-time conversation applications such as video chat, video conferencing, DVDs and Blu-ray discs, video content acquisition and editing systems, and camcorders for security applications.

[0003] The amount of video data required to depict even a relatively short video can be substantial, and as a result, difficulties may arise when the data is streamed or otherwise communicated over a communication network with limited bandwidth capacity. Thus, video data is typically compressed before being communicated over modern telecommunications networks. If the video is stored on a storage device, the size of the video may also be an issue, since memory resources may be limited. Video compression devices often use software and / or hardware at the source to code the video data before transmission or storage, thereby reducing the amount of data required to represent a digital video picture. The compressed data is then received at the destination by a video decompression device, which decodes the video data. With limited network resources and a constantly increasing demand for higher video quality, improved compression and decompression techniques are desired that improve compression ratios with little or no sacrifice in picture quality.

[0004] In recent years, deep learning has become increasingly popular in the field of picture and video encoding and decoding. Summary of the Invention [Means for solving the problem]

[0005] The present application provides methods and apparatus that can improve the configurability of neural networks, thereby achieving greater efficiency.

[0006] The above and other objects are achieved by the subject matter of the independent claims. Further implementations are evident from the dependent claims, the description and the drawings.

[0007] Particular embodiments are outlined in the accompanying independent claims, while other embodiments are outlined in the dependent claims.

[0008] According to a first aspect, the present disclosure relates to a method for processing picture feature data from a bitstream using a neural network comprising a plurality of neural network layers, the method comprising the steps of obtaining picture feature data from the bitstream and processing the picture feature data using the neural network, wherein for each of one or more preconfigured positions in the neural network, the processing comprises determining whether to collect auxiliary data for processing by one of a plurality of neural network layers at said preconfigured positions based on a collection condition, and if it is determined that auxiliary data should be collected, the processing by the layer at said preconfigured position is based on the auxiliary data.

[0009] A preconfigured location is a location in the neural network where auxiliary information may be collected. Whether auxiliary information is actually collected at a particular preconfigured location is determined in the decision step described above. The term "location" refers to a node in the neural network. A node corresponds to feature tensors input to and / or output from a layer (or a block of layers or modules depending on the architecture of the neural network). In particular, a location may be specified by the number of neural network layers that precede the location in the processing direction.

[0010] Auxiliary information is any information about the picture data or picture feature data that is added to the neural network. Such information may be input to the neural network to further improve processing. Some specific examples of auxiliary information are provided in the following exemplary implementations.

[0011] A collection condition is a condition or precondition that must be met for auxiliary data to be input at a particular preconfigured location. The collection condition may include a comparison of some picture characteristics or picture feature characteristics with a threshold value to determine whether to collect auxiliary information for a particular location. The picture characteristics or picture feature characteristics may be known to the encoder and decoder so that no additional signaling is required. Alternatively, or in addition, the collection condition may be configured by the encoding side by setting an indicator of whether auxiliary information should be collected for a preconfigured location. The indicator may be provided in the bitstream available at the decoder.

[0012] Configuring the location of inputting auxiliary information into the neural network has the effect of greater flexibility, allowing dynamic neural network architecture modification. Such flexibility may result in better adaptation based on acquisition conditions, leading to more efficient encoding and / or decoding.

[0013] In a possible implementation, applying the collection condition in the determination results in the auxiliary data being collected for a single one of one or more pre-configured locations.

[0014] Such implementations provide a similar effect to selecting, for a particular auxiliary information, the location in the neural network where the auxiliary information should be collected, which allows the assistance to be provided at the location that may be most appropriate according to some criteria, such as processing time or complexity and / or coding efficiency, which may include rate or distortion.

[0015] In a possible alternative implementation, applying a collection condition in a determination results in the auxiliary data being collected for two or more of the pre-configured locations.

[0016] Such an alternative implementation to the one described above may ensure that auxiliary information is available at any layer that may benefit from it, which may further increase the configurability and thus flexibility of the network.

[0017] In a possible implementation, there are two or more of the preconfigured locations (the processing is performed for two or more preconfigured locations), the auxiliary data is scalable in size to match the dimensions of the input channels processed by the layer at two or more of the preconfigured locations, and as a result of applying the collection conditions in the determination, the auxiliary data is (i) collected, or (ii) collected and scaled for a single one of the preconfigured locations.

[0018] Thus, the auxiliary data may be appropriately scaled to enable its combination with the feature data. Such scaling allows for the provision of a wide variety of auxiliary information that may come from different stages or sources.

[0019] In a possible implementation, the collection conditions are based on picture characteristics or picture feature data characteristics obtained from the bitstream.

[0020] This implementation allows for content adaptation of the neural network and may improve the performance of the encoding or decoding or other processing performed by the neural network.

[0021] In a possible implementation, the picture characteristics or picture feature data characteristics include resolution and the collection conditions include a comparison of the resolution to a preconfigured resolution threshold.

[0022] Resolution is a relevant decision criterion since it affects the level of detailed features, which may degrade in quality when further downsampled or otherwise processed in a lossy manner.

[0023] In a possible implementation, the picture is a video picture, the picture characteristics include a picture type, and the collection conditions include determining whether the picture type is a temporally predicted picture type or a spatially predicted picture type.

[0024] Picture type is a relevant decision criterion since it affects the quality of the prediction. Furthermore, encoding intra pictures with high quality may be desirable since it may also affect inter pictures, which typically utilize intra-coded pictures as a reference for temporal prediction.

[0025] According to an exemplary implementation, the method further includes obtaining an instruction from the bitstream specifying whether to collect auxiliary data for one or more preconfigured locations, the collection conditions for each of the one or more preconfigured locations being as follows: if the instruction specifies that auxiliary data should be collected for the preconfigured location, the decision is positive; and if the instruction specifies that auxiliary data should not be collected for the preconfigured location, the decision is negative.

[0026] Explicit signaling of the collection location allows for greater configurability and allows for closer adaptation to the picture's content even though the predefined collection conditions may not capture the picture's content.

[0027] In a possible implementation, the auxiliary data provides information about the picture feature data that is processed by the neural network to generate the output.

[0028] Providing additional information regarding picture feature data may improve reconstruction and / or other types of processing. Depending on the processing type, different auxiliary information may be relevant and applicable.

[0029] In a possible implementation, the auxiliary data includes predictive data that is a prediction of the picture or a prediction of picture feature data after processing by one or more of the layers of the neural network.

[0030] Moving picture (video) coding efficiency may depend heavily on removing temporal correlation from adjacent pictures, therefore providing or predicting data or prediction error data may improve neural network processing.

[0031] In a possible implementation, the auxiliary data is a combined pair of predicted data and supplemental data that is combined with the predicted data.

[0032] Collecting predictions and prediction errors in the same region may provide a relatively simple and effective way of collecting step selections.

[0033] In a possible implementation, the prediction data and the supplementary data have the dimensions of data processed by layers at different positions in the neural network.

[0034] In other words, predictions may be provided at a different stage than the stage at which prediction residuals or supplemental information are provided. Thus, efficiencies by better adapting to the content may be achieved.

[0035] In a possible implementation, the neural network comprises a sub-network for lossless decoding having at least one layer, the auxiliary data being input to said sub-network for lossless decoding.

[0036] Recently, variable autoencoders with hyper-priors have been adopted and may be efficiently supported by conditional collection (based on collection conditions) of the hyper-prior. In some cases, it may be beneficial (leading to more compact bitstreams) if a sub-network is used to derive the probability model parameters. In other cases, it may be too computationally expensive if a default or context-based probability model already works well.

[0037] In a possible implementation, the neural network is trained to perform at least one of still picture decoding, video picture decoding, still picture filtering, video picture filtering, and machine vision processing including object detection, object recognition, or object classification.

[0038] This implementation allows the deployment of the methods described herein to many important applications that may benefit from flexible location (and possibly also resolution) of the auxiliary information.

[0039] In one embodiment, the method is performed for each of a plurality of auxiliary data including first auxiliary data and second auxiliary data, where the first auxiliary data is associated with a first set of one or more preconfigured locations and the second auxiliary data is associated with a second set of one or more preconfigured locations.

[0040] This allows the neural network to efficiently collect two or more types of auxiliary information, even if they are quite different, e.g., prediction-related auxiliary information and lossless coding-related auxiliary information that are naturally used in different positions.

[0041] In a possible implementation, the first set of one or more preconfigured locations and the second set of one or more preconfigured locations share at least one preconfigured location.

[0042] This is an example implementation that may be suitable for some applications: for example, in the case of prediction and prediction error, all pre-configured positions may be shared.

[0043] In a possible implementation, the neural network is trained to process video pictures, and the decision of whether to collect auxiliary data for processing by the layer at the preconfigured location is made for every predetermined number of video pictures, where the predetermined number of video pictures is one or more.

[0044] Setting some granularity for collection location adaptation may contribute to efficiency as it may involve less complexity and, in some embodiments, less signaling overhead.

[0045] According to a second aspect, the invention relates to a method for processing a picture using a neural network comprising a plurality of neural network layers to generate a bitstream. The method comprises a step of processing a picture using the neural network. The processing step comprises a step of determining, for each of one or more preconfigured positions in the neural network, whether to collect auxiliary data for processing by a layer at said preconfigured positions based on a collection condition, and if it is determined that auxiliary data should be collected, the processing by the layer at said preconfigured positions is based on the auxiliary data. The method further comprises a step of inserting data obtained by processing the picture by the neural network into the bitstream.

[0046] The encoding portion of this disclosure may provide the same advantages as discussed above for the decoding portion: the encoder prepares a bitstream and provides it to the decoder so that the decoder can decode or reconstruct the data with the desired quality and application in mind.

[0047] To avoid redundancy, the advantages provided for the corresponding decoding process claims apply in an analogous manner to the encoding process.

[0048] In a possible implementation, applying the collection condition in the determination results in the auxiliary data being collected for a single one of one or more pre-configured locations.

[0049] In a possible implementation, applying a collection condition in a determination results in the auxiliary data being collected for two or more of the pre-configured locations.

[0050] In a possible implementation, there are two or more of the preconfigured locations (the processing is performed for two or more preconfigured locations), the auxiliary data is scalable in size to match the dimensions of the input channels processed by the layer at two or more of the preconfigured locations, and as a result of applying the collection conditions in the determination, the auxiliary data is (i) collected, or (ii) collected and scaled for a single one of the preconfigured locations.

[0051] In a possible implementation, the collection conditions are based on picture characteristics or picture feature data characteristics included in the bitstream.

[0052] In a possible implementation, the picture characteristics or picture feature data characteristics include resolution and the collection conditions include a comparison of the resolution to a preconfigured resolution threshold.

[0053] In a possible implementation, the picture is a video picture, the picture characteristics include a picture type, and the collection conditions include determining whether the picture type is a temporally predicted picture type or a spatially predicted picture type.

[0054] In some embodiments, the method further includes generating an indication specifying whether to collect auxiliary data for one or more preconfigured locations, and including the indication in the bitstream.

[0055] According to an example implementation, the method further includes selecting, for one or more preconfigured positions, whether to collect auxiliary data based on optimizing a cost function including at least one of rate, distortion, accuracy, speed, or complexity.

[0056] Determining the location for collection based on a cost function can improve the adaptation of the neural network and the results to the desired requirements. Such optimization can therefore improve performance. In combination with indicating the collection location thus obtained, flexibility is further improved.

[0057] In a possible implementation, the auxiliary data provides information about the picture feature data that is processed by the neural network to generate the output.

[0058] In a possible implementation, the auxiliary data includes predictive data that is a prediction of the picture or a prediction of picture feature data after processing by one or more of the layers of the neural network.

[0059] In a possible implementation, the auxiliary data is a combined pair of predicted data and supplemental data to be combined with the predicted data.

[0060] In a possible implementation, the prediction data and the supplementary data have the dimensions of data processed by layers at different positions in the neural network.

[0061] In a possible implementation, the neural network comprises a sub-network for lossless decoding having at least one layer, the auxiliary data being input to said sub-network for lossless decoding.

[0062] In a possible implementation, the neural network is trained to perform at least one of still picture encoding, video picture encoding, still picture filtering, video picture filtering, and machine vision processing including object detection, object recognition, or object classification.

[0063] According to an example implementation, the method is performed for each of a plurality of auxiliary data including first auxiliary data and second auxiliary data, where the first auxiliary data is associated with a first set of one or more preconfigured locations and the second auxiliary data is associated with a second set of one or more preconfigured locations.

[0064] In a possible implementation, the first set of one or more preconfigured locations and the second set of one or more preconfigured locations share at least one preconfigured location.

[0065] In a possible implementation, the neural network is trained to process video pictures, and the decision of whether to collect auxiliary data for processing by the layer at the preconfigured location is made for every predetermined number of video pictures, where the predetermined number of video pictures is one or more.

[0066] According to a third aspect, the invention relates to an apparatus for processing picture feature data from a bitstream using a neural network comprising a plurality of neural network layers. The apparatus comprises a processing circuit configured to obtain picture feature data from the bitstream and to process the picture feature data using the neural network for each of one or more preconfigured positions in the neural network. The processing comprises determining whether to collect auxiliary data for processing by one of the plurality of neural network layers at said preconfigured positions based on a collection condition, and if it is determined that auxiliary data should be collected, the processing by the layer at said preconfigured position is based on the auxiliary data.

[0067] For the advantageous effects of the present disclosure, please refer to the description of the first aspect. Details will not be described again here. The decoding device has a function of implementing the actions in the example of the method in the first aspect. The functions can be implemented by hardware, or can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. In a possible implementation form, the decoding device includes a feature data input module configured to obtain picture feature data from a bitstream, and a neural network module configured to perform the above-mentioned processing. These modules may perform corresponding functions in the example of the method in the first aspect. For details, please refer to the detailed description in the example of the method. Details will not be described again here.

[0068] According to a fourth aspect, the invention relates to an apparatus for processing pictures using a neural network comprising a plurality of neural network layers to generate a bitstream, the apparatus comprising a processing circuit configured to process pictures using a neural network, the processing comprising: for each of one or more preconfigured positions in the neural network, determining based on a collection condition whether to collect auxiliary data for processing by the layer at said preconfigured positions, and if it is determined that auxiliary data should be collected, the processing by the layer at said preconfigured positions inserts data obtained by processing the picture by the neural network based on the auxiliary data into the bitstream.

[0069] For the advantageous effects of the present invention, please refer to the description of the second aspect. Details will not be described again here. The encoding device has a function of implementing the actions in the example of the method in the second aspect. The functions can be implemented by hardware, or can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. In a possible implementation form, the encoding device includes a neural network module configured to perform the above-mentioned processing, and a bitstream output module configured to process the picture by the neural network to insert the obtained data into a bitstream and output the bitstream. These modules may perform the corresponding functions in the example of the method in the second aspect. For details, please refer to the detailed description in the example of the method. Details will not be described again here.

[0070] The method according to the first aspect of the invention may be performed by an apparatus according to the third aspect of the invention. Further features and implementations of the method according to the first aspect of the invention depend directly on the functionality and implementations of the apparatus according to the third aspect of the invention.

[0071] The method according to the second aspect of the invention may be performed by an apparatus according to the fourth aspect of the invention. Further features and implementations of the method according to the second aspect of the invention depend directly on the functionality and implementations of the apparatus according to the fourth aspect of the invention.

[0072] According to a fifth aspect, the invention relates to a video stream decoding device including a processor and a memory, the memory storing instructions for causing the processor to perform the method according to the first aspect.

[0073] According to a sixth aspect, the invention relates to a video stream encoding device including a processor and a memory, the memory storing instructions for causing the processor to perform the method according to the second aspect.

[0074] According to a seventh aspect, a computer-readable storage medium is proposed having stored thereon instructions which, when executed, cause one or more processors to encode video data, the instructions causing the one or more processors to perform a method according to the first or second aspect, or any possible embodiment of the first or second aspect.

[0075] According to an eighth aspect, the present invention relates to a computer program product comprising a program code for performing the method according to the first or second aspect or any possible embodiment of the first or second aspect, when the program code is executed on a computer.

[0076] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, drawings, and claims.

[0077] In the following, embodiments of the invention are explained in more detail with reference to the accompanying figures and drawings. [Brief description of the drawings]

[0078] [Figure 1]FIG. 2 is a block diagram illustrating an example network architecture on the encoder and decoder sides including a hyper-prior model. [Diagram 2] FIG. 1 is a block diagram showing a general network architecture on the encoder side including a hyper-prior model. [Diagram 3] FIG. 1 is a block diagram showing a general network architecture on the decoder side including a hyper-prior model. [Figure 4] FIG. 1 is a schematic diagram showing a general scheme of a neural network based encoder and decoder. [Diagram 5] FIG. 2 is a block diagram illustrating an example autovariant encoder and decoder. [Figure 6] FIG. 1 is a general block diagram illustrating the application of auxiliary information at predefined stages of encoding and decoding neural networks. [Figure 7] FIG. 13. Visualization of the predicted signal in the signal domain after processing with one processing layer with downsampling and after processing with N layers with downsampling. [Figure 8] FIG. 1 is a block diagram illustrating an example video coding pipeline that combines a traditional block-based codec with a deep learning-based approach. [Figure 9] FIG. 1 is a block diagram illustrating an example configurable conditional autoencoder with multiple selectable positions for collecting auxiliary information. [Figure 10] FIG. 1 is a block diagram illustrating an example configurable conditional autoencoder with one selectable position for collecting auxiliary information. [Figure 11] FIG. 1 is a block diagram illustrating an example neural network process including multiple collection opportunities (positions) for collecting prediction information and residual information as auxiliary information. [Figure 12] FIG. 1 is a schematic diagram illustrating a comparison of performance for downsampling and reconstructing two example pictures with different characteristics. [Figure 13]FIG. 13 is a block diagram illustrating the application of a configurable collection unit for the purpose of inputting indicators of a probabilistic model as auxiliary information. [Figure 14] FIG. 2 is a block diagram showing an encoding-side and decoding-side neural network with an exemplary numbering of the processing stages (or layers). [Figure 15] FIG. 1 is a block diagram showing an encoding and decoding side neural network with both prediction and residual in the 0th position of the pipeline. [Figure 16] FIG. 1 is a block diagram showing an encoding and decoding side neural network with both prediction and residual in the first position of the pipeline. [Figure 17] FIG. 2 is a block diagram showing an encoding-side and decoding-side neural network with prediction at the i-th stage of the pipeline and residual at the (i+r)-th stage. [Figure 18] FIG. 2 is a block diagram showing an encoding and decoding side neural network with prediction at stage 0 of the pipeline and residual at stage K. [Figure 19] FIG. 2 is a block diagram showing an encoding-side and decoding-side neural network with prediction in the first stage of the pipeline and residual in the Kth stage. [Figure 20] FIG. 2 is a block diagram showing encoding and decoding side neural networks with predictions in the first stage of the pipeline and residuals in the (K−1)th stage. [Figure 21] FIG. 2 is a block diagram illustrating a possible handling of residuals and predictions according to a first exemplary implementation. [Figure 22] FIG. 13 is a block diagram illustrating a possible handling of residuals and predictions according to a second exemplary implementation. [Diagram 23] FIG. 1 is a block diagram illustrating an example general architecture for handling prediction and residual in the signal domain. [Figure 24]FIG. 1 is a block diagram illustrating an example overview architecture for handling predictions and residuals in a latent space. [Diagram 25] FIG. 2 is a block diagram showing an exemplary general architecture for handling predictions and residuals in a latent space, where the latent space of the prediction is from the processing of a previous picture. [Figure 26] FIG. 1 is a block diagram illustrating an example of a video coding system configured to implement embodiments of the present invention, where the system uses a neural network to encode or decode video pictures. [Figure 27] FIG. 12 is a block diagram illustrating another example of a video coding system configured to implement embodiments of the present invention, where a video encoder and / or video decoder uses a neural network to encode or decode video pictures. [Figure 28] 1 is a block diagram illustrating a video coding device configured to implement an embodiment of the present invention. [Figure 29] 1 is a block diagram illustrating a video coding device configured to implement an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0079] The embodiments of the present application provide an AI-based video picture compression technique, and in particular, a neural network-based video compression technique.

[0080] Video coding typically refers to the processing of a sequence of pictures, which form a video or video sequence. In the field of video coding, the terms "picture", "frame" and "image" may be used synonymously. Video coding (or coding in general) includes two parts: video encoding and video decoding. Video encoding is performed at the source side and typically involves processing the original video picture (e.g., by compression) to reduce the amount of data required to represent the video picture (for more efficient storage and / or transmission). Video decoding is performed at the destination side and typically involves the reverse processing compared to the processing of the encoder to reconstruct the video picture. The embodiments referring to "coding" of a video picture (or pictures in general) should be understood to relate to "encoding" or "decoding" of the video picture or the respective video sequence. The combination of the encoding and decoding parts is also called CODEC (encoding and decoding).

[0081] In the case of lossless video coding, the original video picture can be reconstructed. In other words, the reconstructed video picture has the same quality as the original video picture (assuming that no transmission loss or other data loss occurs during storage or transmission). In the case of lossy video coding, further compression is performed, for example by quantization, to reduce the amount of data required to represent the video picture, and the video picture cannot be completely reconstructed at the decoder side. In other words, the quality of the reconstructed video picture is lower or inferior to the quality of the original video picture.

[0082] Some H.26x video coding standards (e.g., H.261, H.263, H.264, H.265, H.266) are used for "lossy hybrid video coding" (i.e., spatial and predictive prediction in the sample domain is combined with 2D transform coding to apply quantization in the transform domain). Each picture of a video sequence is typically divided into a set of non-overlapping blocks, and coding is typically performed at the block level. Specifically, at the encoder side, a video is usually processed, i.e., encoded, at the block (video block) level. For example, through spatial (intra-picture) prediction and temporal (inter-picture) prediction, a predictive block is generated, the predictive block is subtracted from a current block (a block being processed or a block to be processed) to obtain a residual block, and the residual block is transformed and quantized (compressed) in the transform domain to reduce the amount of data to be transmitted. At the decoder side, the inverse processing part to the encoder is applied to the encoded or compressed block to reconstruct the current block for representation. Additionally, the encoder replicates the decoder processing loop so that both generate the same prediction (eg, intra-prediction and inter-prediction) and / or reconstruction for processing, i.e., coding, subsequent blocks.

[0083] The present disclosure relates to processing picture data using neural networks for the purposes of encoding and decoding of picture data, which may still refer to or comprise some components known from the framework of the above mentioned standards.

[0084] The following provides a brief introduction to some of the terms used in this specification.

[0085] Picture Size: Refers to the width, or height, or width and height pair of a picture. Image width and height are usually measured in number of luma samples.

[0086] Downsampling: Downsampling is a process in which the sampling rate of a discrete input signal is reduced. For example, if the input signal is an image with sizes of h and w, the output of downsampling is h2 and w2, where at least one of the following is true: h2 <h w2 <w

[0087] In one exemplary implementation, downsampling may be implemented to retain only every mth sample and discard the remainder of the input signal (e.g., an image). However, downsampling may be performed by other means, such as convolution or other filtering.

[0088] Upsampling: Upsampling is the process in which the sampling rate of a discrete input signal is increased. For example, if the input image has sizes h and w, the output of downsampling is h2 and w2, where at least one of the following is true: h <h2 w <w2

[0089] Resampling: The downsampling and upsampling processes are both examples of resampling. Resampling is a process in which the sampling rate (sampling interval) of the input signal is changed.

[0090] Sampling ratio: When the picture sans is changed during the resampling process (upsampling or downsampling), the ratio between the output picture size and the input picture size is called the sampling ratio. The sampling ratio may be different in the horizontal and vertical dimensions.

[0091] Interpolation filtering: During the upsampling or downsampling process, filtering may be applied to improve the accuracy of the resampled signal and reduce the effects of aliasing. An interpolation filter typically involves a weighted combination of sample values ​​at sampling positions surrounding the resampling position. This includes: f(x r ,y r )=Σs(x,y)C(k) where f() is the resampled signal and (x r ,y r ) are the coordinates in the resampled image, C(k) are the interpolation filter coefficients, and s(x,y) are the samples of the input signal. The summation operation is r ,y r ) for (x,y) in the neighborhood of

[0092] Cropping: Trimming the outer edges of a digital image. Cropping can be used to make an image smaller (in number of samples) and / or to change the aspect ratio (length vs. width) of an image.

[0093] Padding: Padding refers to increasing the size of an image by generating new samples (usually at the borders of the image), for example by using predefined sample values ​​or by using sample values ​​of locations within the image.

[0094] Resizing: Resizing is the general term where an input image is resized. This may be done using one of the methods of padding or cropping, or it may be done by resampling.

[0095] Integer division: Integer division is division in which the fractional part (remainder) is truncated.

[0096] Convolution: Convolution is given by the following general formula: In the following f() may be defined as the input signal and g() may be defined as the filter.

[0097]

number

[0098] This convolution is a discrete convolution, where n and m are sampling indices and therefore integers. The range of the indices n and m may depend on the signal size (dimension) and the filter size. In general, they can theoretically range from minus infinity to infinity.

[0099] NN module: Neural network module, a component of a neural network. It can be a layer or a sub-network within a neural network. A neural network is a sequence of one or more NN modules.

[0100] Pipeline position (stage): specifies a position in a processing pipeline in a processing network that is or may comprise a neural network. A position in a network may be specified, for example, by the number of NN modules (or layers) preceding that position. Applying a transformation at the i-th position of a pipeline means applying the transformation to the output of the i-th NN module and using the result of the transformation as the input of the (i+1)-th NN module. Applying a transformation at the 0-th position of a pipeline is interpreted as applying the transformation to the input of a NN. Applying a transformation at the K-th position of a pipeline is interpreted as applying the transformation to the output of a NN.

[0101] Latent Space: An intermediate step in neural network processing, latent space representations contain the outputs of the input or hidden layers, which are usually not meant to be displayed.

[0102] Lossy NN Module: Information processed by a lossy NN module results in information loss, and the lossy module cannot restore the processed information.

[0103] Lossless NN module: Information processed by a lossless NN module does not result in information loss, and lossless processing allows the processed information to be restored.

[0104] Bottleneck: A latent space tensor that may be input to a lossless coding module. The term bottleneck refers to the fact that the channel size is usually smaller than the previous stage.

[0105] NN layer: A processing step in a neural network that performs one processing operation on input data.

[0106] Variational Autoencoder (VAE) Exemplary deep learning-based image and video compression algorithms follow the variational autoencoder (VAE) framework, e.g., Z. Cui, J. Wang, B. Bai, T. Guo, Y. Feng, “G-VAE: A Continuously Variable Rate Deep Image Compression Framework,” arXiv preprint arXiv:2003.02012, 2020.

[0107] FIG. 1 illustrates a VAE framework. The VAE framework can be considered as a nonlinear transform coding model. At the encoder side of the network, an encoder 1 maps an image x to a latent representation via a function y=f(x). The encoder includes or consists of a neural network. A quantizer 2 converts the latent representation to a discrete value y_hat=Q(y) of a desired bit length and / or precision. The quantized signal (latent space) y_hat is included in a bitstream (bitstream 1) using arithmetic coding, denoted AE, which is an abbreviation for arithmetic encoder 5.

[0108] At the decoder side of the network, the encoded latent space is decoded from the bitstream by an arithmetic decoder AD6. A decoder 4 converts the quantized latent representation output by AD6 into a decoded image x_hat=g(y_hat). The decoder 4 includes or consists of a neural network.

[0109] In Fig. 1, two sub-networks are connected to each other. The first network comprises the above-mentioned processing units 1 (encoder 1), 2 (quantizer), 4 (decoder), 5 (AE), and 6 (AD). At least units 1, 2, and 4 are called an autoencoder / decoder, or simply an encoder / decoder network.

[0110] The second sub-network comprises at least units 3 and 7 and is called hyperencoder / decoder or context modeler. In particular, the second sub-network models the probability model (context) for AE5 and AD6. The entropy model, or in this case the hyperencoder 3, estimates a distribution z of the quantized signal y_hat that approaches the minimum rate achievable with lossless entropy source coding. The estimated distribution is quantized by the quantizer 8 to obtain a quantized probability model z_hat that represents side information that can be communicated to the decoder in a bitstream. To do so, the arithmetic encoder AE9 may encode the probability model into the bitstream 2. The bitstream 2 may be communicated to the decoder side together with the bitstream 1 and also provided to the encoder. In particular, to be provided to AE5 and AD6, the quantized probability model z_hat is arithmetically decoded by the AD10 and then decoded with the hyperdecoder 7 and inserted into AD6 and AE5.

[0111] FIG. 1 shows the encoder and the decoder in a single diagram. On the other hand, the encoder and the decoder may operate separately, so FIG. 2 and FIG. 3 show the encoder and the decoder separately. In other words, the encoder may generate bitstream 1 and bitstream 2. The decoder may receive such a bitstream from a storage or via a channel, etc., and may decode it without further communication with the encoder. The above description of the encoder and decoder elements also applies to FIG. 2 and FIG. 3.

[0112] Many deep learning-based image and video compression systems reduce the dimensionality of a signal before converting it into binary digits (bits).

[0113] In the VAE framework, for example, an encoder, which is a nonlinear transform, maps an input image x to y, where y has a smaller width and height than x. Because y has a smaller width and height and therefore a smaller size, the dimensionality of the signal is reduced, and therefore it becomes easier to compress the signal y.

[0114] The general principle of compression is illustrated in FIG. 4. The input signal x corresponds to the input data, which is the input of the encoder. The transformed signal y corresponds to the latent space, which has a smaller dimensionality than the input signal, and is therefore also called the bottleneck. Typically, the dimensionality of the channel is minimal at this processing position in the encoder-decoder pipeline. Each row of circles in FIG. 4 represents a layer in the processing chain of the encoder or decoder. The number of circles in each layer indicates the size or dimensionality of that layer. The latent space, which is the output of the encoder and the input of the decoder, represents the compressed data y. At the decoder side, the latent space signal y (encoded image) is processed by the decoder neural network, leading to an expansion of the channel dimension until we obtain a reconstructed data x_hat, which may have the same dimensionality as the input data x, but which is different from the input data x, especially if lossy processing is applied. The dimension of the channel processed by the decoder layer is typically higher than the y dimension of the bottleneck data. In other words, the encoding operation usually corresponds to a reduction in the size of the input signal, and the decoding operation corresponds to a reconstruction of the original size of the image, and thus corresponds to the name bottleneck.

[0115] As mentioned above, the reduction in signal size can be achieved by downsampling or rescaling. The reduction in signal size usually occurs stepwise along a chain of processing layers, rather than all at once. For example, if an input image x has dimensions h and w (denoting height and width) and a latent space y has dimensions h / 16 and w / 16, the reduction in size may occur in four layers during encoding, where each layer reduces the size of the signal by a factor of two in each dimension.

[0116] Known deep learning based video / image compression methods typically employ multiple downsampling layers. An exemplary VAE is shown in FIG. 5 and utilizes six downsampling layers marked 51 through 56. Layers that include downsampling are indicated with a downward arrow in the layer description. The layer description "Conv N×5×5 / 2↓" means that the layer is a convolutional layer ("Conv") with N channels and the convolution kernel is of size 5×5 samples. The "2↓" means that downsampling by a factor of 2 occurs in this layer. Downsampling by a factor of 2 results in one of the dimensions of the input signal being reduced by half at the output. In FIG. 5, the 2↓ indicates that both the width and height of the input image are reduced by a factor of two. There are six downsampling layers that halve the size in one dimension, so that if the width and height of the input image x (514) are given by w and h, then the output signal z_hat (513) has a width and height equal to w / 64 and h / 64, respectively.

[0117] In the first sub-network, several convolutions are followed by a generalized division normalization (GDN) at the encoder side and an inverse GDN (IGDN) at the decoder side. In the second sub-network, the activation function applied is a rectified linear function (ReLU). It should be noted that the present disclosure is not limited to such an implementation and in general, other activation functions may be used instead of GDN or ReLU.

[0118] The network architecture in Fig. 5 includes a hyper-prior model. a , g s ) shows the image autoencoder architecture, and on the right (h a , h s ) corresponds to an autoencoder that implements a hyper-prior as explained above with reference to Figures 1 to 3. The model is based on the analysis and synthesis transformation g a and g sAs in Fig. 1 to Fig. 3 described above, Q represents quantization, and AE and AD represent arithmetic encoder and decoder, respectively. The encoder applies g a resulting in a latent representation y with spatially varying standard deviation.

[0119] The result is h a , which summarizes the distribution of standard deviations in z. Then z is quantized, compressed, and transmitted as side information. The encoder calculates the spatial distribution of standard deviations, which is used to obtain probability values ​​(or frequency values) for arithmetic coding (AE).

[0120]

number

[0121] The decoder uses the quantized vector z_hat 513 to estimate y_hat, and uses that estimate to compress and transmit the quantized latent representation y_hat 515. The decoder first recovers z_hat from the compressed signal. The decoder then uses h to obtain y_hat. s provides a correct probability estimate for successfully recovering y_hat. The decoder then converts y_hat to g to obtain the reconstructed image. s Supply to.

[0122] The decoder comprises upsampling layers 57-59 and 510-512. A further layer 420, implemented as a convolutional layer but which does not provide upsampling to the received input, is provided between upsampling layers 411 and 410 in the processing order of the inputs. A corresponding convolutional layer "conv Mx3x3 / 1" is also shown for the decoder. Such layers are provided in the NN to perform operations on the input that do not change the size of the input but do change certain characteristics. However, such layers need not be provided.

[0123] When viewed in the processing order of the bitstream 2 through the decoder, the upsampling layers proceed in reverse order, i.e., from upsampling layer 512 to upsampling layer 57. Each upsampling layer is shown here as providing upsampling with an upsampling ratio of 2, which is indicated by ↑. Of course, not all upsampling layers have the same sampling ratio, and other upsampling ratios such as 3, 4, 8, etc. may be used. Layers 57-512 are implemented as convolutional layers (conv). Specifically, since the upsampling layers are intended to provide an inverse operation on the input to that of the encoder, the upsampling layers may apply a deconvolution operation to the received input such that the size of the received input is increased by a factor corresponding to the upsampling ratio. However, the present disclosure is not limited to deconvolution, and upsampling may be performed in any other manner, such as bilinear interpolation between two neighboring samples, or nearest neighbor sample copying.

[0124] FIG. 6 shows a general block diagram of a deep learning based picture (sil or video) codec with auxiliary information applied after the first layer. The deep learning architecture can be any architecture including conditional autoencoders and hyper priors. Conditional VAE means that the network is conditioned by collecting additional information, referred to herein as auxiliary information. Such auxiliary information is processed by one or more layers of the NN along with feature data. For example, the input 602 of the picture encoder is an image, which may be provided in multiple channels. The exemplary representation of FIG. 6 shows three color channels (e.g., YUV color space, etc.) with luminance and two subsampled chrominances, as is one of the usual approaches in picture coding. In general, there may be more channels (e.g., additional colors such as white, or depth or other features, such as motion) or there may be fewer channels (e.g., only one grayscale channel, black and white or color palette, etc.). The input picture channels 602 are placed into an input tensor x and input to the encoder neural network 600A. The encoder neural network 600A may include multiple layers or layer stages, which may include downsampling layers that may be implemented, for example, by convolutions with or without strides, nonlinear activations such as sigmoid, ReLU, or GDN, several masked convolutions that do not change the dimensionality of the tensors, etc. It should be noted that the layers may also include entropy encoding or other processing layers.

[0125] The input signal X may be processed by one or more layers including convolution ("conv") and downsampling ("↓") along with some nonlinear operations (NonLinAct) and masked convolution (MaskConv). The encoder may comprise two or more layers and write the compressed information in the latent space to one or more bitstreams ("----") 620 and / or 630. For example, one of the bitstreams (e.g., bitstream 620) may correspond to bitstream 1 and the other bitstream (e.g., bitstream 630) may correspond to bitstream 2 shown in Figures 1 to 3. In Figure 6, the stages or layers are shown as downsizing (narrower in the processing direction). However, this is for illustrative purposes and in general the network may also include layers that do not change the tensor size and / or layers that increase the tensor size. Following layer 1, collection 610 or auxiliary information may be performed. The collector 610 may collect the auxiliary information and combine it with the output data of layer 1. The combined information is then input to Layer 2 for further processing.

[0126] Correspondingly, the decoder portion 600B of the network analyzes one or more bitstreams 620 and / or 630 and reconstructs the signal using one or more layers of (de)convolution, upsampling ("↑"), non-linear operations, and masked convolution. This exemplary decoder network 600B is symmetrical to the encoder network 600A (although this is not generally necessary). The collector 650 collects auxiliary information and applies a combination of the collected auxiliary information and the output of the penultimate layer to the last decoding layer (symmetrical to layer 1 of the encoder). After processing by the last layer, an output tensor is obtained, in this example the reconstructed picture 692.

[0127] Auxiliary information in the encoding / decoding pipeline In a picture codec, auxiliary information may be used to facilitate the encoder and the decoder. These auxiliary information may be applied at a specific position or positions in the coding (encoding and / or decoding) pipeline together with the encoded or decoded tensors. In the example of FIG. 6, the auxiliary information is used to assist in encoding and decoding the signal processed after the first layer. Since the network architecture is primarily predefined and fixed for a specific application (picture encoding and decoding), the application of these auxiliary information is typically tied to a specific fixed position in the processing pipeline, as shown by the collection blocks 610 and 650 in FIG. 6. The parameters (e.g., weights) applied in the neural network are usually pre-trained (e.g., provided before encoding and / or decoding). Thus, the architecture of the neural network is pre-determined, since the weights and positions to apply the auxiliary information are pre-determined. This may limit the flexibility of the network and its adaptation to various parameters and characteristics of the encoded or decoded content.

[0128] A possible example for auxiliary information is inter-prediction frames (prediction) applied in the domain of the input signal x. Inter-prediction exploits the temporal redundancy between adjacent frames in a video and the current frame, since the content of the neighboring frames usually does not change completely. Instead, the content usually moves only a little between frames. Therefore, inter-prediction requires two pieces of information: one or more neighboring frames as reference pictures, and motion information (e.g., motion vectors). Inter-prediction takes these two as inputs and applies motion compensation to generate a prediction for the current frame.

[0129] Another form of prediction is intra prediction within one frame (picture). Intra prediction exploits spatial redundancy within a frame. Given an area within a frame, sample values ​​in the area are usually correlated. Therefore, intra prediction uses reconstructed neighboring samples of a frame to predict the value of a sample at a current position within the same frame.

[0130] In Fig. 7 we provide a visual example of prediction as supplementary information. The top left image is a predicted frame 710 from a common test sequence BasketballDrill. The prediction is made in the same domain as the input signal x, i.e., in this example, the NN input picture domain. The original width and height of the luma frame are denoted as W and H, respectively. The smaller frame 720 located in the bottom left corner is the same predicted frame, but after one convolutional layer conv(1,↓2) with one channel and a downsampling of 2. Thus, both the width and height are half the original resolution, i.e., W / 2 and H / 2. On the right side, an example of prediction in the latent space domain after N convolutional layers, with a downsampling of 2 for each convolutional layer, is shown 730. Thus, here the width and height of the prediction are 2 N The figure shows a grid of downsampled frames, where the grid size is determined by the number K of channels 740, where there are a total of K=13*11=143 channels. In other words, the mosaic picture 730 shows the auxiliary information (here prediction) in the latent space domain after n convolutional layers with K channels N*conv(K,↓2).

[0131] Figure 8 shows a combination approach that uses both traditional block-based video codec (HEVC) and deep learning to capture various frame characteristics according to Lu, Z. Ma, L. Xu, D. Wang, "Tests on Decomposition, Compression, Synthesis (DCS)-based Technology," JVET-U0096, Joint Video Experts Team (JVET) 21st Meeting by teleconference, January 6-15, 2021. Auxiliary information such as prediction is used at multiple positions in the coding pipeline. The coding pipeline in Figure 8 can be divided into three stages: decomposition, compression, and synthesis. In the figure, ↓ represents downsampling, ↑ represents upsampling, and E and D represent HEVC video encoder and decoder, respectively. STF represents spatial texture frame, which is an intra-predicted frame within a group of pictures (GOP). TMF represents temporal texture frame, which is an inter-predicted frame within a GOP. The prediction and residual coding of the HEVC encoder and decoder are located at static (fixed) locations in the HEVC video codec pipeline.

[0132] In Fig. 8, the input GOP is first classified into two categories, STF and TMF, in the decomposition stage. The motion information in TMF can be captured at a coarse granularity and is therefore downsampled before being encoded.

[0133]

number

[0134] (i.e., the low-resolution input for the TMF). In comparison, the STF is

[0135]

number

[0136] (i.e., the original high-resolution input with respect to the STF). Both the STF and the downsampled TMF are encoded using a conventional HEVC codec in a second compression stage, where the HEVC encoder and decoder are represented by the boxes labeled E and D, respectively. When encoding the TMF, its reference pictures are

[0137]

number

[0138] The downsampled version of the reconstructed signal of the STF, denoted as (i.e., reconstructed downsampled low-resolution signal with respect to the STF). The encoded information of the STF and TMF are written into the bitstream. They are then parsed and used by the HEVC decoder to reconstruct the STF and TMF. Again, when decoding the TMF, its reference frame is the STF.

[0139]

number

[0140] The outputs of the compression stage are, respectively,

[0141]

number

[0142] and

[0143]

number

[0144] , and the reconstructed signals of STF and TMF are denoted as

[0145] In the synthesis stage, the reconstructed STF and TMF are used to generate high-fidelity video at its native resolution. First, we extract the temporally smooth and spatially fine-grained motion representation of the current TMF by aggregating information across neighboring TMFs.

[0146]

number

[0147] A motion compensation network (denoted as motion compensation) is utilized to generate the temporal motion features generated by the motion compensation network. The temporal motion features generated by the motion compensation network are then converted into the decoded and upsampled TMF

[0148]

number

[0149] The decoded STF, along with its resampled version, is fed to a non-local texture transfer network to learn and transfer cross-resolution information for high-fidelity frame reconstruction with both spatial details and temporal smoothness. The hybrid coding pipeline uses auxiliary information at various positions in the codec, such as the temporal motion features generated by the motion compensation network, the prediction and residual information used in the HEVC encoder and decoder (not shown in FIG. 8, but used in blocks E and D). However, the auxiliary information is applied at a fixed position in the coding pipeline.

[0150] Such implementations may not be able to effectively adapt to the characteristics of images and video sequences, and therefore produce suboptimal coding results.

[0151] A residual signal (also called a prediction error signal or residual signal) obtained by subtracting a prediction signal from an original signal may also be used as side information. In some implementations of the present disclosure, the prediction and the residual signal may occur not only in the same domain as the input signal x, but also in a domain one or more layers later in the processing order of the encoder part 600A, or one or more layers earlier in the processing order of the decoder part 600B. Conventionally, the position of utilizing these side information together with the encoded and / or decoded tensors in the encoder and / or decoder is fixed.

[0152] However, depending on the content, and in particular, depending on the characteristics of the input picture, the auxiliary information may not work well at a pre-designed fixed position in the coding pipeline. Therefore, utilizing the auxiliary information at a fixed position in the coding pipeline may lead to suboptimal coding results. Therefore, an adaptive solution that conditionally applies these auxiliary information at different positions based on the input characteristics may be desirable.

[0153] Selection of auxiliary information input position based on collection conditions According to one embodiment, a method is provided for processing picture feature data from a bitstream using a neural network. The neural network comprises multiple (i.e., two or more) neural network layers, also referred to as layers hereinafter. Such a neural network 900B is illustrated in FIG. 9. The term picture feature data refers to any latent space data obtained by picture processing on the encoder side. In other words, the picture feature data may be the output of the encoding process.

[0154] The method includes obtaining picture feature data based on the bitstream 940. For example, the picture feature data may be directly parsed or parsed and decoded from the bitstream. Furthermore, the picture feature data may be obtained by some processing based on the decoded data. The method further includes processing the picture feature data using the neural network 900B. For each of one or more preconfigured positions in the neural network, the processing includes: a) determining whether to collect (960_1, 960_2, 960_3) auxiliary data 950 for processing by one of a plurality of neural network layers at a preconfigured position (following the preconfigured position in the feature data processing direction) based on collection conditions 910_1, 910_2, 910_3; b) if it is determined that auxiliary data 950 should be collected, processing by the layer at said preconfigured location is based on the auxiliary data 950; Includes.

[0155] A pre-configured location is a location in the neural network where the collection of (the same) auxiliary information is possible. There may be one such location, or there may be two or more (or all) such locations. In FIG. 9, there are three pre-configured locations, namely locations of collection 960_1, 960_2, 960_3. As mentioned above, a location in the network may be specified, for example, by the number of NN modules (here layers) that precede it in the order of processing of picture feature data. The pre-configured locations may be fixed as part of the neural network architecture. Nevertheless, flexibility may be improved by allowing or forbidding auxiliary data to be input to one or more layers. However, the present disclosure is not limited to fixed locations. Alternatively, the pre-configured locations may also be configurable by side information directly or indirectly included in the bitstream. Here, direct configuration may signal the actual location, while indirect configuration may be based on the same rules known in the encoder and decoder, for example, based on the values ​​of encoding parameters (e.g., from the bitstream).

[0156] If it is determined that auxiliary data is not collected for processing by a particular layer, the processing by the particular layer is not based on the auxiliary data. In the exemplary architecture in FIG. 6, the position where the auxiliary data is input in the example of FIG. 6 is fixed and located after the first layer. In contrast, in FIG. 9, the auxiliary information can be inserted at multiple potential positions in the neural network. This is shown by the collection units 960_1, 960_2, and 960_3 controlled by the conditions 910_1, 910_2, and 910_3 (labeled as question mark circles) throughout the neural network.

[0157] A similar process is performed on the encoder side. A method on the encoder side is provided for processing a picture 902 using a neural network 900A with multiple neural network layers to generate a bitstream 940. The method includes a step of processing a picture 902 using the neural network 900A. The process includes a step of determining, for each of one or more preconfigured positions in the neural network, whether to collect (e.g., by respective collectors 920_1, 920_2, and 920_3) auxiliary data 950 for processing by a layer at said preconfigured position based on collection conditions 910_1, 910_2, and 910_3, and if it is determined that auxiliary data should be collected, the processing by the layer at said preconfigured position is based on the auxiliary data 950. The method further includes a step of inserting data obtained by processing the picture by the neural network into the bitstream 940. As seen in FIG. 9, there may be an additional bitstream (or bitstream portion) 945 that may be provided when a hyper-a priori sub-network for providing lossless encoding of probability model parameters is applied.

[0158] In this way, the location of application of the auxiliary information may be dynamically changed depending on the content and / or characteristics of the images or frames in the video sequence. Thus, the configurable neural network has a dynamic architecture, since the application of the auxiliary information and the corresponding collection parameters can occur at different locations.

[0159] The term collection condition in this specification refers to a requirement that, for example, when met, determines whether the collector collects auxiliary information 950. The requirement may be different for each collection unit 960_1, 960_2, and 960_3 (and each encoder-side collection unit 920_1, 920_2, and 920_3). Such a decision has the same effect as the selection of the location where the auxiliary information is input. Depending on the design of the requirement, the auxiliary information may be selected for only one of the preconfigured locations. This may be achieved, for example, by providing mutually exclusive requirements (collection conditions) for each collector. In other words, as a result of applying the collection condition in the decision step, the auxiliary data will be collected for a single location among one or more preconfigured locations.

[0160] However, the above-mentioned examples do not limit the present disclosure. As seen in FIG. 9, there is one auxiliary information 950 or one type of auxiliary information 950 that can be collected at a preconfigured location. In some exemplary implementations, the application of the collection condition in the determining step results in the same auxiliary data being collected for two or more of the preconfigured locations. The above-described techniques for collecting auxiliary information can be applied to two or more types of auxiliary information. Then, for each of the auxiliary information types, the collection location can be determined from a set of preconfigured locations. It should be noted that the set of preconfigured locations can be different for different types of auxiliary information. However, the sets can overlap or be the same for some types of auxiliary information. For example, the auxiliary information can be a prediction signal, a prediction error signal, a probability model, etc. Some of them can be applicable to the same or overlapping stages at corresponding locations, for example, a prediction signal and a prediction error signal can be used together. Some of the auxiliary information types can be applicable at different stages, for example, a probability model for entropy encoding a latent space signal.

[0161] When we refer to signals in this specification, we primarily mean discrete signals, such as input pictures to an encoding network or input picture feature data to a decoding network, or data representing the latent space at any stage, or auxiliary information.

[0162] As mentioned above, different layers of a neural network may process feature tensors with different sizes. In picture processing, feature tensors usually have three dimensions, namely width, height, and number of channels, as already discussed with reference to FIG. 7. A feature picture with width and height may be considered to correspond to a channel. Thus, some resizing or rescaling may be performed to support the (selectable) collection of the same auxiliary information in multiple stages of the pipeline. For example, in an exemplary implementation, two or more of the preconfigured locations are preconfigured. The auxiliary data is scalable in size to match the dimensions of the input channels processed by the layers at two or more of the preconfigured locations. Thus, in this exemplary implementation, as a result of applying the collection condition in the decision step, auxiliary information (data) may be (i) collected, or (ii) collected and scaled for a single location among the preconfigured locations. This is equivalent to selecting a single location among the preconfigured locations where auxiliary information is collected, and adapting the auxiliary information (if necessary) to the resolution processed by the layer at which the auxiliary information is to be input. However, the present disclosure is not limited to always applying auxiliary information at least in one position.It is also conceivable to design the collection condition such that, as a result of applying the collection condition, auxiliary information is not collected in any of the pre-configured information for some pictures.In other words, (iii) the third possibility of not collecting auxiliary information can be implemented.

[0163] The embodiments described herein focus on the selectable collection of auxiliary information. However, apart from such selectable collection, the neural network architecture may provide one or more collection stages in which certain types of auxiliary conditions are always collected (without relying on preconditions such as satisfying the collection conditions), as in the case of conventional conditional VAE frameworks.

[0164] This embodiment allows adaptively changing the location where the auxiliary information is applied according to the collection conditions.

[0165] It should be noted that the present disclosure is not limited to a particular NN framework such as VAE. Furthermore, the present disclosure is not limited to image or video compression, but may also be applied to any picture processing system, including object detection, object recognition, or classification systems, in general, the human visual system, where pictures are reconstructed for human visual purposes, or machine vision systems, where pictures are not necessarily reconstructed, but picture features suitable for deriving a desired output, such as segmentation maps, depth maps, object classification, object detection, etc.

[0166] 10 illustrates an example embodiment in which the collection conditions are based on picture characteristics or picture feature data characteristics obtained from the bitstream. The neural network of FIG. 10 processes an input picture 1002 and outputs a processed picture 1092.

[0167] For example, the picture characteristics or picture feature data characteristics include resolution. The collection conditions include a comparison of the resolution with a preconfigured resolution threshold. For example, if the picture resolution exceeds the preconfigured threshold, the auxiliary data should be collected for a first one of the preconfigured locations. On the other hand, if the picture resolution does not exceed the preconfigured threshold, the auxiliary data should be collected for a second location different from the first one of the preconfigured locations. The threshold may be empirically defined as part of the design of the neural network architecture, or may be trained and / or configurable by including corresponding parameters in the bitstream.

[0168] In addition to or alternatively to the resolution-based condition, in an exemplary embodiment, the picture is a video picture and the picture characteristics include a picture type. The collection conditions include determining whether the picture type is a temporally predicted picture type or a spatially predicted picture type. A spatially predicted picture is predicted using only samples in its own picture. In this case, samples from other pictures are not used. A temporally predicted picture must contain some areas that are temporally predicted based on samples on other pictures. However, a temporally predicted picture may also contain some areas that use spatial prediction, i.e., areas that use samples in its own picture.

[0169] Both the picture type and the picture resolution or other picture characteristics are known to both the encoder and the decoder so that both sides can operate in the same way without needing any additional side information.

[0170] In Figure 10, the first processing layer is a downsampling layer 1010 in the encoder 1010A and an upsampling layer 1070 in the decoder 1010B. The downsampling 1010 and upsampling 1070 in this example are by a factor of 2 in both the width and height of the frame. Auxiliary information such as prediction may potentially be applied at two or more locations before or after the first layer 1010 in the encoder and the corresponding layer 1070 in the decoder.

[0171] On the encoder side, in this example, the collection unit 1030 is a weighted sum of two input signals, one of the full size of the original frame to be encoded and the other of a quarter of the original frame size (full size) corresponding to a downsampled version of the original frame.

[0172] For example, a weight pair of (1,0) indicates encoding using auxiliary information at full size (indicated by weight factor 1) rather than quarter size (indicated by weight factor 0), while a weight pair of (0,1) indicates encoding using auxiliary information at quarter size rather than full size. In this way, the location of applying auxiliary information can be made configurable within the neural network. The conditions that control where the auxiliary information is used can be, for example, the frame size (resolution) of the original frame, and / or the frame type.

[0173] For example, the condition may be whether the frame size is equal to or greater than 4K resolution and whether the frame type is interframe. In an exemplary implementation, if the condition is true, a (0, 1) weight pair is used, so that these input frames are encoded at a quarter size using the auxiliary information. Other frames are encoded at their original resolution using the auxiliary information. The position of applying the auxiliary information may be dynamically changed at the frame level based on the frame type and / or frame size.

[0174] In other words, in FIG. 10, an example is shown in which the collection unit 1030 (and accordingly the decoder 1060) is simply a weighted sum. The weights can have values ​​(1,0) or (0,1) as explained above. The weights (1,0) mean that a full-size picture is encoded, and the weights (0,1) mean that coding is performed on a quarter-size signal corresponding to a signal downsampled by 2. The collection condition 1050 in this example is whether the video has a resolution of 4K or higher, and at the same time, whether the frame of interest is a temporally predicted frame. However, as mentioned above, the present disclosure is not limited to such combined collection conditions. Rather, the thresholded resolution may only be whether the resolution exceeds a certain value or belongs to a certain set of resolutions. Alternatively, only the frame type may be the evaluation condition, without considering the resolution. Further examples are possible, including other conditions used alone or in combination.

[0175] For example, coding parameters such as prediction mode, motion information (e.g., for deriving the strength or type of motion) may be applied in the collection condition. The configurability of the neural network in the above-described exemplary embodiment is achieved by switching on / off auxiliary information at a specific location based on a content adaptation condition. The content may include direct characteristics of the image, such as frame size and frame type, as shown in the example of FIG. 10. However, the characteristics may also be indirect characteristics of the content, such as a histogram of sample values ​​in a given area. The granularity of the content may also be variable. For example, the conditions determining the location of the auxiliary information may be updated at the frame level, block level, or the level of a group of pictures or any amount of pictures. In other words, in some embodiments, the neural network is trained to perform processing of video pictures. The determination of whether to collect auxiliary data for processing by a layer at the preconfigured location is performed every predetermined number of video pictures, where the predetermined number of video pictures is one or more.

[0176] The condition may also have different forms. As described below, the condition may be determined from a parsed flag at the decoder. At the encoder, the flag value may be determined based on some RDO calculations or based on other cost functions. It is also possible that the condition is not explicitly signaled and is determined by other information of the image / frame in the video sequence.

[0177] One advantage of the flexible location of applying the auxiliary information is that pictures or frames can be adaptively encoded and decoded, thus resulting in better performance, which may include a lower rate of the resulting bitstream 1055 at the same quality, or vice versa (higher quality at the same rate).

[0178] According to one embodiment, the method comprises, at the decoder side, a step of obtaining from the bitstream an indication specifying whether to collect auxiliary data for one or more preconfigured positions, the collection condition for each of the one or more preconfigured positions being: - if the instructions specify that auxiliary data should be collected for said preconfigured location, the determination is positive; If the instructions specify that for said preconfigured location, no auxiliary data should be collected, the decision is negative.

[0179] The indication may be a flag supported by the bitstream syntax that directly indicates whether to collect specific auxiliary information at a particular location. Alternatively, the indication may be included directly in the bitstream according to the syntax and specify a single location among preconfigured locations where auxiliary information should be collected. The indicator may also be designed to take a value indicating that auxiliary information should not be collected at any of the preconfigured locations. In this embodiment, the collection condition is based on evaluating the indicator.

[0180] The above-mentioned embodiments have been described with reference to auxiliary information. In general, auxiliary information (or auxiliary data) provides some (supplementary) information about the picture feature data that is processed by the neural network to generate an output. In a particular example also shown in Fig. 11, the auxiliary data includes prediction data that is a prediction of a picture (e.g., in an input picture domain for the encoder) or a prediction of the picture feature data after processing by one or more of the layers of the neural network (e.g., outside the input picture domain for the encoder).

[0181] As shown in FIG. 11, in some exemplary implementations, the auxiliary data is a combined pair of predicted data and auxiliary data combined with the predicted data. The term "combined" here refers to the fact that the prediction and supplementary data are obtained from the same pipeline position (stage). The auxiliary data may be, for example, prediction error data (data representing a prediction error), which may be a prediction residual (the difference between a predicted picture and a prediction of the picture) in certain cases. The prediction signal and the residual signal may be obtained after zero layers, one layer, or several layers of the neural network, as shown in FIG. 11. Applying the auxiliary information at the i-th position means using the auxiliary information for a transformation at the i-th position in the pipeline. The transformation here may be a general transformation performed by a layer, or an actual frequency domain transformation performed as a processing step (layer) of the network, for example, an orthogonal transformation such as a Fourier transform (e.g., a fast Fourier transform (FFT) or a discrete cosine transform (DCT) or their integer versions, etc.). As mentioned above, in general, the auxiliary information is a tensor that may have the same size as the encoder-side input signal or the latent space representation in the coding pipeline depending on the collection position.

[0182] In general, the number of positions where the prediction signal and the residual signal can be applied is N+1. This corresponds to position 0 at the decoder side, which represents the input signal (input to the NN), and to the output signal and N further positions 1 to N at the encoder. Note that pre-configured positions are not necessarily provided after each layer. Figure 11 shows an example where the auxiliary information is the prediction signal and the residual signal. They can be collected at several positions, which are shown by providing the collection unit 1150 multiple times in the encoder part 1100A and the decoder part 1100B of the neural network.

[0183] In the example of FIG. 11, a single one of these locations applies the prediction signal and the residual signal during encoding and decoding. In other words, the locations that apply the side information do not overlap, but rather change within the coding pipeline. The collection stage 1150 collects the prediction signal (pred i ) 1110 and subtracting the prediction from the picture feature data of the ith position (calculated in the decoder process direction) to obtain the residual (residual i ) 1120. The residuals are then encoded into a bitstream 1140. Thus, the residuals may be collected in the decoder 1100B by adding the residuals 1160 to a prediction obtained by a process that may include, for example, a motion compensation process or an intra-prediction process.

[0184] On the encoder side, the method may further include selecting for one or more preconfigured positions whether to collect auxiliary data based on optimizing a cost function including at least one of rate, distortion, accuracy, latency, or complexity. In other words, the positions may be determined by RDO or a similar approach. For example, on the encoder side, the decision regarding the stage in the pipeline where the auxiliary information is applied may be made by multi-pass encoding. In one example, applying the auxiliary information is only allowed at one position as described above. In this case, the encoder tries to apply the auxiliary information at the i-th position of the pipeline for preconfigured positions i=0,1,2,..,N and obtains the distortion and rate for each variant. The optimal variant (position) for the defined rate / distortion ratio is then selected.

[0185] In another example, if it is possible to apply auxiliary information at any one or more positions in the pipeline, then for each Boolean vector of length (N+1), the encoder attempts to apply the auxiliary information at a position corresponding to the position in the Boolean vector, so that 2 N+1variants are tried. If some a priori knowledge is taken into account, the search space can be reduced. For each position i, a Boolean vector indicates whether auxiliary information is collected at that position.

[0186] However, the present disclosure is not limited to predictions and prediction residuals as auxiliary information. Rather, auxiliary data is applied as a kind of guidance data that allows directionality in the decoding process. In other words, auxiliary data conditions the process by some supplementary information. In general, auxiliary data can be labels or other data to help the generative network.

[0187] It should be noted that in this disclosure, the meaning of the prediction and residual signals is somewhat generalized, i.e., unlike conventional codecs such as HEVC, where the prediction and residual signals are generated in the same region of the input signal x, the prediction and residual signals herein may refer to the signals after one or more layers in a neural network, as shown in Figure 11. In the embodiment of Figure 11, the prediction and residual signals are combined, i.e., the co-located prediction signal pred i and the residual signal resi i is enabled or disabled as soon as the collection conditions are met.

[0188] However, there are embodiments in which the prediction data and the supplementary data have the dimensions of data processed by layers at different locations in the network. In other words, they are collected from different stages. The (selected) collection location of the prediction and the (selected) collection location of the residuals may be different. Some embodiments of such approaches are described below in the section "Combining Data from Different Processing Stages of a Neural Network" and these embodiments may be combined with changing the collection location as described herein.

[0189] Figure 12 shows some experimental data. In the experiments, the network configuration can be selected based on the analysis of the input signal. This analysis is performed in the frequency domain. An example is shown in Figure 12, where the transform FFT2 (see https: / / www.mathworks.com / help / matlab / ref / fft2.html), corresponding to the two-dimensional Fast Fourier Transform, is used as a transform tool to perform the spectral analysis. For a given image, it is possible to predict (or measure) the spectral changes introduced by the FFT2 transform before and after a network layer with 2x downsampling. If the main signal energy is preserved, it is advantageous to apply an auxiliary signal to a lower resolution representation, since the error between the original and the reconstructed picture is probably negligible. Figure 12 shows the following stages of analysis: Stage (a) shows the input (original) picture. The experiments here were performed on two different pictures (see the top and bottom of (a)). Stage (b) shows the FFT2 result when FFT2 is applied to the original image. Step (c) shows the FFT2 result when FFT2 is applied to the original picture after downsampling by 2. Step (d) shows the reconstructed image based on the subsampled signal. Step (e) shows FFT2 applied to the reconstructed image. Step (f) shows the difference between the reconstructed image and the original image. Step (g) shows FFT2 of the difference image. As can be seen, the top image in FIG. 12 has fewer high frequency components. The reconstructed frame is obtained by first applying the auxiliary signal at the downsampled resolution and then upsampling. Such resampling has little effect on the signal content. In contrast, the loss of high frequency energy is evident in both picture details and frequency content for the bottom image, where more high frequency components are observed. This observation probably also applies to other transformations. Overall, the network is adaptively configured such that the auxiliary signal is applied to the convolutional layer at the appropriate resolution and the spectral energy of the input signal is well preserved. It is conceivable that the collection condition is based on FFT2, or another transformation that may provide some spectral information.For example, some statistical parameter of the spectral distribution may be thresholded to define at which locations auxiliary information should be collected.

[0190] It should be noted that the present disclosure is not limited to providing the prediction signal and the residual signal as side information. Another exemplary possibility is the application of the side information in lossless compression.

[0191] Thus, potential locations where auxiliary information may be applied include locations where the signal may be processed losslessly. As shown in Figure 13, the neural network may include a subnetwork where the signal y_hat may be recovered losslessly from the decoder, as described above with reference to Figures 1 to 3. According to one embodiment, after obtaining the signal y_hat there is a collection unit 1310, where the auxiliary information may be applied later in the neural network.

[0192] In other words, in an embodiment, the neural network includes a sub-network for lossless decoding with at least one layer, and the auxiliary data is input to said sub-network for lossless decoding. Thus, the collection condition 1320 may control whether a probability model for entropy coding (encoding and decoding as described above for the VAE framework) is provided as auxiliary information from the lossless coding sub-network. If no auxiliary information is provided, the probability model may be determined, for example, by a default probability distribution or by bitstream signaling, etc. The generation condition may be based on an optimization based on a cost function, etc., taking into account the complexity and latency caused by the sub-network on the one hand and the contribution of the sub-network to reducing the resulting bitstream size. Alternatively, the collection condition may be based on statistics of previously coded data, for example their variance. If approximately stationary or slowly changing, the application of auxiliary information from the sub-network may not be necessary.

[0193] In general, the neural network may be trained to perform at least one of still picture decoding, video picture decoding, still picture filtering, video picture filtering, and machine vision processing including object detection, object recognition, or object classification.

[0194] The encoding and / or decoding described above may be performed on each of a plurality of auxiliary data, including first auxiliary data and second auxiliary data, where the first auxiliary data is associated with a first set of one or more preconfigured positions, and the second auxiliary data is associated with a second set of one or more preconfigured positions. In other words, the present disclosure may be applied to different types of auxiliary data in the same network. For example, the first auxiliary data is a prediction and residual signal, and the second auxiliary data is a probability model for lossless coding. In some embodiments, the first set of one or more preconfigured positions and the second set of one or more preconfigured positions share at least one preconfigured position. However, this is merely an example, and in some cases, the sets do not necessarily share any preconfigured positions. This may be the case, for example, in the example of prediction / residual as the first data and probability model as the second data.

[0195] As shown in Figure 9, a configurable conditional autoencoder neural network is provided as an exemplary embodiment. Within this architecture, the position of applying auxiliary information such as predictions and / or residuals can be adaptively determined based on conditions. Thus, the architecture of the neural network can be dynamically modified depending on the characteristics of the images or frames in the video sequence.

[0196] In general, embodiments of the present disclosure provide an approach to dynamically configurable neural network-based picture codecs, where the position in the coding pipeline to apply auxiliary information depends on conditions. The auxiliary information may be a combined pair of a prediction signal and a residual signal. The paired prediction and residual signals may be located either in the same domain as the input signal x, or in the domain after processing by one or more layers of the neural network. The position to apply auxiliary information may be variable, which may be achieved by using the auxiliary information exclusively at certain positions among the potential positions in the neural network based on a collection condition. Such a collection condition controls the position to apply auxiliary information in a content-adaptive manner, i.e., the condition may be determined based on one or more characteristics of the input signal (picture). Furthermore, the condition controlling the position to apply auxiliary information may be updated on the fly, for example, when encoding a video sequence, where the condition may be updated at a block level (within a frame), or at a frame level, or at a group level of frames. One of the potential positions to apply auxiliary information is the position at which the signal is losslessly coded, as shown in FIG. 13. The change of position may be applied to two or more independent auxiliary signals. Given two independent auxiliary signals A1 and A2, there must be two corresponding sets of potential positions to apply A1 and A2, respectively. Assuming these two sets of potential positions are S1 and S2, the positions in S1 and S2 are not restricted, i.e., they may or may not overlap. The same applies to three or more independent auxiliary signals.

[0197] The embodiments have been described mainly in terms of methods. However, the present disclosure is not so limited. Rather, the present invention also relates to an apparatus for processing picture feature data from a bitstream using a neural network comprising multiple neural network layers. The apparatus comprises a processing circuit configured to obtain picture feature data from a bitstream and process the picture feature data using a neural network, and for each of one or more preconfigured positions in the neural network, the processing includes determining whether to collect auxiliary data for processing by one of the multiple neural network layers at said preconfigured positions based on a collection condition, and if it is determined that auxiliary data should be collected, the processing by the layer at said preconfigured position is based on the auxiliary data. The processing circuit may be, for example, one or more processors configured by corresponding software, as described in detail with reference to Figures 26 to 29.

[0198] Similarly, the invention relates to an apparatus for processing pictures using a neural network comprising a plurality of neural network layers to generate a bitstream, the apparatus comprising: a processing circuit configured to process pictures using the neural network, the processing comprising determining, for each of one or more preconfigured positions in the neural network, based on a collection condition, whether to collect auxiliary data for processing by the layer at said preconfigured positions, and if it is determined that auxiliary data should be collected, the processing by the layer at said preconfigured positions is based on the auxiliary data, the processing circuit configured to insert data obtained by processing the picture by the neural network into the bitstream. The processing circuit may for example be one or more processors configured by corresponding software, as will be described in detail with reference to figures 26 to 29.

[0199] Combining data from different processing stages of a neural network FIG. 14 is a block diagram showing the architecture of a neural network for processing picture data, in particular the numbering of layers applied on the encoder side and the decoder side. An input picture 1402 is stacked into an input data tensor and input to the encoder side neural network 1400A. As can be seen in the figure, the encoder side network 1400A comprises a number of cascaded processing layers (or blocks of layers) numbered from 1 to K. The stages here indicate stages of processed feature tensors. The input data tensor corresponds to the 0th stage, the feature tensor after processing the input data tensor with layer 1 corresponds to the 1st stage, etc., up to the last (Kth) stage, which corresponds to the output of the neural network at the bottleneck. The output here corresponds to a bottleneck feature tensor (also called output or latent space tensor or output feature data) that can be encoded (e.g., by an arithmetic encoder as exemplified above) to form a bitstream 1450.

[0200] The decoding-side network 1400B comprises a number of cascaded processing layers (or blocks of layers) numbered from K to 1. In this example, the processing layers (or blocks of layers) here correspond to the respective processing layers (or blocks of layers) at the encoder side for simplicity of explanation. Note that in general, the encoder-side network and the decoder-side network do not necessarily have to be strictly symmetrical, which means that they do not have to have the same number of layers, but can still have the same number of layers. The specific architecture may depend on the task for which they are deployed. Even for the same task (e.g., video / image coding), the structure of the NN may be different. Nevertheless, the number of downsampling layers at the encoder side is almost always equal to the number of upsampling layers at the decoder side.

[0201] The output of the decoding side network, in this example, is a reconstructed picture 1492. In general, however, the output of the decoding side does not have to be a reconstructed picture for human viewing: it can be feature data for, or the result of, computer processing, such as computer vision processing.

[0202] FIG. 15 shows in more detail the processing and data flow in a neural network that utilizes predictions and prediction residuals for encoding and decoding picture data at the same stage of the pipeline. In particular, these predictions and residuals are used at the same stage of the pipeline, namely the 0th stage. This can be a fixed arrangement or the result of the auxiliary information input position selection mentioned above, as explained above. In this example, the network described with reference to FIG. 11 is shown in more detail. FIG. 15 shows at the bottom an encoding and decoding network 1586 that processes the current picture, i.e. the frame at time instance t+1. The network 1586 basically corresponds to the networks 1400A and 1400B described with reference to FIG. 14. However, at stage 0 on the encoder and decoder side, auxiliary information is collected (1560, 1565) (similar to that shown in FIG. 11).

[0203] The information collected at the encoder side is a prediction signal. The prediction signal is combined with the input picture data (input tensor) at the 0th stage in part 1560 (corresponding to the collector). That is, the element-wise difference between the input tensor and the prediction is calculated, a residual is obtained, and the residual is further encoded, i.e., processed at the encoding side of the network 1586, and inserted into the bitstream 1572. It should be noted that the element-wise difference is only one exemplary possibility of implementing a common use or combination of data. In general, the combination may be performed in different ways. For example, if the two types of combined data have different resolutions or dimensions, the combination may further include rescaling. Depending on the neural network architecture and the training of the network, generally, other types of combinations, such as concatenation, may be used. Thus, even if some examples showing addition or subtraction are provided in the figures, the present disclosure is not limited to such operations. Rather, they may show a general combination of data, as will be further described with reference to Figures 21 and 22.

[0204] The decoder side uses layers K to 1 to process latent space data from the bitstream 1572. The resulting data at the 0th stage (the output of the first layer or layer block at the decoder side) are the decoded residuals. They are combined with the predictions collected in a unit denoted as 1565. The collecting here includes combining the predictions with the decoded residuals. The combining here is an element-wise sum of the decoded residual tensor and the predictions. Note that the predictions here have the same size (dimension) as the residual signal in order to perform element-wise processing in the collecting units 1560 and 1565. As mentioned above, the element-wise sum shown here is just an exemplary combination of two signals (types of data). In general, the combining can be done differently and can mean tensor concatenation or stacking, and / or rescaling, etc.

[0205] The upper part 1506 of FIG. 15 shows how the predictions collected for the processing of the current picture are obtained. In particular, the predictions are obtained based on reference frames (reference data) generated by the same network for the previous picture (at time instant t) and possibly on further data such as motion information that may be provided in the bitstream 1590 of the current picture (at time instant t+1). The neural network processing of the t-th (previous) picture by the encoding part of the network 1506 includes similar processing as described above for the current picture. For t=0, there are usually no previously coded frames. Therefore, only the decoded samples of the current (t=0) frame can be used as predictions. For example, intra prediction can be applied. Alternatively, such frames can be encoded in a different way, for example only by lossless coding.

[0206] The neural network processing of the t-th (previous) picture by the encoding part of the network 1506 includes providing a reference frame (generated in the 0-th stage by adding the decoded residual to the previously obtained reference frame) for further processing, i.e., motion estimation 1510 at the encoder side and motion compensation 1530 at the decoder side.

[0207] In motion estimation 1510, the motion between the reference picture (decoded at time instant t) and the current picture (at t+1) is estimated. The motion is then represented by motion information and inserted into the bitstream 1590. Furthermore, the estimated motion is used in motion compensation 1520. Motion compensation is performed according to any well-known technique. The motion information may include the entire optical flow or a subsampled (and possibly further encoded by lossless or lossy coding) optical flow for a particular part of the picture. In general, the motion information allows to generate a prediction from the reference picture. The motion information specifies which sample values ​​from the reference picture contribute (and how) to each sample value or predicted picture. The motion compensation 1520 generates a predicted picture, which is then collected in a collection unit 1560 to generate a residual signal (prediction error).

[0208] At the decoder side, the motion information is decoded from the bitstream 1590. During the decoding of the current picture, motion compensation 1530 generates a prediction picture based on the reference picture from the 0th stage of decoding of the previous picture and based on the motion information, in the same way as described for motion compensation 1520 at the encoder side.

[0209] It should be noted that FIG. 15 mainly shows the signal flow. In practice, buffering may be performed in several stages, for example, the reference picture may be stored in a reference picture buffer at the encoder and decoder side. The predicted picture may be generated using one reference picture as described above. However, the present disclosure is not limited to such processing, and the predicted picture may be generated using two or more predicted pictures. The state of the art provides various approaches using two reference pictures (bi-prediction), for example, a picture preceding the current picture in display order and a picture following the current picture in display order (bi-prediction). It should be further noted that the time instances t and t+1 are measured in units of processed pictures. However, they are also merely exemplary. The present disclosure is not limited to using the immediately preceding picture as a reference. In particular, for temporal scalability, a reference picture preceding or following the current picture in display order (but preceding the current picture in encoding and decoding order) may be used. Bitstreams 1590 and 1572 may be multiplexed into one bitstream.

[0210] FIG. 16 is similar to FIG. 15, since it shows the collection of predictions in the same stage as the residuals, but the collection of predictions and residuals is done in stage 1. Thus, the prediction does not have to be done in the input signal domain. In this case, the prediction is done in the latent space domain in a first stage, which is the stage after processing the current input picture 1601 (at time instant t+1) with the first layer or block or layer 1605. In particular, the first layer 1605 may include downsampling, in which case the handling of predictions in the latent space may provide improved efficiency, for example, in terms of complexity and latency. As seen in FIG. 16, the residuals are obtained in a collector unit 1660, for example, by subtracting (generally combining) the prediction tensor from the output of the first layer 1605. The output of the first layer 1605 in this case may be a subsampled (and possibly further processed, e.g. filtered) input picture tensor of the same size as the prediction tensor, which is obtained based on a reference frame obtained from the first stage (network 1606) of the decoder pipeline of the previous picture. The corresponding motion information may then lead to less data being conveyed in the bitstream 1690, which may be multiplexed with the residual bitstream 1672. On the decoder side, the collection 1665 is also performed in the corresponding stage (here the first stage, e.g. the stage preceding the processing by the first layer or blocks of layers in the decoding processing order). The collection includes a combination (in this case element-wise summation) of the prediction signal and the decoded residual in the latent space of the first stage, as in FIG. 15.

[0211] As will be clear to those skilled in the art, following Fig. 15 and Fig. 16, the prediction signal collection may be performed at any stage from 2 to K. The higher the stage, the lower the accuracy of the prediction may be, especially if the layer (or block of layers) includes subsampling. On the other hand, as the stage increases, the rate of the generated bitstream may decrease, and the complexity and latency may also decrease. Thus, for low bitrate / quality coding, it may be advantageous to have the above-mentioned stages later (corresponding to a larger stage index), while for high bitrate / quality, it may be advantageous to have the above-mentioned stages earlier (corresponding to a smaller stage index). The possibility to independently select the prediction / residual collection stage may improve the network operation. Also, the stage selection may be adaptively performed depending on the content characteristics (e.g., spectrum), as described in the above embodiments and examples described with reference to Figs. 1 to 13.

[0212] 15 and 16 show examples where the prediction and the residual are obtained from the same stage of processing (0th or 1st in these examples). However, according to one embodiment of the present disclosure, this is not necessarily the case. It may be advantageous to obtain the prediction and the residual from different stages.

[0213] FIG. 17 shows an example embodiment in which prediction is obtained within the i-th stage and residual coding is performed in the (i+r)-th stage of the NN pipeline.

[0214] According to such an embodiment, a method is provided for processing feature data of one or more pictures from a bitstream 1772 (and possibly 1790) using a neural network. The neural network 1786 comprises two or more stages (K to 1), including a first stage and a second stage, where the "first" stage and the "second" stage are any of stages K to 1, and the terms "first" and "second" are used as placeholders for the stage number.

[0215] The method includes obtaining first data 1760 based on a bitstream and processing the first data using a neural network. For example, the first data in FIG. 17 is a residual decoded from a bitstream 1772 and possibly also processed by one or more layers. In FIG. 17, the first data is a residual decoded from a bitstream 1772 and processed by layers K to i+r+1. In other words, the residual is used in the (i+r)th stage of the decoding pipeline.

[0216] Note that in general, the first data may be feature data entropy coded directly from the bitstream, without necessarily having to be processed by one or more layers. As discussed below, the present disclosure may also apply to entropy encoding / decoding.

[0217] The process using neural networks is - obtaining second data 1765 from a second stage of the neural network 1706, the second data 1765 being based on data previously processed by the neural network; - using the first data together with the second data to generate inputs to a first stage of a neural network, the first stage preceding the second stage in the neural network's feature data processing (in the example of FIG. 17, the order of processing in the decoding portion of the network, i.e., from stage K to stage 0); Includes.

[0218] The term "previous" here means a previous frame or a previous block (intra), etc. The method includes a step of outputting a result of the process 1702 (e.g. a reconstructed picture or picture feature data). In general, this embodiment is applicable whenever the first stage and the second stage are different. However, as shown with reference to the embodiments described with reference to Figures 1 to 16, the first and the second stages can also be the same stage.

[0219] The second data in this example may be a predicted frame. Obtaining the second data may include obtaining a reference frame and motion information. However, in general, the reference frame, or in general, the reference data, may be obtained directly as a prediction (e.g., when motion is not signaled or when intra-prediction or another type of prediction is used). Note that FIG. 17 shows processing at the picture (frame) level. However, the present disclosure is not limited thereto, and in general, a picture portion may be encoded / decoded.

[0220] In the example of FIG. 17, the first stage is the (i+r)th stage of the decoding pipeline, and the second stage is the ith stage of the decoding pipeline.

[0221] In an exemplary implementation, the first data (e.g., residual) is obtained in a third stage of the neural network preceding or equal to the first and second stages in the neural network feature data processing. In other words, the first data may be obtained in a different stage from the stage where the first data is combined with the second data. This may be advantageous for some neural network architectures and may correspond to a kind of skip connection. The Kth to (i+r+1)th layers may be used to pre-process the decoded residual signal and transform it into the same feature space as the second data processed using the (i+1) to (i+r) layers. Essentially, layers (i+1) to (i+r) and K to (i+r+1) are used to transform the first data and the second data into the same feature space (same domain) in the exemplary implementation shown in FIG. 17. As soon as the processed first data and the processed second data are in the same domain, they may be combined as shown in FIG. 21-FIG. 22.

[0222] 17, the first data represents the prediction error (prediction residual) and the second data represents the prediction. In particular, the prediction is obtained by: - obtaining reference feature data, which is the feature data output by the neural network in said second stage. In the general case of the i-th stage, the reference data is not in the picture domain but in the latent space domain, and is therefore generally referred to herein as "reference feature data". As already discussed with reference to Figure 15, the reference feature of the 0-th stage is the reference picture. - Obtaining prediction information, including motion information or spatial information related to reference feature data (at the same or different stage) based on the bitstream. The relation here is given by the fact that the prediction information was determined with respect to the reference feature data in the encoding part of the network. In Fig. 17, the motion information included in the bitstream 1790 is obtained by performing motion estimation for the motion between the reference feature data and the feature data processed by layers 1 to i at the encoder side. Other exemplary implementations are possible, such as for example using the input picture data 1701 stacked in a tensor to generate the motion information and then downsampling it. As will be clear to those skilled in the art, rescaling or resizing may be applied, if necessary. - generating a prediction based on the reference feature data and the prediction information. In Fig. 17, the generation of the prediction is performed by motion compensation, as explained above with reference to Fig. 15. Thus, the second data is a prediction 1765 corresponding to the output of the motion compensation.

[0223] In Fig. 17, the prediction error is obtained by processing the current picture using a neural network. The prediction information is motion information, and the reference feature data is generated by neural network processing of picture data representing a picture preceding the current picture in decoding order, and possibly further applying motion estimation. As also mentioned above, the decoding order is not necessarily the display order, and the prediction may be based on two or more references.

[0224] Using the first data 1760 together with the second data 1765 described above includes an element-wise addition of a prediction or a rescaled prediction and a prediction error or a rescaled prediction error. Such rescaling of the prediction data is shown in FIG. 17 and is included, for example, within the processing by layers i+1 to i+r (which may be, for example, convolutional layers). As described above, the element-wise addition may be replaced by a different kind of combination, such as concatenation or other operation or sequence of operations.

[0225] It should be noted that the present disclosure is not limited to the auxiliary data being predictions. As described below, in some embodiments, the auxiliary (second) data may be obtained from a different (sub)network as a hyper-prior and used together with the data encoded in the 0th stage to generate latent space feature data at the input of the decoding side of the neural network. In such a case, using together refers to applying probability model information, for example, to arithmetic decoding of picture feature data.

[0226] In Figure 17, the motion information includes motion vectors. However, this disclosure is not limited to any particular method of motion estimation and compensation. Thus, in general, the motion information is not limited to motion vectors. Rather, the motion information may include affine transformation parameters. In addition to the motion vectors, the motion information may include positions relative to the signaled motion vectors (in the case of non-regular motion field sampling).

[0227] Figure 18 is a specific case of the more general embodiment of Figure 17. In particular, in Figure 18, the second stage is the output of the (decoder) neural network, i.e. the 0th stage. In other words, the reference frame is obtained from the 0th stage of the decoding process, as already shown in Figure 15. Note that the 0th stage in this case is the stage of processing the previous picture (time instance t). However, in this example, it is assumed that the network architecture does not change for the processing of the current picture or picture feature data (time instant t+1) and the previous picture or picture feature data (time instant t).

[0228] In Figure 18, the first stage (equal to the third stage described above) is K, i.e. the residual is calculated (at the encoder) and used (at the decoder) in the Kth stage of the pipeline. However, the disclosure is not limited thereto and the first stage may be the input of a neural network on the decoder side, especially in the case of embodiments relating to lossless coding, which are described in more detail below.

[0229] Figure 19 shows another example where the reference picture (and therefore also the prediction) is obtained at stage 1 of the pipeline, i.e. the second stage is stage 1 of the decoding pipeline. This corresponds to Figure 16. The first stage for calculating and using the residual remains stage K of the encoding and decoding pipeline, as in Figure 18 described above.

[0230] Another example is shown in FIG. 20. In FIG. 20, the second stage for obtaining a reference frame is stage 1 as in FIG. 16 and FIG. 19. However, the first stage for calculating and using a residual is stage (K-1) of the encoding and decoding pipeline. In other words, on the encoding side, after obtaining the residual (after combining the first data and the second data), the combined data may be processed by one or more layers 2020 of the neural network before inserting the processed data into the bitstream 2050. In this exemplary processing pipeline, as well as in the examples of FIG. 17 to FIG. 19, the prediction (corresponding to the second data) may also be processed by one or more layers 2010 of the neural network on the encoding side. Correspondingly, on the decoding side, before combining the prediction further processed in stage K-1, the data from the bitstream may be processed by one (layer K in FIG. 20) or more layers 2022. Also, the prediction may be processed by one or more layers 2012 of the neural network on the decoding side. As mentioned above, processing of the first data and the second data before their combination may include, but is not limited to, adapting them to a common size (resolution).

[0231] The above example shows a symmetric encoding and decoding pipeline and a symmetric collection of predictions at corresponding stages in the encoding and decoding pipeline. Note that this is not necessarily the case. Due to the possibility of rescaling and resizing, asymmetric solutions are possible.

[0232] 15 to 20 described a specific exemplary embodiment in which prediction is applied to obtain a residual or to obtain picture (feature) data. Illustratively, on the encoding side, to obtain a residual, prediction data (second data) was subtracted from the picture (feature) data input to the neural network (and possibly processed by several layers). The subtraction was element-wise. Similarly, on the decoding side, the residual was combined with the prediction element-wise. Such an approach is illustrated in FIG. 21. In FIG. 21, the difference between the prediction block and the original block was determined element-wise and included in the bitstream. Then, the difference (also denoted as the residual or prediction error in this disclosure) is obtained from the bitstream on the decoder side and added to the prediction block to obtain a reconstructed block. This example is merely schematic, and the blocks may directly represent picture data or picture feature data.

[0233] However, in some embodiments, direct combining of original blocks and predicted blocks and / or direct combining of residual blocks and predicted blocks is not always possible for each block. For example, as discussed above, prediction, input (original) data, and residual may have different sizes (resolutions). Thus, FIG. 22 shows a generalized combining in an encoder and a decoder. For example, in an encoder, obtaining residual may include upsampling or downsampling at least one of original data and predicted data. Similarly, in a decoder side, applying residual to prediction may include rescaling / resizing of residual and / or prediction, etc. In one example, input (original) data is stacked with predicted data, so that the number of channels becomes the sum of the number of original data channels and predicted data channels, and such combined data is further processed by one or more convolutional layers (possibly by downsampling or strided convolution), and the result is a residual. At the decoder side, the application of the residual is done in the following way: the residual is stacked with the prediction data and then processed (possibly by downsampling or strided convolution) by one or more convolution layers, and the result of such processing is the reconstructed data (which can have more than one channel since it is a latent space). At the encoder side, if the original data and the prediction data have different sizes (resolutions), rescaling / resizing can be applied before stacking the channels. At the decoder side, if the residual and the prediction data have different sizes (resolutions), rescaling / resizing can be applied before stacking the channels. Also, some convolution layers can be used to pre-process the prediction and / or the prediction error before combining. In general, such kind of pre-processing can be different for prediction and prediction error.

[0234] In the following, an embodiment is provided for application in lossless encoding and decoding. In general, the first data is obtained after the encoding part of the autoencoder and is further quantized before arithmetic coding. In this case, such an embodiment using the second data as the entropy model can still function for the lossy coding preceding the lossless coding.

[0235] Thus, the first stage is an input of a neural network. The first data is entropy encoded data. The second data is probability model data for the feature data of the second stage. For example, the second data is probability model data for entropy decoding of the first data.

[0236] In this exemplary implementation, the second data is used to make a prediction of the probability of the symbols in the first data, which may substantially improve arithmetic coding. The technical effect is comparable to subtracting the prediction from the signal and encoding the difference (residual), for example close to zero. If the symbols encoded by arithmetic coding are the quantized output of the encoding part of the autoencoder (which is one of the advantageous specific practical embodiments), lossy coding is implemented. If the symbols are raw image samples (which are already integers and can be encoded by arithmetic coding), lossless coding is implemented.

[0237] This embodiment is an extension of the embodiment described with reference to Fig. 14. In Fig. 14, the probability model is determined based on feature data in the 0th stage and applied in the 0th stage of encoding and decoding (for arithmetic encoding and arithmetic decoding). However, this does not generally have to be the case. Rather, the second data corresponding to the probability model information may be obtained, for example, by applying a hyper prior sub-network based on picture feature data in the latent space, for example, from stages greater than 0.

[0238] In general, in any of the above examples or embodiments, the position of the first stage and / or the second stage may be configurable. For example, the position of the first stage and / or the second stage may be configurable in a neural network, and the processing method at the encoding side and / or the decoding side includes configuring the position of the first stage and / or the second stage according to a collection condition based on one or more picture coding parameters. In other words, a technique for selecting a collection position based on a collection condition may be applied to the first and / or the second stage. The collection condition may be based on parameters (information) available to both the encoder and the decoder.

[0239] Alternatively or in addition, the collection condition may depend on a selection performed in the encoder and signaled to the decoder. For example, the decoding method may include parsing a stage selection indicator from the bitstream, which designates the first stage and / or the second stage. The positions of the first stage and the second stage in the neural network are configured according to the parsed stage selection indicator. At the encoding side, a stage selection indication is generated and inserted into the bitstream. As explained above, the encoder may decide to select a particular stage based on some optimization based on a cost function including rate, distortion, complexity, latency, accuracy, etc. Alternatively, other techniques such as the amount of lost information (e.g., FFT2 as explained above) may be applied.

[0240] In summary, the present disclosure provides some embodiments that may make a neural network-based image / video codec, or picture processing in general, dynamically configurable, where the position in the coding pipeline to apply auxiliary information may depend on conditions. Further or alternative improvements may result from an architecture in which auxiliary information obtained in one pipeline stage in a reference frame may be used in other pipeline stages for a current frame. An example of such a scheme for prediction and residual auxiliary information is discussed with reference to FIG. 17. For example, a prediction obtained from the i-th stage of the pipeline on the reference frame side may be further processed by r subsequent NN modules and used within the (i+r)-th stage of the pipeline for a current frame to calculate residuals. In this case, the residuals are calculated between the predictions processed by NN modules i+1, i+2, ..i+r and the current frame processed by NN modules 1, .., i+r. The residuals are then further processed by NN modules (i+r+1), .., K. The output of the Kth NN module (at the encoder side) is the bottleneck and is signaled in the bitstream. At the decoder side, the same scheme is applied: predictions are processed by NN modules i+1, i+2, .. i+r, and the residuals obtained from the bitstream are processed by modules K, .., (i+r+1). The combined signal obtained from the predictions and the residuals is then processed by modules (i+r), .., 1.

[0241] As explained above for one-stage selection, on the encoder side, the decision regarding the stage in the pipeline where the side information is applied can be made by multi-pass encoding. In one example, applying side information is allowed only at one position. In this case, the encoder for k=0, 1, 2, .., K will try to apply side information at the kth position of the pipeline and obtain the distortion and rate for each variant. The best variant for the defined rate / distortion ratio is then selected. In another example, if applying side information is possible at all positions in the pipeline, the encoder for each Boolean vector of length (K+1) will try to apply side information at the position corresponding to the position in this Boolean vector, so 2 K+1 variants are tried.

[0242] In the context that more than one stage may be selected, all combinations of i and r may be tried. The following condition must be satisfied: 0≦i≦i+r≦K. To avoid trying all possible positions for the auxiliary information application and thus reduce the complexity of the encoder, heuristic methods may be used instead of an exhaustive search. For example, the spectral analysis described above (e.g., using FFT2) may be used.

[0243] In the exemplary implementations shown in Figures 15-20, the prediction is calculated only once and the residual is applied only once. In general, a coding (encoding and / or decoding) scheme may comprise two or more prediction / residual acquisition blocks at different stages (e.g., for processing by different NN layers). In summary, in the above-mentioned embodiments, the auxiliary information may include a prediction signal and / or a residual signal. The prediction signal is obtained from the i-th pipeline position of a previously coded frame, and the residual signal is applied at the k-th pipeline position during a current frame processing. In some embodiments, i and k are different from each other. The values ​​i and k may be pre-determined, e.g., fixed in the neural network architecture, or configurable in the neural network architecture. For example, the values ​​i and k depend on conditions derived at the encoder and decoder side. Alternatively, or in addition, the values ​​i and k are determined based on syntax elements parsed from the bitstream.

[0244] In some embodiments, k is equal to i+r, where r is equal to or greater than zero. Such an example is shown in the general scheme in FIG. 17. FIG. 20 illustrates the scheme for i=1, i+r=K-1, where K is the number of NN modules in the encoder pipeline. The modules may be shown as blocks and may include one or more layers. In a possible implementation, i is equal to 0, i.e., prediction is performed in a single domain (the 0th position in the pipeline). In some other possible implementations, i+r is equal to K, where K is the number of NN modules in the encoder pipeline. Residual coding is performed in the bottleneck, i.e., in the Kth position in the pipeline. FIG. 18 illustrates the scheme for i=0, i+r=K. FIG. 19 illustrates the scheme for i=1 and i+r=K.

[0245] In any of the above examples, the prediction signal may be calculated based on information from the p-th pipeline position of the previously coded frame and additionally signaled information. For example, the additionally signaled information includes motion information. For example, the residual signal is calculated by a module having two inputs: the prediction signal and the output of the k-th NN module in the pipeline. In some embodiments, the residual signal is calculated as the difference between the prediction signal and the output of the k-th NN module in the pipeline. Furthermore, the application of the residual signal is performed by a module having two inputs: the residual signal and the prediction signal. Finally, the application of the residual signal is performed by adding the residual signal to the prediction signal.

[0246] 23 to 25 summarize some possible architectures of the processing pipeline.

[0247] FIG. 23 shows an example where the original picture, the prediction and the residual are in the signal domain, i.e., the 0th stage of processing by the encoding neural network. The original picture and the prediction are combined into the obtained residual. The residual is encoded using a VAE with a hyper-prior. The result is a bitstream containing the coded residual and possibly some side information. The decoder side is input with the bitstream. The decoder here is a VAE with a hyper-prior that outputs a reconstructed residual. The reconstructed residual may be combined with the prediction signal to obtain a reconstructed picture. The reconstructed picture may be further compared to the original picture, for example by applying a distance measure, for example an objective or perceptual quality metric.

[0248] FIG. 24 shows an example where the original picture, the prediction, and the residual are in the latent space domain, i.e., the first stage of processing by the encoding neural network. In this case, the original picture and the prediction are first processed by one or more layers of the VAE encoder, and then combined to obtain the residual in the latent space domain. These are encoded by a hyper-ahead encoder, thereby generating a bitstream. The bitstream can be decoded by a hyper-ahead decoder to obtain a reconstructed latent space residual. The reconstructed latent space residual is combined with the prediction (also in the latent space domain) in the latent space domain, thus obtaining a reconstructed picture feature tensor that is input to the VAE decoder to obtain a reconstructed picture. The reconstructed picture can be further compared to the original picture, for example, by applying a distance measure, for example, an objective or perceptual quality metric.

[0249] FIG. 25 shows that an exemplary original picture is encoded by a VAE encoder, and then a residual is obtained in the latent space by subtracting (or combining) the encoded picture with a prediction obtained from the latent space of the previous frame. The residual is encoded into a bitstream using a hyper-ahead encoder. The bitstream can be decoded by a hyper-ahead decoder, thus obtaining a reconstructed residual in the residual space. These are combined with the prediction obtained from the latent space of the previous frame, and thus a reconstructed picture feature tensor in the latent space is obtained. The reconstructed picture is obtained by transforming the obtained reconstructed picture feature tensor by a VAE decoder. The reconstructed picture can be further compared with the original picture, for example, by applying a distance measure, for example, an objective or perceptual quality metric.

[0250] Any of these three architectures can be the basis for using the embodiments and exemplary implementations described above. These architectures included a VAE encoder and a decoder. However, in general, the present disclosure is not limited to a VAE with a hyper-prior. Other approaches to symbol probability estimation for arithmetic coding can be applied as well. For example, a context-based approach can be used. In this case, information about previously decoded samples is additionally used by the symbol probability estimation NN. An example of a probability estimation NN using a context approach is shown in FIG. 13.

[0251] Overview of Some Embodiments The present application further provides a method and apparatus for processing picture data or picture feature data using a neural network having two or more layers. The present disclosure may be applied in the field of artificial intelligence (AI)-based video or picture compression technology, in particular in the field of neural network-based video compression technology. According to some embodiments, two types of data are combined during processing, including processing by the neural network. The two types of data are obtained from different stages of processing by the network. Some of the advantages may include higher scalability and more flexible design of the neural network architecture, which may further lead to better encoding / decoding performance.

[0252] According to a first aspect, the present disclosure relates to a method for processing feature data of one or more pictures from a bitstream using a neural network comprising two or more stages including a first stage and a second stage. The method includes the steps of obtaining first data based on the bitstream, processing the first data using the neural network, and outputting a result of the processing. The processing includes obtaining second data from the second stage of the neural network, the second data being based on data previously processed by the neural network, and using the first data together with the second data to generate an input to the first stage of the neural network, where the first stage precedes the second stage in the neural network feature data processing.

[0253] Obtaining two types of data that are used together in one stage and obtaining at least one type of data in another stage further increases the flexibility of the neural network architecture and may result in greater efficiency in terms of reduced complexity or latency, or rate, or higher quality.

[0254] In a possible implementation, the first data is obtained in a third stage of the neural network preceding the first and second stages or equal to the first stage in the neural network feature data processing.

[0255] Acquiring the first and second data at different stages may further improve efficiency.

[0256] In a possible implementation, the first data represents a prediction error and the second data represents a prediction.

[0257] The present disclosure is easily applicable to prediction error and prediction signal. By obtaining at least one of them in a resolution different from the resolution in which they are combined, it may save complexity, bitstream length, and latency. Furthermore, in picture coding, since there is high correlation in the spatial and temporal domains, prediction is auxiliary data that may improve the performance of decoding (and encoding) using neural networks. Furthermore, there are many techniques known from the art that may be very efficient in providing prediction.

[0258] In a possible implementation form, the prediction is obtained by obtaining reference feature data, which is the feature data output by the neural network in the second stage, obtaining prediction information including motion information or spatial information regarding the reference feature data based on the bitstream, and generating a prediction based on the reference feature data and the prediction information.

[0259] This implementation is an example of obtaining a prediction with the help of additional prediction information that may help to further improve the prediction quality.

[0260] In a possible implementation, the prediction error is obtained by processing the current picture with a neural network, the prediction information being motion information and the reference feature data being generated by neural network processing of picture data representing a picture preceding the current picture in decoding order.

[0261] Temporal prediction is typically more efficient than spatial prediction, or lossless coding. Therefore, the above neural network architecture can further improve the decoding (as well as encoding) performance.

[0262] In a possible implementation, using the first data together with the second data includes element-wise addition of a prediction or a rescaled prediction and a prediction error or a rescaled prediction error.

[0263] Such combinations may be particularly relevant for processing picture or picture feature data, predictions, and residuals.

[0264] In a possible implementation, the motion information includes motion vectors.

[0265] Motion vectors are an efficient means for indicating motion information, and there are many available techniques for their acquisition and signaling that can be readily applied to the embodiments described herein.

[0266] In a possible implementation, the second stage is the output of a neural network.

[0267] The output of the neural network is the reconstructed picture (on the decoding side). Obtaining second data from this stage can provide full resolution and improve quality.

[0268] In a possible implementation, the first stage is the input of a neural network.

[0269] The input of the neural network on the decoding side is the encoded bitstream. The embodiments of the present disclosure can be effectively applied to the decoding of bottleneck feature data.

[0270] In a possible implementation, the first stage is an input of a neural network, the first data is entropy encoded data, the second stage is a stage different from the output stage of the neural network, and the second data is probability model data for the feature data of the second stage, e.g., the second data is probability model data for entropy decoding of the first data.

[0271] The collection of probability model data may enable further reduction in bitstream size. Obtaining probability model data from different stages may provide a flexible architecture for better tradeoff between performance and complexity.

[0272] In a possible implementation, the positions of the first stage and / or the second stage are configurable within a neural network, and the method includes a step of configuring the positions of the first stage and / or the second stage according to collection conditions based on one or more picture coding parameters.

[0273] The possibility to configure the location of the first and second stages in the neural network applies additional flexibility. This may allow dynamic neural network architecture changes. Such flexibility may result in better adaptation based on acquisition conditions, leading to more efficient encoding and / or decoding.

[0274] The collection condition is a condition or precondition that must be met for the second data and / or the first data to be input to a particular processing stage. The collection condition may include a comparison of some picture characteristics or picture feature characteristics with a threshold value to determine whether to collect auxiliary information for a particular position. The picture characteristics or picture feature characteristics may be known to the encoder and decoder so that no additional signaling is required. Alternatively, or in addition, the collection condition may be configured by the encoding side by setting an indicator of whether auxiliary information should be collected for a preconfigured position. The indicator may be provided in the bitstream available at the decoder.

[0275] In particular, the method further comprises the step of parsing a stage selection indicator from the bitstream specifying the first stage and / or the second stage, and the positions of the first stage and the second stage within the neural network are configured according to the parsed stage selection indicator.

[0276] According to a second aspect, the present invention relates to a method for processing at least one picture using a neural network comprising two or more stages, including a first stage and a second stage, to generate a bitstream, the method comprising the steps of processing at least one picture using the neural network, the processing comprising the steps of obtaining first data based on the at least one picture, obtaining second data in said second stage of processing, the second data being based on data previously processed by the neural network, and using the first data together with the second data to generate inputs in said first stage of the neural network, the first stage preceding the second stage in the feature data processing of the neural network. The method further comprises the step of inserting the feature data obtained by the processing into the bitstream.

[0277] The encoding portion of this disclosure may provide the same advantages as discussed above for the decoding portion: the encoder prepares a bitstream and provides the bitstream to the decoder such that the decoder may decode or reconstruct the data with the desired quality and use in mind.

[0278] In order to avoid redundancy, the advantages provided in the corresponding decoding process claims apply in an analogous manner to the encoding.

[0279] In a possible implementation, the first data is obtained in a third stage of the neural network that precedes the first and second stages in the neural network feature data processing or is equal to the first stage.

[0280] In a possible implementation, the first data represents a prediction error and the second data represents a prediction.

[0281] In a possible implementation form, the prediction is obtained by obtaining reference feature data, which is the feature data output by the neural network in the second stage, obtaining prediction information including motion information or spatial information regarding the reference feature data based on at least one picture, generating a prediction based on the reference feature data and the prediction information, and inserting the obtained prediction information into the bitstream.

[0282] In a possible implementation, the prediction error is obtained by processing the current picture with a neural network, the prediction information is motion information, and the reference feature data is generated by neural network processing of picture data representing a picture preceding the current picture in decoding order.

[0283] In a possible implementation, using the first data together with the second data includes using the first data or Is It involves element-wise subtraction of the prediction or the rescaled prediction from the scaled first data.

[0284] In a possible implementation, the motion information includes motion vectors.

[0285] In a possible implementation, the second stage is the output of a decoding neural network representing the reconstructed picture data.

[0286] In a possible implementation, the first stage is the output of a neural network.

[0287] In a possible implementation, the first stage is the output of a neural network, the first data is processed data to be entropy encoded, the second stage is a stage different from the input stage of the neural network, and the second data is probabilistic model data related to the feature data of the second stage.

[0288] In a possible implementation, the second data is probability model data for entropy encoding of the first data.

[0289] In a possible implementation, the positions of the first stage and / or the second stage are configurable within a neural network, and the method includes a step of configuring the positions of the first stage and / or the second stage according to collection conditions based on one or more picture coding parameters.

[0290] In a possible implementation, the method further includes determining and including in the bitstream a stage selection indicator designating the first stage and / or the second stage, and the positions of the first stage and the second stage within the neural network are configured according to the determined stage selection indicator.

[0291] In a possible implementation, the determination of the stage selection indicator is based on an optimization procedure performed according to a cost function including one or more of rate, distortion, latency, accuracy, and complexity.

[0292] The determination of the stage based on the cost function can improve the adaptation of the neural network and the results to the desired requirements. Thus, such optimization can improve the performance. In combination with indicating the stage positions thus obtained, the flexibility is further improved.

[0293] According to a third aspect, the present disclosure relates to an apparatus for processing feature data of one or more pictures from a bitstream using a neural network comprising two or more stages including a first stage and a second stage, the apparatus comprising a processing circuit configured to obtain first data based on the bitstream, process the first data using the neural network, and output a result of the processing, the processing including obtaining second data from the second stage of the neural network based on data previously processed by the neural network, and using the first data together with the second data to generate an input to the first stage of the neural network, the first stage preceding the second stage in the neural network feature data processing.

[0294] For the advantageous effects of the present disclosure, please refer to the description of the first aspect. Details will not be described again here. The decoding device has a function of implementing the actions in the example of the method in the first aspect. The functions can be implemented by hardware, or can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. In a possible implementation form, the decoding device includes a bitstream decoding module for obtaining first data based on a bitstream, and a neural network module configured to perform the above-mentioned processing and output. These modules may perform the corresponding functions in the example of the method in the first aspect. For details, please refer to the detailed description in the example of the method. Details will not be described again here.

[0295] According to a fourth aspect, the present disclosure relates to an apparatus for processing at least one picture using a neural network comprising two or more stages, including a first stage and a second stage, to generate a bitstream, the apparatus comprising a processing circuit. The processing circuit is configured to process at least one picture using the neural network. The processing includes an operation of obtaining first data based on the at least one picture, obtaining second data in the second stage of processing, the second data being based on data previously processed by the neural network, and an operation of using the first data together with the second data to generate an input in the first stage of the neural network, the first stage preceding the second stage in the neural network feature data processing. The method further includes an operation of including the feature data obtained by the processing in the bitstream.

[0296] For the advantageous effects of the present invention, please refer to the description of the second aspect. Details will not be described again here. The encoding device has a function of implementing the actions in the example of the method in the second aspect. The functions can be implemented by hardware, or can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. In a possible implementation form, the encoding device includes a neural network module configured to perform the above-mentioned processing, and a bitstream output module configured to process the picture by the neural network to insert the obtained data into a bitstream and output the bitstream. These modules may perform the corresponding functions in the example of the method in the second aspect. For details, please refer to the detailed description in the example of the method. Details will not be described again here.

[0297] The method according to the first aspect of the invention may be performed by an apparatus according to the third aspect of the invention. Further features and implementations of the method according to the first aspect of the invention depend directly on the functionality and implementations of the apparatus according to the third aspect of the invention.

[0298] The method according to the second aspect of the invention may be performed by an apparatus according to the fourth aspect of the invention. Further features and implementations of the method according to the second aspect of the invention depend directly on the functionality and implementations of the apparatus according to the fourth aspect of the invention.

[0299] According to a fifth aspect, the present invention relates to a bitstream decoding device comprising a processor and a memory, the memory storing instructions for causing the processor to carry out the method according to the first aspect.

[0300] According to a sixth aspect, the invention relates to a bitstream encoding device comprising a processor and a memory, the memory storing instructions for causing the processor to carry out the method according to the second aspect.

[0301] According to a seventh aspect, a computer-readable storage medium is proposed having stored thereon instructions which, when executed, cause one or more processors to encode video data, the instructions causing the one or more processors to perform a method according to the first or second aspect or any possible embodiment of the first or second aspect.

[0302] According to an eighth aspect, the present invention relates to a computer program product comprising a program code for performing the method according to the first or second aspect or any possible embodiment of the first or second aspect, when said program code is executed on a computer.

[0303] Software and Hardware Implementation In the following, embodiments of the video coding system 10, the video encoder 20, and the video decoder 30 will be described with reference to FIGS.

[0304] 26 is a schematic block diagram illustrating an example coding system 10, e.g., video coding system 10 (coding system 10 for short), that may utilize techniques of the present application. Video encoder 20 (or encoder 20 for short) and video decoder 30 (or decoder 30 for short) of video coding system 10 represent examples of devices that may be configured to perform techniques according to various examples described in the present application. Video encoder 20 and video decoder 30 may be implemented using (including or as) a neural network, as described in the above embodiments and examples. The remainder of system 10 may be, but need not be, implemented by a neural network.

[0305] As shown in FIG. 26, coding system 10 includes a source device 12 configured to provide encoded picture data 21 to a destination device 14 for decoding, for example, encoded picture data 13.

[0306] The source device 12 includes an encoder 20, and may additionally, i.e. optionally, include a picture source 16, a pre-processor (or pre-processing unit) 18, such as a picture pre-processor 18, and a communications interface or unit 22.

[0307] Picture source 16 may include or be any type of picture capture device, e.g., a camera for capturing real-world pictures, and / or any type of picture generation device, e.g., a computer graphics processor for generating computer-animated pictures, or any type of other device for obtaining and / or providing real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). Picture source may be any type of memory or storage that stores any of the aforementioned pictures.

[0308] To distinguish from the pre-processor 18 and the processing performed by the pre-processing unit 18 , the picture or picture data 17 may also be referred to as a raw picture or raw picture data 17 .

[0309] The pre-processor 18 is configured to receive the (raw) picture data 17 and perform pre-processing on the picture data 17 to obtain a pre-processed picture 19 or pre-processed picture data 19. The pre-processing performed by the pre-processor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal. It may be understood that the pre-processing unit 18 may be an optional component. It may be understood that the pre-processing unit 18 may be an optional component.

[0310] Video encoder 20 is configured to receive pre-processed picture data 19 and to provide encoded picture data 21 (as described in further detail in 1 to 25 above).

[0311] Communications interface 22 of source device 12 may be configured to receive encoded picture data 21 and transmit the encoded picture data 21 (or any further processed version thereof) via communications channel 13 to another device, such as destination device 14 or any other device, for storage or direct reconstruction.

[0312] The destination device 14 includes a decoder 30 (e.g., a video decoder 30), and may additionally, i.e. optionally, include a communications interface or unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34.

[0313] The communications interface 28 of the destination device 14 is configured to receive the encoded picture data 21 (or any further processed version thereof), for example directly from the source device 12 or any other source, for example a storage device, for example an encoded picture data storage device, and to provide the encoded picture data 21 to a decoder 30.

[0314] Communications interface 22 and communications interface 28 may be configured to transmit or receive encoded picture data 21 or encoded data 13 via a direct communications link between source device 12 and destination device 14, e.g., a direct wired or wireless connection, or via any type of network, e.g., a wired or wireless network or any combination thereof, or any type of private and public network or any combination thereof.

[0315] The communications interface 22 may be configured, for example, to package the encoded picture data 21 into an appropriate format, e.g., packets, and / or to process the encoded picture data using any type of transmission encoding or processing for transmission over a communications link or network.

[0316] Communications interface 28, which forms a counterpart of communications interface 22, may be configured to, for example, receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or depackaging to obtain encoded picture data 21.

[0317] Both communication interface 22 and communication interface 28 may be configured as one-way communication interfaces, as indicated by the arrow pointing from source device 12 to destination device 14 for communication channel 13 in FIG. 26, or as two-way communication interfaces, and may be configured to, for example, send and receive messages to, for example, set up a connection, confirm and exchange a communication link and / or any other information related to a data transmission, e.g., the transmission of encoded picture data.

[0318] The video decoder (or decoder) 30 is configured to receive the encoded picture data 21 and to provide decoded picture data 31 or decoded pictures 31 (further details are described above, for example based on Figures 1 to 25).

[0319] The post-processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), e.g., a decoded picture 31, to obtain post-processed picture data 33, e.g., a post-processed picture 33. The post-processing performed by the post-processing unit 32 may include, e.g., color format conversion (e.g., from YCbCr to RGB), color correction, cropping or resampling, or any other processing, e.g., to prepare the decoded picture data 31, e.g., for display, e.g., by a display device 34.

[0320] A display device 34 of destination device 14 is configured to receive the post-processed picture data 33, for example, to display the picture to a user or viewer. The display device 34 may be or include any type of display, e.g., an integrated or external display or monitor, for displaying the reconstructed picture. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any type of other display.

[0321] The coding system 10 further includes a training engine 25. The training engine 25 is configured to train the encoder 20 (particularly the neural network module in the encoder 20) or the decoder 30 (particularly the neural network module in the decoder 30) to process input pictures, picture regions, or picture blocks. It should be noted that the embodiments and examples described above have been described for neural networks that process pictures (sometimes called frames). However, the present disclosure is also applicable to any other granularity, where a portion (region) of a complete picture may be treated as a picture during encoding and decoding.

[0322] It should be noted that since neural networks according to embodiments may use pre-trained networks, a training engine need not be provided within the system of the present disclosure.

[0323] The training data may be stored in a database (not shown). The training engine 25 performs training based on the training data to obtain a target model (for example, the target model may be a neural network used for object recognition, object classification, picture segmentation, or picture encoding and reconstruction, etc.). It should be noted that the source of the training data is not limited in this embodiment of the present application. For example, the training data may be obtained from a cloud or another location to perform model training.

[0324] The target model obtained through training by training engine 25 may be applied to coding systems 10 and 40, such as to source device 12 (e.g., encoder 20) or destination device 14 (e.g., decoder 30) shown in FIG. 26. Training engine 25 may obtain the target model through training in the cloud, and coding system 10 downloads the target model from the cloud and uses the target model. Alternatively, training engine 25 may obtain the target model through training in the cloud and use the target model, and coding system 10 obtains the processing results directly from the cloud.

[0325] 26 illustrates source device 12 and destination device 14 as separate devices, an embodiment of the devices may also include both or both functionality, source device 12 or corresponding functionality and destination device 14 or corresponding functionality. In such an embodiment, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, or by separate hardware and / or software, or by any other combination thereof.

[0326] As will be clear to a person skilled in the art based on the description, the presence and (exact) division of functions within the source device 12 and / or destination device 14 as shown in FIG. 26 may vary depending on the actual device and application.

[0327] Encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., decoder 30), or both encoder 20 and decoder 30, may be implemented via processing circuitry as shown in FIG. 27, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated to video coding, or any combination thereof. Encoder 20 may be implemented via processing circuitry 46 to embody various modules as discussed with respect to encoder 20 and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented via processing circuitry 46 to embody various modules as discussed with respect to decoder 30 and / or any other decoder system or subsystem described herein. The processing circuitry may be configured to perform various operations as discussed below. When the technique is partially implemented in software, as shown in FIG. 29, the device may store instructions for the software in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the technique of the present invention. Either the video encoder 20 and the video decoder 30 may also be integrated as part of a combined encoder / decoder (encoder / decoder, codec) in a single device, as shown in FIG. 27, for example.

[0328] Source device 12 and destination device 14 may include any of a wide variety of devices, including any type of handheld or fixed device, e.g., a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or content distribution server), a broadcast receiver device, a broadcast transmitter device, and the like, and may use no operating system or any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0329] In some cases, the video coding system 10 shown in FIG. 26 is merely an example, and the techniques of the present application may be applied to video coding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between the encoding device and the decoding device. In other examples, data is retrieved from a local memory or streamed over a network. A video encoding device may encode data and store the encoded data in a memory, and / or a video decoding device may retrieve data from a memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other, but simply encode data to a memory and / or retrieve data from a memory and decode it.

[0330] 27 is an illustration of an example of a video coding system 40 including an encoder 20 and / or a decoder 30, according to an example embodiment. The video coding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by processing circuitry 46), an antenna 42, one or more processors 43, one or more memories 44, and / or a display device 45.

[0331] 27, imaging device 41, antenna 42, processing circuitry 46, video encoder 20, video decoder 30, processor 43, memory 44, and / or display device 45 may be in communication with one another. Video coding system 40 may include only video encoder 20 or only video decoder 30 in different examples.

[0332] In some examples, the antenna 42 may be configured to transmit or receive an encoded bitstream of video data. Additionally, in some examples, the display device 45 may be configured to present the video data. The processing circuitry 46 may include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. The video coding system 40 may also include an optional processor 43. The optional processor 43 may likewise include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. In addition, the memory 44 may be any type of memory, for example, a volatile memory (e.g., a static random access memory (SRAM) or a dynamic random access memory (DRAM)), or a non-volatile memory (e.g., a flash memory). In a non-limiting example, the memory 44 may be implemented by a cache memory. In other examples, the logic circuitry 47 and / or the processing circuitry 46 may include a memory (e.g., a cache) for implementing a picture buffer.

[0333] In some examples, video encoder 20 implemented using logic circuitry may include a picture buffer (e.g., implemented by processing circuitry 46 or memory 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the picture buffer. The graphics processing unit may include video encoder 20 implemented via logic circuitry 46 to embody various functional modules described herein. The logic circuitry may be configured to perform various operations described herein.

[0334] In some examples, the decoder 30 may be implemented by logic circuitry 46 in a similar manner to implement the various modules described with reference to the decoder 30 and / or any other decoder system or subsystem described herein. In some examples, the decoder 30 implemented by using logic circuitry may include a picture buffer (implemented by the processing circuitry 44 or memory 44) and a graphics processing unit (e.g., implemented by the processing circuitry 46). The graphics processing unit may be communicatively coupled to the picture buffer. The graphics processing unit may include the video encoder 30 implemented via logic circuitry 46 to embody the various modules described herein.

[0335] In some examples, antenna 42 may be configured to receive an encoded bitstream of video data. As described, the encoded bitstream may include data related to the video frame coding described herein, indicators, index values, mode selection data, etc., such as data related to a coding partition (e.g., transform coefficients or quantized transform coefficients, optional indicators (as described), and / or data defining a coding partition). Video coding system 40 may further include a decoder 30 coupled to antenna 42 and configured to decode the encoded bitstream. Display device 45 is configured to present the video frames.

[0336] In this embodiment of the application, with respect to the examples described with reference to the encoder 20, it should be understood that the decoder 30 may be configured to perform the reverse process. With respect to the signaling syntax elements, the decoder 30 may be configured to receive and parse such syntax elements and decode the associated video data accordingly. In some examples, the encoder 20 may entropy encode the syntax elements into an encoded video bitstream. In such examples, the video decoder 30 may parse such syntax elements and decode the associated video data accordingly.

[0337] FIG. 28 is a schematic diagram of a video coding device 400 according to one embodiment of the present disclosure. The video coding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video coding device 400 may be a decoder such as the video decoder 30 of FIG. 26 or an encoder such as the video encoder 20 of FIG. 26. As mentioned above, when referring to a picture (video) encoder and a decoder in this specification, it is not necessarily an encoder and a decoder that provide encoding decoding of picture data including reconstruction for a human viewer. Rather, the encoder may provide a bitstream and the decoder may decode picture data or picture feature data to be used in a computer vision task.

[0338] Video coding device 400 includes an ingress port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data, a processor, logic unit, or central processing unit (CPU) 430 for processing the data, a transmitter unit (Tx) 440 for transmitting the data, and an egress port 450 (or output port 450) for transmitting the data. Video coding device 400 may also include optical-to-electrical (OE) and electrical-to-optical (EO) components coupled to the ingress port 410, the receiver unit 420, the transmitter unit 440, and the egress port 450 for the entry and exit of optical or electrical signals.

[0339] The processor 430 is implemented by hardware and software. The processor 430 may be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the ingress port 410, the receiver unit 420, the transmitter unit 440, the egress port 450, and the memory 460. The processor 430 includes a coding module 470 (e.g., a neural network NN-based coding module 470). The coding module 470 implements the disclosed embodiments described above. For example, the coding module 470 implements, processes, prepares, or provides various coding operations. Thus, the inclusion of the encoding / decoding module 470 provides substantial improvements to the functionality of the video coding device 400 and affects the switching of the video coding device 400 to different states. Alternatively, the coding module 470 is implemented as instructions stored in the memory 460 and executed by the processor 430.

[0340] Memory 460 may include one or more disks, tape drives, and solid state drives, and may be used as an overflow data storage device to store programs when such programs are selected for execution and to store instructions and data read during program execution. Memory 460 may be volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0341] FIG. 29 is a simplified block diagram of an apparatus 500 that may be used as either or both of source device 12 and destination device 14 in FIG. 26, according to an example embodiment.

[0342] The processor 502 in the device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices capable of manipulating or processing information now existing or later developed. Although the disclosed implementations may be implemented with a single processor as shown, e.g., processor 502, advantages in speed and efficiency may be achieved using two or more processors.

[0343] The memory 504 in the apparatus 500 may be a read-only memory (ROM) or random access memory (RAM) device in one implementation. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 that is accessed by the processor 502 using a bus 512. The memory 504 may further include an operating system 508 and application programs 510, which include at least one program that enables the processor 502 to perform the methods described herein. For example, the application programs 510 may include applications 1 through N, which further include a video coding application that performs the methods described herein.

[0344] The apparatus 500 may also include one or more output devices, such as a display 518. The display 518, in one example, may be a touch-sensitive display that combines a display with a touch-sensing element operable to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0345] Although shown here as a single bus, bus 512 of device 500 may be comprised of multiple buses. Additionally, secondary storage 514 may be directly coupled to the other components of device 500 or may be accessed over a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, device 500 may be implemented in a wide variety of configurations.

[0346] The embodiments of the present application relate to the application of neural networks. For ease of understanding, the following will first explain some nouns or terms used in the embodiments of the present application. The nouns or terms are also used as part of the content of the present invention.

[0347] (1) Neural Networks A neural network (NN) is a machine learning model. A neural network may include neurons. A neuron is a set of x s and an intercept of 1 as input, the output of the computation unit may be:

[0348]

number

[0349] where s=1, 2, ..., or n, where n is a natural number greater than 1, and W s x s where σ is the weight of the neuron and b is the bias of the neuron. f is the activation function of the neuron, and the activation function is used to introduce nonlinear features into the neural network to convert the input signal in the neuron into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by concatenating many single neurons. Specifically, the output of a neuron can be the input of another neuron. The input of each neuron can be connected to a local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be a region containing several neurons.

[0350] (2) Deep Neural Networks A deep neural network (DNN), also called a multi-layer neural network, can be understood as a neural network with many hidden layers. The "many" here has no special metric. DNNs are divided based on the location of different layers, and the neural network in a DNN can be divided into three types: input layer, hidden layer, and output layer. In general, the first layer is the input layer, the last layer is the output layer, and the middle layer is the hidden layer. The layers are fully connected. Specifically, any neuron in the i-th layer is surely connected to any neuron in the (i+1)-th layer. Although DNNs seem complicated, DNNs are actually not complicated in terms of the work of each layer, and can be expressed by the following linear relationship:

[0351]

number

[0352] This is simply expressed as:

[0353]

number

[0354] is the input vector,

[0355]

number

[0356] is the output vector,

[0357]

number

[0358] is the bias vector, W is the weight matrix (also called coefficients), and α() is the activation function.

[0359]

number

[0360] is the input vector

[0361]

number

[0362] Since there are many layers in a DNN, there are many coefficients W and bias vectors

[0363]

number

[0364] There are also. The definition of these parameters in the DNN is as follows. The coefficient W is used as an example. In a three-layer DNN, the linear coefficient from the fourth neuron in the second layer to the second neuron in the third layer is

[0365]

number

[0366] The superscript 3 denotes the layer in which the coefficient W is located, and the subscripts correspond to the index 2 of the third layer for the output and the index 4 of the second layer for the input. In conclusion, the coefficients from the kth neuron in the (L-1)th layer to the jth neuron in the Lth layer are

[0367]

number

[0368] It should be noted that there is no parameter W in the input layer. In deep neural networks, more hidden layers allow the network to describe more complex cases in the real world. Theoretically, a model with more parameters indicates higher complexity and larger "capacity", indicating that the model can complete more complex learning tasks. Training a deep neural network is the process of learning weight matrices, and the final goal of training is to obtain the weight matrices of all layers of the trained deep neural network (weight matrices formed by the vector W in many layers).

[0369] (3) Convolutional Neural Networks A convolutional neural network (CNN) is a deep neural network with a convolutional structure and is a deep learning architecture. In deep learning architectures, multi-layer learning is performed at various levels of abstraction using machine learning algorithms. As a deep learning architecture, a CNN is a feed-forward artificial neural network. Neurons in a feed-forward artificial neural network may respond to pictures input to the neurons. A convolutional neural network includes a feature extractor composed of a convolutional layer and a pooling layer. The feature extractor may be considered as a filter. The convolution process may be considered as using a trainable filter to perform convolution on an input image or a convolutional feature plane (feature map).

[0370] A convolution layer is a neuron layer in a convolutional neural network where a convolution process is performed on an input signal. The convolution layer 221 may include multiple convolution operators. A convolution operator is also called a kernel. In image processing, a convolution operator acts as a filter to extract specific information from an input image matrix. A convolution operator may be essentially a weight matrix, which is usually predefined. In the process of performing a convolution operation on an image, the weight matrix usually processes pixels at a granularity level of one pixel (or two pixels depending on the value of stride) in the horizontal direction on the input image to extract specific features from the image. The size of the weight matrix should be related to the size of the picture. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input picture. During the convolution operation, the weight matrix spans the entire depth of the input picture. Thus, a single depth dimension convolution output is generated through convolution with a single weight matrix. However, in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row x column), i.e., multiple matrices of the same type, are applied. The outputs of the weight matrices are stacked to form the depth dimension of the convolution picture. The dimension here can be understood as being determined based on the above "multiple". Different weight matrices can be used to extract different features from the picture. For example, one weight matrix is ​​used to extract edge information of the picture, another weight matrix is ​​used to extract a specific color of the picture, and a further weight matrix is ​​used to blur unwanted noise in the picture. The size (row x column) of the multiple weight matrices is the same. The size of the feature maps extracted from the multiple weight matrices with the same size is also the same, and the extracted multiple feature maps with the same size are combined to form the output of the convolution operation. The weight values ​​in these weight matrices need to be obtained through large-scale training in real applications.Each weight matrix formed by using the weight values ​​obtained through training can be used to extract information from the input image to enable the convolutional neural network to make a correct prediction. When the convolutional neural network has multiple convolutional layers, a relatively large amount of general features are usually extracted in the initial convolutional layers. The general features are sometimes called low-level features. As the depth of the convolutional neural network increases, the features extracted in the subsequent convolutional layers, such as high-level semantic features, become more complex. Features with higher levels of semantics are more applicable to the problem to be solved.

[0371] Often, the amount of training parameters needs to be reduced. Therefore, often, a pooling layer needs to be introduced periodically after a convolution layer. A convolution layer may be followed by a pooling layer, or multiple convolution layers may be followed by one or more pooling layers. During picture processing, the pooling layer is used only to reduce the spatial size of the picture. The pooling layer may include an average pooling operator and / or a maximum pooling operator to sample the input picture to obtain a picture with a relatively small size. The average pooling operator may be used to calculate pixel values ​​in a certain range in the picture to generate an average value. The average value is used as the average pooling result. The maximum pooling operator may be used to select a pixel with a maximum value in a certain range as the maximum pooling result. In addition, just as the size of the weight matrix in the convolution layer needs to be related to the size of the picture, the operator in the pooling layer also needs to be related to the size of the picture. The size of the processed picture output from the pooling layer may be smaller than the size of the picture input to the pooling layer. Each pixel in the picture output from the pooling layer represents the average or maximum value of the corresponding sub-region of the picture input to the pooling layer.

[0372] After the processing performed in the convolutional layer / pooling layer, as described above, the convolutional neural network is not ready to output the required output information, since in the convolutional layer / pooling layer, only features are extracted and the resulting parameters are reduced from the input image. However, in order to generate the final output information (required class information or other related information), the convolutional neural network needs to use neural network layers to generate the output of one required class or a group of required classes. Therefore, the convolutional neural network layer may include multiple hidden layers. The parameters included in the multiple hidden layers may be obtained through pre-training based on related training data of a specific task type. For example, the task type may include image recognition, image classification, and super-resolution image reconstruction.

[0373] Optionally, in the neural network layer, multiple hidden layers are followed by an output layer of the entire convolutional neural network. The output layer has a loss function similar to multi-class cross entropy, which is used to calculate prediction error in particular. After the forward propagation of the entire convolutional neural network is completed, the backward propagation is started to update the weight values ​​and deviations of each of the above-mentioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network and the ideal result by using the output layer.

[0374] (4) Recurrent Neural Networks Recurrent neural networks (RNNs) are used to process sequence data. In traditional neural network models, from the input layer to the hidden layer and then the output layer, the layers are fully connected, and the nodes in each layer are not connected. Such general neural networks solve many difficult problems, but still cannot solve many other problems. For example, when a word in a sentence is to be predicted, the adjacent words in the sentence are not independent, so the previous word usually needs to be used. The reason why RNNs are called recurrent neural networks is because the current output of a sequence is also related to the previous output of the sequence. The specific representation form is that the network remembers previous information and applies the previous information to the calculation of the current output. Specifically, the nodes in the hidden layer are connected, and the input of the hidden layer not only includes the output of the input layer, but also the output of the hidden layer at the previous moment. In theory, RNNs can process sequence data of any length. The training of RNNs is the same as the training of traditional CNNs or DNNs. The backpropagation algorithm is also used, but there is a difference: when the RNN is extended, parameters such as W of the RNN are shared. This is different from the traditional neural network described in the previous example. In addition, during the gradient descent algorithm, the output at each step depends not only on the network at the current step, but also on the network state at several previous steps. The learning algorithm is called the backward propagation through time (BPTT) algorithm.

[0375] Now that convolutional neural networks exist, why do we need recurrent neural networks? The reason is simple. In convolutional neural networks, it is assumed that the elements are independent of each other, and the input and output are also independent, like cats and dogs. However, in the real world, many elements are interrelated. For example, stock prices change over time. In another example, a person says, I like to travel, and my favorite place is Yunnan. I will go if I have the opportunity. Here, people should know that the person is going to "Yunnan". The reason is that people can infer the answer based on the content of the context. However, how can a machine know this? Enter RNNs. RNNs are intended to allow machines to remember like humans. Therefore, the output of RNNs needs to depend on the current input information and the information stored in the past.

[0376] (5) Recursive residual convolutional neural network (RR-CNN)

[0377] (5) Artificial neural network (ANN)

[0378] (6) Loss function In the process of training a deep neural network, since it is expected that the output of the deep neural network is as close as possible to the actually expected predicted value, the predicted value of the current network can be compared with the actually expected target value, and then the weight vector of each layer of the neural network is updated based on the difference between the predicted value and the target value (indeed, there is usually an initialization process before the first update, specifically, parameters are pre-configured for all layers of the deep neural network). For example, if the predicted value of the network is large, the weight vector is adjusted to reduce the predicted value, and the adjustment is continuously made until the deep neural network can predict the actually expected target value or a value very close to the actually expected target value. Therefore, "how to obtain the difference between the predicted value and the target value by comparison" needs to be defined in advance. This is the loss function or objective function. The loss function and the objective function are important formulas used to measure the difference between the predicted value and the target value. The loss function is used as an example. A higher output value (loss) of the loss function indicates a larger difference. Training a deep neural network is therefore the process of minimizing the loss as much as possible.

[0379] (7) Backpropagation Algorithm The convolutional neural network may correct the parameter values ​​in the initial super-resolution model in the training process according to the error backpropagation (BP) algorithm so that the error loss in reconstructing the super-resolution model is smaller. Specifically, the input signal is forward forwarded until an error loss occurs at the output, and the parameters in the initial super-resolution model are updated based on the backpropagation error loss information to converge the error loss. The backpropagation algorithm is an error loss-centric backpropagation movement intended to obtain parameters such as the weight matrix of the optimal super-resolution model.

[0380] (8) Generative Adversarial Networks Generative adversarial networks (GAN) are deep learning models. The model includes at least two models: a generative model and a discriminative model. The two models learn from each other to generate better outputs. Both the generative model and the discriminative model can be neural networks, specifically, deep neural networks or convolutional neural networks. The basic principle of GAN is as follows. Using GAN to generate pictures as an example, assume that there are two networks, G (generator) and D (discriminator). G is a network for generating pictures. G receives random noise z and generates pictures by using the noise, where the picture is denoted as G(z). D is a discriminative network used to determine whether the picture is "real". The input parameters of D are x, where x represents a picture, and the output D(x) represents the probability that x is a real picture. When the value of D(x) is 1, it indicates that the picture is 100% real. When the value of D(x) is 0, it indicates that the picture cannot be real. In the process of training a generative adversarial network, the purpose of the generative network G is to generate a picture that is as real as possible to deceive the discriminant network D, and the purpose of the discriminant network D is to distinguish the picture generated by G from the real picture as much as possible. In this way, there is a dynamic "gaming" process between G and D, specifically the "adversary" in the "generative adversarial network". The final gaming result is that in an ideal state, G can generate a picture G(z) that is difficult to distinguish from the real picture, and D is difficult to judge whether the picture generated by G is real, specifically D(G(z))=0.5. In this way, a good generative model G can be obtained and used to generate pictures.

[0381] Those skilled in the art can appreciate that the functions described with reference to the various exemplary logical blocks, modules, and algorithm steps disclosed and described herein may be implemented by hardware, software, firmware, or any combination thereof. When implemented by software, the functions described with reference to the exemplary logical blocks, modules, and steps may be recorded in or transmitted via a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or may include any communication medium that facilitates the transmission of a computer program from one place to another (following a communication protocol). Thus, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in this application. A computer program product may include a computer-readable medium.

[0382] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other compact disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer. In addition, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or another remote source over a coaxial cable, optical fiber, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, radio, or microwave, the coaxial cable, optical fiber, twisted pair, DSL, or wireless technology such as infrared, radio, or microwave, is included within the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals, or other transitory media, and in fact refer to non-transitory tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), and Blu-ray disc. Disks usually reproduce data magnetically, while discs reproduce data optically by using a laser. Combinations of the foregoing items should also be included within the scope of computer-readable media.

[0383] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor" as used herein may refer to any of the foregoing structures, or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described with reference to the exemplary logic blocks, modules, and steps described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or may be incorporated into a composite codec. In addition, the techniques may all be implemented within one or more circuits or logic elements.

[0384] The technology in this application may be implemented in a variety of apparatuses or devices, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). To highlight functional aspects of an apparatus configured to implement the disclosed technology, various components, modules, or units are described herein, but are not necessarily implemented by different hardware units. In practice, as described above, the various units may be combined into a codec hardware unit in combination with appropriate software and / or firmware, or may be provided by interoperable hardware units (including one or more processors as described above).

[0385] The above description is merely an example of a specific implementation of the present application, and is not intended to limit the scope of protection of the present application. Any variation or replacement that is easily conceived by a person skilled in the art within the technical scope disclosed in the present application shall fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the scope of protection of the claims.

[0386] In summary, the present disclosure provides a method and apparatus for processing picture data or picture feature data using a neural network having two or more layers. The present disclosure may be applied in the field of artificial intelligence (AI)-based video or picture compression technology, in particular in the field of neural network-based video compression technology. According to some embodiments, a location in the neural network where auxiliary information may be input for processing is selectable based on a collection condition. The collection condition may evaluate whether some prerequisites are met. Some of the advantages may include better performance in terms of rate and / or disclosure due to the effect of increased flexibility in neural network configuration. [Explanation of symbols]

[0387] 1 Encoder, Processing Unit, Unit 2 Quantizer, Unit 3 Units, HyperEncoder 4 Decoder, unit 5 Arithmetic Encoder, AE 6 Arithmetic Decoder AD, AD 7 units, Hyper Decoder 8 Quantizer 9 Arithmetic Encoder, AE 10 AD, video coding system, coding system, system 12 Source Device 13 Encoded picture data, communication channel, encoded data 14 Destination Device 16 Picture Source 17 Picture or picture data, raw picture or raw picture data 18 Preprocessor, preprocessing unit, picture preprocessor 19 Preprocessed Picture, Preprocessed Picture Data 20 Video Encoder, Encoder 21 Encoded picture data 22 Communication interface or communication unit, communication interface 28 Communication interface or communication unit, communication interface 30 Video Decoder, Decoder 32 Post-processor, post-processing unit 34 Display Devices 40 Video Coding System 41 Imaging Device 42 Antenna 43 Processors 44 Memory 45 Display Devices 46 Processing Circuit 47 Logic Circuits 57-59 Upsampling Layer 57~512 layers 221 Convolutional Layer 400 Video Coding Device 410 upsampling layer, inlet port, input port 411 Upsampling Layer 420 layers, receiver unit 430 Processor, Logic Unit, or Central Processing Unit 440 Transmitter Unit 450 Exit Port 460 Memory 470 coding module, neural network (NN)-based coding module, encoding / decoding module 500 devices 502 processor 504 Memory 506 Code and Data 508 Operating Systems 510 Application Program 510~512 upsampling layer 512 Bus 514 Secondary Storage 518 Display 600A Encoder Neural Network, Encoder Network, Encoder Part 600B Decoder part, decoder network 602 Input Picture Channel 610 Collection, Collector 620 bitstream 630 Bitstream 692 Reconstructed Pictures 710 Predicted Frames 720 frames 730 Example, Mosaic Picture 740 Channels 900A Neural Network 900B Neural Network 902 Pictures 910_1 Collection conditions, conditions 910_2 Collection conditions, conditions 910_3 Collection conditions, conditions 920_1 Collection unit 920_2 Collection unit 920_3 Collection unit 940 bitstream 945 Bitstream 950 Auxiliary Data 960_1 Collection, collection unit 960_2 Collection, collection unit 960_3 Collection, collection unit 1002 Input Picture 1010 downsampling layers 1010A Encoder 1010B Decoder 1030 Collection Unit 1050 Collection Conditions 1060 Decoder 1070 upsampling layer 1092 Processed Pictures 1100A Encoder part 1100B Decoder part, decoder 1110 Prediction signal (predi) 1120 Residual (resii) 1140 Bitstream 1150 Collection unit, collection stage 1160 Residual 1310 Collection Unit 1320 Collection Conditions 1400A Neural Network, Encoder Side Network, Network 1400B Decoding side network, network 1402 Input Picture 1450 bitstream 1492 Reconstructed Pictures 1506 Top, Network 1510 Motion Estimation 1520 Motion Compensation 1530 Motion Compensation 1560 parts, collection units 1572 bitstream 1586 encoding and decoding network, network 1590 Bitstream 1601 Input Picture 1605 First layer or block or layer, first layer 1606 Network 1660 Collector Unit 1665 Collected 1672 Residual Bitstream 1690 Bitstream 1701 Input picture data 1702 Processing 1706 Neural Networks 1760 First Data 1765 Second Data, Prediction 1772 Bitstream 1786 Neural Networks 1790 Bitstream 2010 layer 2012 layer 2020 tier 2022 layers 2050 bitstream

Claims

1. 1. A method for processing picture feature data of a picture from a bitstream (940) using a neural network (900B) comprising a plurality of neural network layers, the method comprising: obtaining the picture feature data from the bitstream; processing said picture feature data using said neural network (900B); and for each of one or more preconfigured positions in the neural network, the processing step comprises: determining whether to collect (960_1, 960_2, 960_3) auxiliary data (950) for processing by one of the plurality of neural network layers at the preconfigured location based on collection conditions (910_1, 910_2, 910_3); If it is determined that the auxiliary data should be collected (960_1, 960_2, 960_3), the processing by the layer at the preconfigured location is based on the auxiliary data. method.

2. 2. The method of claim 1, wherein applying the collection conditions (910_1, 910_2, 910_3) in the determination results in the auxiliary data (950) being collected (960_1, 960_2, 960_3) for a single one of the one or more preconfigured locations.

3. 2. The method of claim 1, wherein applying the collection conditions (910_1, 910_2, 910_3) in the determination results in the auxiliary data (950) being collected (960_1, 960_2, 960_3) for two or more of the preconfigured locations.

4. two or more of the preconfigured locations are present; the auxiliary data (950) is scalable in size to match a dimension of an input channel processed by the layer in two or more of the preconfigured positions; As a result of applying the collection conditions (910_1, 910_2, 910_3) in the determination, the auxiliary data is (i) collected (960_1, 960_2, 960_3); or (ii) collected (960_1, 960_2, 960_3) and scaled to a single one of the preconfigured locations; 4. The method according to any one of claims 1 to 3.

5. The method according to claim 1 , wherein the collection conditions (910_1, 910_2, 910_3) are based on picture characteristics or picture feature data characteristics obtained from the bitstream (940).

6. the picture characteristics or the picture feature data characteristics include a resolution; the collection conditions (910_1, 910_2, 910_3) include a comparison of the resolution to a preconfigured resolution threshold; The method of claim 5.

7. the picture is a video picture, and the picture characteristics include a picture type; the collection conditions (910_1, 910_2, 910_3) include determining whether the picture type is a temporally predicted picture type or a spatially predicted picture type; The method according to claim 5 or 6.

8. obtaining from the bitstream an indication specifying whether to collect (960_1, 960_2, 960_3) the auxiliary data (950) for the one or more preconfigured locations; the collection conditions (910_1, 910_2, 910_3) for each of the one or more pre-configured locations are as follows: If the instructions specify that the auxiliary data (950) should be collected for the preconfigured location, the determination is in the affirmative; if the instructions specify that the auxiliary data (950) should not be collected for the preconfigured location, the determination is negative; 5. The method according to any one of claims 1 to 4.

9. 9. The method of claim 1, wherein the auxiliary data (950) provides information about the picture feature data that is processed by the neural network to generate an output.

10. 10. The method of claim 1, wherein the auxiliary data comprises predictive data that is a prediction of the picture or of the picture feature data after processing by one or more of the layers of the neural network.

11. The method of claim 10 , wherein the auxiliary data is a combined pair of the predicted data (1110) and supplemental data (1120) that is combined with the predicted data.

12. 12. The method of claim 11, wherein the prediction data and the supplemental data have dimensions of data processed by layers at different locations within the neural network.

13. the neural network (900B) includes a sub-network (1308) for lossless decoding having at least one layer; Second auxiliary data (1350) is input (1310) to the sub-network for lossless decoding; 10. The method according to any one of claims 1 to 9.

14. 14. The method of claim 1, wherein the neural network (900B) is trained to perform at least one of still picture decoding, video picture decoding, still picture filtering, video picture filtering, and machine vision processing including object detection, object recognition, or object classification.

15. A method performed on each of a plurality of auxiliary data (950, 1350) including first auxiliary data (950) and second auxiliary data (1350), comprising: the first auxiliary data (950) is associated with a first set of one or more preconfigured locations, and the second auxiliary data (1350) is associated with a second set of one or more preconfigured locations; 15. The method according to any one of claims 1 to 14.

16. 16. The method of claim 15, wherein the first set of one or more preconfigured locations and the second set of one or more preconfigured locations share at least one preconfigured location.

17. said neural network (900B) being trained to process video pictures; determining whether to collect auxiliary data for processing by layers at the preconfigured locations occurs for every predetermined number of video pictures, the predetermined number of video pictures being one or more; 17. The method according to any one of claims 1 to 16.

18. 1. A method for processing a picture (902) using a neural network (900A) comprising a plurality of neural network layers to generate a bitstream (940), the method comprising: Processing the picture using the neural network (900A), the processing comprising: for each of one or more preconfigured positions in the neural network: determining whether to collect (920_1, 920_2, 920_3) auxiliary data (950) for processing by layer at the preconfigured positions based on collection conditions (910_1, 910_2, 910_3); If it is determined that the auxiliary data (950) should be collected (920_1, 920_2, 920_3), the processing by the layer at the preconfigured location is based on the auxiliary data (950). Steps and inserting data (940) obtained by processing said pictures by said neural network (900A) into said bitstream; Including, method.

19. 20. The method of claim 18, wherein applying the collection conditions (910_1, 910_2, 910_3) in the determination results in the auxiliary data (950) being collected (920_1, 920_2, 920_3) for a single one of the one or more preconfigured locations.

20. 20. The method of claim 18, wherein applying the collection conditions (910_1, 910_2, 910_3) in the determination results in the auxiliary data being collected (920_1, 920_2, 920_3) for two or more of the preconfigured locations.

21. two or more of the preconfigured locations are present; the auxiliary data is scalable in size to match a dimension of an input channel processed by the layer in two or more of the preconfigured locations; As a result of applying the collection conditions (910_1, 910_2, 910_3) in the determination, the auxiliary data (950) is (i) collected (920_1, 920_2, 920_3); or (ii) collected (920_1, 920_2, 920_3) and scaled for a single one of the preconfigured locations; 21. The method according to any one of claims 18 to 20.

22. 22. The method according to claim 18, wherein the collection conditions (910_1, 910_2, 910_3) are based on picture characteristics or picture feature data characteristics included in the bitstream (940).

23. the picture characteristics or the picture feature data characteristics include a resolution; the collection conditions (910_1, 910_2, 910_3) include a comparison of the resolution to a preconfigured resolution threshold; 23. The method of claim 22.

24. the picture is a video picture, and the picture characteristics include a picture type; the collection conditions (910_1, 910_2, 910_3) include determining whether the picture type is a temporally predicted picture type or a spatially predicted picture type; 24. The method of claim 22 or 23.

25. generating an indication specifying whether to collect the auxiliary data (950) for the one or more preconfigured locations; including said indication in said bitstream (940); Further comprising:

22. The method according to any one of claims 18 to 21.

26. 26. The method of claim 25, further comprising: selecting, for the one or more preconfigured positions, whether to collect (920_1, 920_2, 920_3) the auxiliary data (950) based on optimizing a cost function including at least one of rate, distortion, accuracy, or complexity.

27. 27. The method of any one of claims 18 to 26, wherein the auxiliary data (950) provides information about picture feature data that is processed by the neural network (900A) to generate an output.

28. 28. The method of claim 18, wherein the auxiliary data (950) comprises prediction data (1110) that is a prediction of the picture or a prediction of picture feature data after processing by one or more of the layers of the neural network.

29. 30. The method of claim 28, wherein the auxiliary data is a combined pair of the predicted data (1110) and supplemental data (1120) that is combined with the predicted data.

30. 30. The method of claim 29, wherein the prediction data and the supplemental data have dimensions of data processed by layers at different locations within the neural network.

31. the neural network (900A) includes a sub-network (1308) for lossless decoding having at least one layer; second auxiliary data (1350) is input to the sub-network (1308) for lossless decoding; 28. The method according to any one of claims 18 to 27.

32. 32. The method of claim 18, wherein the neural network (900A) is trained to perform at least one of still picture encoding, video picture encoding, still picture filtering, video picture filtering, and machine vision processing including object detection, object recognition, or object classification.

33. A method performed on each of a plurality of auxiliary data including first auxiliary data (950) and second auxiliary data (1350), the method comprising: the first auxiliary data (950) is associated with a first set of one or more preconfigured locations, and the second auxiliary data (1350) is associated with a second set of one or more preconfigured locations; 33. The method of any one of claims 18 to 32.

34. 34. The method of claim 33, wherein the first set of one or more preconfigured locations and the second set of one or more preconfigured locations share at least one preconfigured location.

35. said neural network (900A) being trained to process video pictures (902); determining whether to collect (920_1, 920_2, 920_3) auxiliary data (950) for processing by layers at the preconfigured locations is performed for every predetermined number of video pictures (902), the predetermined number of video pictures being one or more; 35. The method of any one of claims 18 to 34.

36. A computer program (510) stored on a computer-readable non-transitory medium (504) and comprising code instructions that, when executed on one or more processors (502), cause the one or more processors (502) to perform the method of any one of claims 1 to 35.

37. 1. An apparatus (400) for processing picture feature data from a bitstream (940) using a neural network (900B) comprising a plurality of neural network layers, the apparatus (400) comprising: Obtaining the picture feature data from the bitstream (940); Processing the picture feature data using the neural network (900B, 470) and for each of one or more preconfigured positions in the neural network, the processing comprises: determining whether to collect (960_1, 960_2, 960_3) auxiliary data (950) for processing by one of the plurality of neural network layers at the preconfigured location based on collection conditions (910_1, 910_2, 910_3); If it is determined that the auxiliary data (950) should be collected (960_1, 960_2, 960_3), the processing by the layer at the preconfigured location is based on the auxiliary data (950). Equipment (400).

38. 1. An apparatus for processing a picture (902) using a neural network (900A) comprising a plurality of neural network layers to generate a bitstream (940), the apparatus comprising: processing the picture with the neural network, the processing comprising: for each of one or more preconfigured positions in the neural network: determining whether to collect (920_1, 920_2, 920_3) auxiliary data (950) for processing by layer at the preconfigured locations based on collection conditions (910_1, 910_2, 910_3); If it is determined that the auxiliary data (950) should be collected (920_1, 920_2, 920_3), the processing by the layer at the preconfigured location is based on the auxiliary data (950). And, inserting data obtained by processing said picture (902) by said neural network into said bitstream (940); a processing circuit configured to: Device.

Images