Video encoding and decoding using motion constraints - Patents.com

Abstract

At least a method and apparatus are presented for encoding or decoding video conforming to a capacity constraint including an indication regarding a number of multiply-and-accumulate operations required to decode the video stream. For example, a video stream is defined that includes coded data representing a sequence of images, an indication of a profile to which the coded data conforms from a plurality of profiles, and an indication of a level to which the coded data conforms from a plurality of levels, the plurality of profiles and the plurality of levels specifying a capacity constraint for decoding the coded data, the capacity constraint including an indication regarding a number of multiply-and-accumulate operations required to decode the coded data.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of European Patent Application No. 22305528.6, filed April 12, 2022, which is incorporated by reference in its entirety.

[0002] FIELD OF THEINVENTION At least one of the present embodiments relates generally to a method or apparatus for video encoding or decoding, and more particularly to a method or apparatus for implementing a capacity constraint that includes an indication regarding the number of multiply-add operations required to decode a video stream. [Background technology]

[0003] To achieve high compression efficiency, image and video coding schemes usually employ prediction, including motion vector prediction, and transformation to exploit spatial and temporal redundancy in the video content. In general, intra- or inter-prediction is used to exploit correlation within or between frames, whereby the difference between an original image and a predicted image, often referred to as a prediction error or prediction residual, is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by an inverse process corresponding to entropy coding, quantization, transformation, and prediction.

[0004] Recently developed methods for encoding and decoding (e.g., neural network (NN)-based methods) can exhibit a huge number of multiply-accumulate (MAC) operations, resulting in a much larger number of MACs than are required in conventional video encoding solutions, making decoder implementations increasingly difficult. Thus, there is a need to improve this situation. Summary of the Invention

[0005] In this application, the disclosed concept is based on the number of MAC operations used as the complexity metric, which can be generalized to other complexity metrics such as the number of operations (OPs), the number of floating-point operations (FLOPs), the number of uses per sample, etc.

[0006] SUMMARY OF THE DISCLOSURE The general aspects described herein overcome and address the shortcomings and drawbacks of the prior art.

[0007] According to a first aspect, there is provided a video stream including coded data representing a sequence of images, an indication of a profile of a plurality of profiles to which the coded data conforms, and an indication of a level of a plurality of levels to which the coded data conforms, wherein the plurality of profiles and the plurality of levels specify capacity constraints for decoding the coded data, the capacity constraints including an indication regarding a number of multiply-add operations required to decode the coded data.

[0008] According to a second aspect, a method is provided, comprising: determining a capability of a decoder for decoding a video stream, the video stream including coded data representing a sequence of images, an indication of a profile to which the coded data conforms from a plurality of profiles, and an indication of a level to which the coded data conforms from a plurality of levels. The plurality of profiles and levels specify a capability constraint for decoding the conforming coded data, advantageously the capability constraint including an indication on a number of multiply-accumulate operations required to decode the coded data. Determining the capability of the decoder for decoding the video stream further includes parsing the indication of the profile to which the coded data conforms, parsing the indication of the level to which the coded data conforms, and determining that in response to determining that the profile is supported by the decoder and that the level is supported by the decoder, the decoder is capable of performing a number of multiply-accumulate operations required to decode the coded data.

[0009] According to a third aspect, a method is provided, the method including video decoding by first determining a capability of a decoder to decode a video stream according to any of the disclosed variations, and in response to determining that the decoder is capable of decoding the video stream, decoding the video stream.

[0010] According to a fourth aspect, a second method is provided, the method including video encoding by generating a bitstream according to any of the disclosed embodiments and variations of a capacity constraint specifying an indication regarding a number of multiply-accumulate operations required to decode the generated bitstream.

[0011] According to another aspect, an apparatus is provided. The apparatus comprises one or more processors, the one or more processors configured to perform a method for video decoding according to any of its variants. According to another aspect, an apparatus for video decoding comprises means for performing any of the variants of video decoding.

[0012] According to another aspect, another apparatus is provided. The apparatus comprises one or more processors, the one or more processors configured to perform a method for video decoding according to any of the variants. According to another aspect, an apparatus for video decoding comprises means for performing any of the variants of video decoding.

[0013] According to another general aspect of at least one embodiment, there is provided a device including an apparatus according to any of the embodiments relating to decoding and at least one of: (i) an antenna configured to receive a signal, the signal including a video block; (ii) a band limiter configured to limit the received signal to a band of frequencies including the video block; or (iii) a display configured to display an output representing the video block.

[0014] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content generated in accordance with any of the described encoding embodiments or variations.

[0015] According to another general aspect of at least one embodiment, a signal is provided that includes coded data representing an image sequence, an indication of a profile of a plurality of profiles to which the coded data conforms, and an indication of a level of a plurality of levels to which the coded data conforms, where the plurality of profiles and the plurality of levels specify a capacity constraint for decoding the coded data, the capacity constraint including an indication regarding a number of multiply-add operations required to decode the coded data.

[0016] According to another general form of at least one embodiment, a computer program product is provided that includes instructions that, when executed by a computer, cause the computer to perform any of the described encoding / decoding embodiments or variations.

[0017] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which is to be read in conjunction with the accompanying drawings. [Brief description of the drawings]

[0018] In the drawings, some example embodiments are illustrated. [Figure 1] FIG. 1 illustrates a block diagram of an exemplary device in which various aspects of the embodiments may be implemented. [Diagram 2] FIG. 2 shows a block diagram of one embodiment of a video encoder in which various aspects of the embodiments may be implemented. [Diagram 3] FIG. 3 shows a block diagram of one embodiment of a video decoder in which various aspects of the embodiments may be implemented. [Figure 4] FIG. 4 illustrates a block diagram of a method for determining a decoder's capabilities for decoding a video stream in accordance with at least one general aspect of the embodiment. [Diagram 5] FIG. 5 illustrates a block diagram for determining a decoder's capabilities for decoding a video stream in accordance with a general aspect of at least one embodiment. [Figure 6] FIG. 6 illustrates a general decoding method in accordance with a general aspect of at least one embodiment. [Figure 7] FIG. 7 illustrates a general encoding method in accordance with a general aspect of at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Various embodiments relate to a video coding system, and in at least one embodiment, it is proposed to adapt the video coding process to implement new motion constraints in a canonical video decoding process, in particular with respect to the number of operations (MAC) required in the decoding process. In order to increase the coding efficiency and improve the codec consistency when the decoding method can exhibit a huge number of multiply-accumulate (MAC) operations, different embodiments are proposed below that introduce some signaling and compatibility modifications. Among others, an encoding method, a decoding method, an encoding device and a decoding device based on this principle are proposed.

[0020] Moreover, although the present aspects describe principles related to particular drafts of the Versatile Video Coding (VVC) or High Efficiency Video Coding (HEVC) specifications or Enhanced Compression Model (ECM) reference software, they are not limited to VVC or HEVC or ECM and may be applied, for example, to other standards and recommendations, whether existing or developed in the future, and extensions of such standards and recommendations, including VVC, and HEVC, and ECM. Unless otherwise indicated or technically excluded, the aspects described in this application may be used individually or in combination.

[0021] The acronyms used in this specification reflect the current state of video coding development and should therefore be considered as examples of naming that may be renamed at a later stage while still representing the same techniques.

[0022] FIG. 1 illustrates a block diagram of an example of a system in which various aspects and embodiments may be implemented. System 100 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include various electronic devices, such as, but not limited to, personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected appliances, and servers. Elements of system 100 may be embodied, alone or in combination, in a single integrated circuit, multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 100 are distributed across multiple ICs and / or separate components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices, for example, through a communication bus or dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more of the aspects described herein.

[0023] The system 100 includes at least one processor 110 configured to execute instructions loaded therein, for example, to implement various aspects described herein. The processor 110 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 100 includes at least one memory 120 (e.g., volatile and / or non-volatile memory devices). The system 100 includes a storage device 140, which may include non-volatile and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives. The storage device 140 may include, by way of non-limiting example, an internal storage device, an attached storage device, and / or a network-accessible storage device.

[0024] The system 100 includes, for example, an encoder / decoder module 130 configured to process data to provide encoded or decoded video, which may include its own processor and memory. The encoder / decoder module 130 represents a module that may be included within a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. It is noted that the encoder / decoder module 130 may be implemented as a separate element of the system 100 or may be incorporated within the processor 110 as a combination of hardware and software known to those skilled in the art.

[0025] Program code to be loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described herein may be stored in the storage device 140 and then loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of various items during execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing of equations, expressions, operations, and operational logic.

[0026] In some embodiments, memory internal to the processor 110 and / or the encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device can be either the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be the memory 120 and / or the storage device 140, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, a high-speed external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as HEVC, or VVC.

[0027] Input to the elements of system 100 may be provided through various input devices, as indicated at block 105. Such input devices may include, but are not limited to, (i) an RF section that receives, for example, an RF signal transmitted over the air by a broadcast station, (ii) a composite input, (iii) a USB input, and / or (iv) an HDMI input.

[0028] In various embodiments, the input devices of block 105 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band that may be referred to (for example) as a channel in a particular embodiment, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs these various functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near baseband frequency) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements perform frequency selection by receiving, filtering, downconverting, and refiltering RF signals transmitted over a wired (e.g., cable) medium to a desired frequency band. Various embodiments rearrange the order of the above-mentioned (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0029] It should be noted that the USB and / or HDMI terminals may include respective interface processors for connecting the system 100 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or in the processor 110, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, for example, in a separate interface IC or in the processor 110, as desired. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 110 and an encoder / decoder 130 operating in combination with memory and storage elements that process the data streams as desired for presentation on an output device.

[0030] The various elements of system 100 may be provided within an integrated housing in which the various elements may be interconnected and transmit data between each other using suitable connection arrangements 115, e.g., internal buses known in the art, including I2C buses, wiring, and printed circuit boards.

[0031] System 100 includes a communication interface 150 that enables communication with other devices over a communication channel 190. Communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 190. Communication interface 150 may include, but is not limited to, a modem or a network card, and communication channel 190 may be implemented in a wired and / or wireless medium, for example.

[0032] Data is streamed to the system 100 in various embodiments using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signal in these embodiments is received by a communication channel 190 and communication interface 150 adapted for Wi-Fi communication. The communication channel 190 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, a set-top box delivering data through an HDMI connection of the input block 105 is used to provide the streamed data to the system 100. In yet other embodiments, an RF connection of the input block 105 is used to provide the streamed data to the system 100.

[0033] System 100 may provide output signals to various output devices, including display 165, speakers 175, and other peripheral devices 185. Other peripheral devices 185, in various examples of embodiments, include one or more of a standalone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 100. In various embodiments, control signals are communicated between system 100 and display 165, speakers 175, or other peripheral devices 185 using signaling such as AV.Link, CEC, or other communication protocols that allow inter-device control with or without user intervention. Output devices may be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, output devices may be connected to system 100 via communication interface 150 and using communication channel 190. Display 165 and speakers 175 may be integrated into a single unit along with other components of system 100, for example in an electronic device such as a television. In various embodiments, display interface 160 includes a display driver, for example a timing controller (T Con) chip.

[0034] Display 165 and speakers 175 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments in which display 165 and speakers 175 are external components, the output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0035] 2 shows an example of a video encoder 200, such as a Versatile Video Coding (VVC) encoder. FIG 2 may also show an encoder that has improvements to the VVC standard or that employs techniques similar to VVC.

[0036] In this application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "encoded" and "coded" may be used interchangeably, and the terms "image", "picture", and "frame" may be used interchangeably. Usually, but not necessarily, the term "reconstructed" is used on the encoder side, while the term "decoded" is used on the decoder side.

[0037] Before being encoded, a video sequence may undergo pre-encoding processing (201), such as applying a color transformation to the input color picture (e.g., from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata can be associated with the pre-processing and added to the bitstream.

[0038] In the encoder 200, a picture is coded by the encoder elements as described below. The picture to be coded is divided (202) into units, e.g., CUs, and processed. Each unit is coded, e.g., using either intra mode or inter mode. When a unit is coded in intra mode, it performs intra prediction (260). In inter mode, motion estimation (275) and motion compensation (270) are performed. The encoder decides (205) whether to use intra mode or inter mode to code the unit, and indicates the intra / inter decision, e.g., by a prediction mode flag. A prediction residual is calculated (210), e.g., by subtracting the predicted block from the original image block.

[0039] The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is coded directly without applying a transform or quantization process.

[0040] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and the predicted block are combined (255) to reconstruct an image block. An in-loop filter (265) is applied to the reconstructed picture to perform, for example, deblocking / Sample Adaptive Offset (SAO) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (280).

[0041] Figure 3 shows a block diagram of an exemplary video decoder 300. In the decoder 300, the bitstream is decoded by decoder elements as described below. The video decoder 300 generally performs a decoding pass that is the inverse of the encoding pass as described in Figure 2. The encoder 200 also generally performs video decoding as part of encoding the video data.

[0042] In particular, the decoder's input includes a video bitstream, which may be generated by the video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, motion vectors, and other coded information. Picture partition information indicates how the picture is partitioned. The decoder may then partition the picture according to the decoded picture partition information (335). The transform coefficients are inverse quantized (340) and inverse transformed (350) to decode the prediction residual. The decoded prediction residual and the predicted block are combined (355) to reconstruct an image block. A prediction block may be obtained (370) from intra prediction (360) or motion compensated prediction (i.e., inter prediction) (375). An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (380).

[0043] The decoded picture may further undergo post-decoding processing (385), such as an inverse color conversion (e.g., from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping that performs the inverse of the remapping process performed in the pre-encoding processing (201). The post-decoding processing may use metadata derived in the pre-encoding processing and signaled in the bitstream.

[0044] At least some embodiments relate to the specification of new operational constraints on a normative video decoding process. In a common video coding standard, the decoding process is normatively described. In addition, normative constraints on decoder capabilities are defined in profiles, tiers, and levels. A profile defines a set of tools that should be supported by a normative decoder. For each profile, a set of levels is defined, which specify quantitative constraints on the maximum normative decoder capabilities, such as, for example, the maximum luma sample rate (the maximum number of luma samples per second processed and generated by the decoder), the maximum bit rate, the maximum picture resolution, etc. An example of constraints specified in the VVC standard is shown in Table 2 below. A constraint on the maximum luma sample rate MaxLumaSr (samples / second) is included in this table, for example. Successive levels generally correspond to incremental capabilities, and a decoder that complies with a given level (e.g., 5.0) is also compatible with all lower levels (1.0 to 4.1). In addition, the concept of tiers is also specified. A tier is a specified category of level constraints imposed on the values ​​of syntax elements or variables in the bitstream. The level constraints are contained within the tiers, such that a decoder that complies with a certain tier and level can decode all bitstreams that complies with the same tier or lower tiers at that level, or any level below that level.

[0045] These constraints are used on the encoder side to ensure that the encoded bitstream conforms to the profile / level targeted by the encoder. The encoder signals the profile and level information in the bitstream, e.g., in a Video Parameter Set (VPS) or a Sequence Parameter Set (SPS), such as in VVC, where a syntax structure (shown in Table 1, named "profile_tier_level()") is inserted into the VPS and optionally into the SPS (depending on the SPS syntax element "sps_ptl_dpb_hrd_params_present_flag"). The structure profile_tier_level() contains syntax elements indicating the profile and level (e.g., general_profile_idc, general_level_idc).

[0046] [Table 1]

[0047] The constraints are used on the decoder side to determine the decoder's capability to decode a video conforming to the profile / level signaled in the bitstream. If the decoder is not able to decode the video, the client receiving and decoding the video bitstream can request another version of the video (another bitstream) conforming to the profile / level supported by the decoder. Different video decoders (hardware cores of the video decoder, or software implementations of the video decoder) may conform to different profiles and levels. This means that one video decoder may be capable of decoding a bitstream conforming to a first given profile / level, but may not be capable of decoding a bitstream conforming to a second given profile / level.

[0048] [Table 2]

[0049] Recent research work has explored neural network (NN) based decoder designs. This could be in the form of NN based coding tools, e.g., new NN based intra prediction tools. Ultimately, the solution is to replace the full conventional design with a full NN based design (e.g., an autoencoder). The main problem with NN based solutions is that they involve a huge number of multiply-accumulate (MAC) operations, leading to a number of MACs much larger than those required in conventional video coding solutions (even in their most complex versions, such as the exploratory coding model -ECM- currently being developed by JVET). Even in the case of conventional coding solutions, the trend is to make the decoding process more and more complex, potentially making the implementation more and more difficult.

[0050] Considering these new advances, it seems important to integrate the new capacity constraints of the decoder, especially with regard to the number of operations (MAC).

[0051] This is solved and addressed by the general aspects described herein, which are directed to defining a process that allows a decoder to determine whether it is capable of decoding a video stream based on new information related to the number of operations (MAC) required to decode the video data of the video stream. The new information is generally defined as constraints that specify a conforming bitstream. These constraints are prescriptive, meaning that the encoder must ensure that the constraints are met given the profile / level it targets, and that a decoder that meets the given profile / level can decode a bitstream that conforms to these constraints. Various embodiments and variations of the present principles include adding new constraints on the number of MACS at a level, such as constraints on the maximum number of MACS per second (maximum MACS rate), constraints on the maximum number of MACS per sample, constraints on the maximum ratio to the maximum sample rate (such as maximum luma sample rate). Various other embodiments and variations of the present principles may include adding these different variations of new constraints at specific MAC levels, i.e., independent of the generally specified level, defining constraints on the number of MACS for NN tools only, non-NN tools only, any type of tool, or separating max bitrate from resolution and adding separate MACS / max bitrate constraints.

[0052] According to a first aspect of at least one embodiment, a video stream is defined that includes coded data representing a sequence of images, an indication of a profile among a plurality of profiles to which the coded data conforms, and an indication of a level among a plurality of levels to which the coded data conforms, where the plurality of profiles and the plurality of levels specify capacity constraints for decoding the coded data, the capacity constraints including an indication regarding a number of multiply-add operations required to decode the coded data.

[0053] Various embodiments for indicating the number of multiply-add operations in a capacity constraint are described below. According to a first embodiment, a constraint on the maximum MAC rate is specified in a level table. Thus, each level of the plurality of levels defines at least a maximum multiply-add operation rate. Therefore, it is proposed to define a video decoder configuration process responsive to a parameter value indicating a complexity level selected from a plurality of complexity levels, each complexity level defining at least a maximum multiply-add (MAC) operation rate (maximum number of kMACs per second). The maximum MAC rate may be indicated in a table defining different levels, as shown in Table 3, which is a modified version of the table.

[0054] [Table 3]

[0055] According to a second aspect of at least one embodiment, a method (400) for determining a capability of a decoder for decoding a video stream is disclosed.

[0056] Fig. 4 shows a block diagram of a method 400 for determining a decoder's capabilities for decoding a video stream according to a general aspect of the first embodiment. In step 401, the decoder parses information signaled in the input bitstream and related to the profile and level to which the coded video data conforms. In step 402, the decoder checks whether the profile information decoded from the bitstream corresponds to a profile supported by the decoder. If the decoder supports the profile (yes), the decoder further checks in step 403 whether the level information decoded from the bitstream corresponds to a level supported by the decoder. If the decoder supports the level (yes), the decoder executes the decoding process of the bitstream in step 404. If the decoder does not support the profile or level (no), the process stops (step 405). According to a specific variant not shown in Fig. 4, in which the bitstream is transmitted via a streaming platform, if the decoder does not support the profile or level (no), the decoder (or receiver) may generate a message to the encoder (or transmitter) to request a bitstream with a lower profile or level. In another variation, the decoder (or receiver) may select from among multiple encoded bitstreams an encoded bitstream having a profile and level that matches the decoder capabilities.

[0057] According to the second embodiment, the constraint on the ratio of the maximum MAC to the maximum sample rate is specified in a level table. Thus, each level of the plurality of levels defines at least the ratio of the maximum multiply-add operation to the maximum sample rate. For example, the constraint on the number of MACs is expressed as a ratio MaxMacsRatio to the maximum luma sample rate, as shown in Table 4. The maximum number of MACS per second (Max MACS rate) is derived as follows: MaxMacsR=MaxMacRatio×MaxLumaSr

[0058] [Table 4]

[0059] The decoder may proceed as described in FIG.

[0060] According to a third embodiment, the constraint on the number of multiply-add operations is specified in another type of level table. Thus, an indication of the multiply-add level (general_MAC_level_idc) to which of the multiple multiply-add levels the coded data fits is further specified in the bitstream. In a particular variant, each multiply-add level of the multiple multiply-add levels defines at least a maximum multiply-add operation rate. Advantageously, the information related to the maximum MAC rate is independent of the level information, for example as known from the profile, tier and level used in VVC. The MAC level is signaled using a parameter different from the level parameter. This is shown in Table 5, where a new syntax element general_MAC_level_idc is added in underlined font.

[0061] [Table 5]

[0062] This syntax element can have the following semantics, based on the semantics already used in VVC for the syntax element general_level_idc. general_MAC_level_idc indicates the MAC rate-related level to which OlsInScope (output layer set in scope) conforms as specified in Annex A. The bitstream shall not contain values ​​of general_MAC_level_idc other than those specified in Annex A (of the VVC specification). Other values ​​of general_MAC_level_idc are reserved for future use by ITU-T|ISO / IEC.

[0063] In this definition, an output layer set (OLS) is defined in VVC as a set of layers where one or more layers are designated as output layers, and a video is possibly one or more layers. The MAC level information general_MAC_level_idc is signaled in the bitstream, e.g., in a video parameter set (VPS) or sequence parameter set (SPS), e.g., in VVC where syntax structures are inserted in the VPS and optionally in the SPS (depending on the SPS syntax element "sps_ptl_dpb_hrd_params_present_flag").

[0064] An example table defining the different MAC levels is provided in the table.

[0065] [Table 6]

[0066] Fig. 5 shows a block diagram of a method 500 for determining a decoder's capabilities for decoding a video stream according to a general aspect of the third embodiment. In step 501, the decoder decodes and parses information related to the profile, level, and MAC level signaled in the input bitstream. In step 502, the decoder checks whether the profile information decoded from the bitstream corresponds to a profile supported by the decoder. If the decoder supports the profile (yes), the decoder further checks in step 503 whether the level information decoded from the bitstream corresponds to a level supported by the decoder. If the decoder supports the level (yes), the decoder further checks in step 504 whether the MAC level information decoded from the bitstream corresponds to a MAC level supported by the decoder. If the decoder supports the MAC level (yes), the decoder performs a bitstream decoding process in step 505. If the decoder does not support the profile, level, or MAC level, the process stops (step 506). According to different variants, determining that the profile to which the coded data fits is supported by the decoder, determining that the level to which the coded data fits is supported by the decoder and determining that the multiply-add level to which the coded data fits is supported by the decoder are performed independently and in any order. According to different variants, signaling of the profile to which the coded data fits is supported by the decoder, signaling of the level to which the coded data fits is supported by the decoder and signaling of the multiply-add level to which the coded data fits is supported by the decoder are performed independently and in any order.

[0067] FIG. 6 illustrates a general decoding method 300 according to a general aspect of at least one embodiment. The block diagram of FIG. 6 partially illustrates modules of a decoder or a decoding method implemented in, for example, the exemplary decoder of FIG. 3. Although not explicitly described, one skilled in the art would clearly derive a general decoding embodiment for any of the capacity constraint embodiments or variations described herein. Thus, for example, in step 610, a determination of the decoder's capability to decode a video stream that conforms to the capacity constraint, including the number of MAC operations, is processed based on the profile and level signaled in the bitstream and compared with the decoder capability. In response to the decoder determining that it is able to decode the bitstream, in 620, the bitstream is decoded. According to a specific variation not shown in FIG. 6, in response to the decoder being unable to decode the bitstream, the decoder (or receiver) may further generate a message to the encoder (or transmitter) to request a bitstream having a lower profile or level.

[0068] With respect to the third embodiment, according to the fourth embodiment, the constraint on the number of multiply-add operations is specified in another type of level table. In a particular feature of the fourth embodiment, each multiply-add operation level of the multiple multiply-add operation levels defines at least a ratio of a maximum multiply-add operation to a maximum sample rate. For example, the information on the maximum number of MACs is defined as a ratio with respect to a maximum luma sample rate (MaxLumaSr) expressed in samples / second. The constraint may be independent of the level information as in the third embodiment and may be controlled by another type of level information such as general_MAC_level_idc. The maximum MAC rate (MACs per second) MaxMacsR may be calculated as follows: MaxMacsR=MaxMacRatio×MaxLumaSr

[0069] An example table defining the different MAC levels is provided in Table 7. The numbers in this table are indicative.

[0070] [Table 7]

[0071] The decoder may proceed as described in FIG.

[0072] With respect to the third embodiment, according to the fifth embodiment, the constraint on the number of multiply-add operations is specified in another type of level table. In a particular feature of the fifth embodiment, each multiply-add level of the multiple multiply-add levels defines at least a maximum multiply-add operation per sample. Thus, the information related to the maximum number of MACs is defined as a maximum number of MACs per sample, not per second, MaxMacsPerSamp. Advantageously, this constraint imposes that for each sample to be processed, the maximum number of MACs is constrained to be smaller than the value MaxMacsPerSamp. This differs from the previous embodiment, where the constraint is global (only regarding the maximum rate expressed in MACs / second) and does not apply to each sample (maximum number of MACs / sample). As mentioned before, the constraint may be independent of the level information as mentioned before and may be controlled by another type of level information, such as general_MAC_level_idc. The maximum MAC rate (MACs per second) MaxMacsR may be calculated (for luma samples) as follows: MaxMacsR=MaxMacsPerSamp×MaxLumaSr

[0073] An example table defining the different MAC levels is provided in Table 8. The numbers in this table are indicative.

[0074] [Table 8]

[0075] The decoder may proceed as described in FIG.

[0076] According to the sixth embodiment, the constraint on the number of multiply-and-accumulate operations is specified for all or part of the decoding process. Thus, the number of multiply-and-accumulate operations required to decode the coded data applies to some or all of the decoding steps. Thus, the constraint associated with the number of MACs applies to some or all of the steps involved in the decoding process, regardless of the type of step (NN-based or not). The total number of MACs involved by each step considered (worst case) must be counted and compared with the constraint to check whether the level or MAC level is satisfied. The encoding process generating a conforming bitstream must ensure that the total number of MACs after decoding data for several consecutive pictures in decoding order or display order at a given granularity such as an access unit, or a picture, or a slice, or at each intra period, or in a given time period, is less than the maximum number of MACs defined by the constraint. For example, a high-level parameter inserted in the SPS or PPS may indicate the granularity to which the total number of MACs is associated. The total number of MACs may be counted by the encoder as follows: The global parameter totalMacsNb is set to 0 at the start of decoding data at a given granularity. For each step included in the decoding process within the encoding process (e.g., steps 240, 245, 250, 260, 265, 270) where the number of MACs must be taken into account, totalMacsNb is incremented by the number of MACs included in this step. The encoder must ensure that totalMacsNb, obtained at the end of the encoding of an access unit, or a picture, or a slice, or at each intra period (depending on the granularity at which the constraint is expressed), respects the constraint on the maximum number of MACs. This process may be described in a section on hypothetical reference decoders specified in the decoder specification (e.g., Annex C.5 of the VVC specification). An example of the specification text is given below, taking two tools, ALF, as an example.

[0077] Example of specification text: Whenever section 8.8.5.2 is invoked, the following applies: totalMacsNb=totalMacsNb+K0 * (1< <CtbLog2SizeY)x(1<<CtbLog2SizeY) Whenever section 8.8.5.4 is invoked, the following applies: totalMacsNb=totalMacsNb+K1 * ((1< <CtbLog2SizeY) / SubWidthC)x ((1< <CtbLog2SizeY) / SubHeightC) Whenever section 8.8.5.7 is invoked, the following applies: totalMacsNb=totalMacsNb+K2 * ((1< <CtbLog2SizeY) / SubWidthC)x ((1< <CtbLog2SizeY) / SubHeightC) Note that in the VVC specification, clause 8.8.5.2 describes the ALF filtering of luma samples in the CTB, clause 8.8.5.4 describes the ALF filtering of chroma samples in the CTB, and clause 8.8.5.7 describes the CC-ALF filtering of chroma samples in the CTB. The parameters K0, K1, K2 are predefined values ​​that correspond to the number of MACs per sample required by these steps. For example, K0=K1=K2=16. }End of example specification text

[0078] According to the seventh embodiment, for some of the decoding steps that perform neural network processing, a constraint is specified on the number of multiply-and-accumulate operations.Therefore, the constraint related to the number of MACs does not relate to the whole decoding process, but only to the steps of the decoding process that involve NN-based operations.The total number of MACs involved in these specific steps must be counted and compared with the constraint to check whether the level or MAC level is met.

[0079] The NN-based operations must be canonically identified in the decoding process, and the number of MACs for each of the identified NN-based operations must be summed.

[0080] For example, in VVC, matrix-based intra prediction can be considered as a shallow NN-based tool. According to a particular variant, if the decoder does not support constraints involving NN-based operations, the decoder can disable such tools. In one variant, the decoder (or receiver) can generate a message to the encoder (or transmitter) to request an encoded bitstream that does not implement such NN tools when decoding. In another variant, the decoder (or receiver) can select an encoded bitstream that does not implement such NN tools when decoding from among multiple encoded bitstreams.

[0081] According to an eighth embodiment, separate constraints on the number of multiply-and-accumulate operations are specified depending on the type of the decoding process. For example, the type of the decoding process may be defined according to a NN or non-NN implementation. Thus, in a particular feature, a first indication (general_MAC_levelA_idc) on the number of multiply-and-accumulate operations required to decode the coded data is applied to a part of the decoding step implementing neural network processing, and a second indication (general_MAC_levelB_idc) on the number of multiply-and-accumulate operations required to decode the coded data is applied to a part of the decoding step implementing no neural network processing. Advantageously, in this embodiment, different constraints associated with the number of MACs are defined depending on the type of steps involved in the decoding process. A first set of MAC levels (MAC level A) is defined for all or part of the steps of the decoding process including NN-based operations (type A steps). A second set of MAC levels (MAC level B) is defined for all or part of the steps of the decoding process including non-NN-based operations (type B steps). Table 9 shows the syntax changes with two new syntax elements "general_MAC_levelA_idc" and "general_MAC_levelB_idc".

[0082] [Table 9]

[0083] According to the ninth embodiment, a separate constraint on the number of multiply-accumulate operations is specified for out-of-loop post-decoding processes. As for the eighth embodiment, a separate constraint associated with the number of MACs is defined for out-of-loop processes such as post-decoding filtering, post-decoding resampling, post-decoding film grain synthesis, post-decoding video enhancement, post-decoding dynamic range adaptation (tone mapping or inverse tone mapping).

[0084] According to the tenth embodiment, the constraints between resolution, bit rate and MACS are de-correlated. In non-broadcast applications, the concept of a level that sets a maximum bit rate limit (having a main or high tier) for a given resolution is actually ignored by many service providers. To overcome this arbitrary setting limit, a service provider may choose to signal a higher level that allows a higher bit rate and scale down the resolution through a different control mechanism.

[0085] In a first variant of this embodiment, three independent syntax parameters are defined as follows, replacing, for example, both the tier and level parameters defined in VVC, as shown in Table 10 (the added syntax elements are underlined):

[0086] [Table 10]

[0087] Here, general_resolution_idc indicates the maximum resolution that corresponds to the maximum luma sample rate MaxLumaSr (samples / sec) in Table 2. In this scenario, the maximum number of slices and tiles is inferred from general_resolution_idc.

[0088] Here, general_bitrate_idc indicates the maximum bitrate corresponding to the maximum bitrate MaxBR (BrVclFactor or BrNalFactor bits / sec) in Table 2.

[0089] Here, general_MAC_idc may be defined as in any of the previous embodiments.

[0090] However, it may be difficult for a suitable decoder to handle the infinite number of combinations.

[0091] In a second variant of this embodiment, "level" and "tier" are further constrained by new syntax elements. According to a particular feature, the concept of level is further constrained by three syntax elements: resolution, bitrate and MACS. Table 11 below shows the syntax changes with the new syntax elements (underlined) when the concept of level is further constrained.

[0092] [Table 11]

[0093] where general_resolution_idc indicates the maximum resolution corresponding to the maximum luma sample rate MaxLumaSr (samples / sec). If not present, MaxLumaSr is as defined in Table 2. If present, the value indicates a lower resolution corresponding to a lower level (maximum luma sample rate as listed in Table 2).

[0094] Here, general_bitrate_idc indicates the maximum bitrate corresponding to the maximum bitrate MaxBR (BrVclFactor or BrNalFactor bits / sec). If not present, MaxBR is as defined in Table 2. If present, its value indicates a lower MaxBR corresponding to a lower level and / or tier as described in Table 2. Note that the values ​​of general_resolution_idc and general_bitrate_idc do not necessarily correlate (hence differing from the concept of level).

[0095] Here, general_MAC_idc may be defined as in any of the previous embodiments.

[0096] According to an eleventh embodiment, a generic encoding method 200 is disclosed that generates a bitstream that complies with the capacity constraints according to any of the embodiments or variants thereof. Although not explicitly described, a person skilled in the art will be able to clearly derive a generic encoding implementation for the specific embodiments described herein.

[0097] In the encoding process, when an encoder performs generation of a bitstream conforming to a specified profile / level, it needs to insert profile, level related parameters into the bitstream and communicate them to the decoding side. Thus, the encoder signals indicators related to the profile, level, and if relevant, the MAC level, or if relevant, the MAC level A and MAC level B, to ensure that a decoder conforming to this profile, level, and if relevant, the MAC level, or if relevant, the MAC level A and MAC level B can decode the bitstream.

[0098] FIG. 7 illustrates a general encoding method 200 according to a general aspect of at least one embodiment. The block diagram of FIG. 7 is implemented in an encoder or, for example, in the exemplary encoder of FIG. 2 and partially represents modules of the encoding method. In step 710, a profile and a level among a plurality of profiles and levels to which the coded data should conform are obtained. According to a salient feature, the plurality of profiles and levels specify a capacity constraint for decoding the conforming coded data, the capacity constraint including an indication on the number of multiply-add operations required to decode the coded data in any of the variants observed herein. Then, in 720, a sequence of images is encoded according to the selected profile and level. An indication of the profile to which the coded data conforms (such as general_profile_idc) and an indication of the level to which the coded data conforms (such as general_level_idc or any of its variants) are further encoded (730) in the bitstream to enable a decoder to determine whether it can decode the bitstream.

[0099] According to a twelfth embodiment, a separate constraint on the number of multiply-add operations is specified for the display processing. According to a particular variant, the capacity constraint includes an indication on the number of multiply-add operations required for the NN-based display processing. According to another particular variant, the video stream further includes an indication of a maximum multiply-add operation rate required for the NN-based display processing. According to yet another particular variant, the indication of a maximum multiply-add operation rate required for the NN-based display processing is embedded in the SEI message.

[0100] Specifying constraints on the MAC rate or number of MACs per sample for NN-based display processing is advantageous for display processing such as super-resolution, filtering, edge enhancement, etc. The constraints that the bitstream conforms to can be signaled, for example, as metadata in an SEI message. The SEI message includes a syntax element max_MACS_rate indicating the maximum number of MACS per second that a device applying the display process must support at most. Alternatively, the SEI message includes a syntax element max_MACS_per_sample indicating the maximum number of MACS per sample that a device applying the display process must support at most.

[0101] Alternatively, the SEI message includes two syntax elements: one syntax element associated with the maximum number of samples per second, max_samples_rate, and one syntax element indicating a maximum ratio related to the maximum sample rate, max_MACS_ratio, which indicates the maximum MACS rate. * max_MACS_ratio).

[0102] Further embodiments and information Various methods are described herein, each of which includes one or more steps or acts for achieving the described method. Unless a specific order of steps or acts is required for proper operation of the method, the order and / or use of specific steps and / or acts may be modified or combined. Additionally, terms such as "first", "second" and the like may be used in various embodiments to modify elements, components, steps, operations, etc., e.g., "first decoding" and "second decoding". The use of such terms does not imply any ordering to the modified operations, unless specifically required. Thus, in this embodiment, the first decoding need not be performed before the second decoding, but may occur, for example, before, during, or during a period overlapping with the second decoding.

[0103] Various methods and other aspects described herein may be used to modify the video encoder 200 and decoder 300 modules as shown in Figures 2 and 3. Moreover, the aspects are not limited to VVC or HEVC, but may be applied, for example, to other standards and recommendations, and extensions of any such standards and recommendations. Unless otherwise indicated or technically precluded, the aspects described herein may be used individually or in combination.

[0104] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes and the described aspects are not limited to these specific values.

[0105] Various implementations involve decoding. As used herein, "decoding" may encompass all or a portion of the processes performed on a received encoded sequence to generate a final output suitable for, for example, a display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as, for example, entropy decoding, inverse quantization, inverse transform, and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to the broader decoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0106] Various implementations involve encoding. Similar to the discussion above regarding "decoding," "encoding" as used in this application may encompass all or a portion of the processes performed on an input video sequence to generate, for example, an encoded bitstream.

[0107] It should be noted that the syntax elements used herein are descriptive terms, and therefore do not exclude the use of other syntax element names.

[0108] Implementations and aspects described herein may be implemented as various information, such as, for example, syntax, that may be transmitted or stored. This information may be packaged or arranged in various manners, including, for example, common in video standards, such as placing the information in an SPS, PPS, NAL unit, header (e.g., a NAL unit header or slice header), or SEI message. Other manners are also available, including, for example, common in system-level or application-level standards, such as placing the information in one or more of the following: SDP (session description protocol), a format for describing multimedia communication sessions for the purposes of session announcement and session invitation, e.g. as described in RFCs and used in conjunction with RTP (Real-time Transport Protocol) transport; DASH Media Presentation Description (MPD) descriptors, e.g. as used in DASH and transmitted over HTTP, that are associated with a representation or a set of representations to provide additional characteristics to the content representation; RTP header extensions, e.g. as used during RTP streaming, The ISO Base Media File Format, such as that used in OMAF and in some specifications that uses boxes, which are object-oriented building blocks defined by a unique type identifier and a length, also known as "atoms", · HLS (HTTP live streaming) manifests transmitted over HTTP. A manifest can be associated with a version or collection of versions of content, for example to provide characteristics of the version or collection of versions.

[0109] The implementations and aspects described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single implementation (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The method may be implemented in, for example, an apparatus such as a processor, which generally refers to processing devices including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include, for example, communication devices such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0110] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation", as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation", as well as other variations, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0111] It should be noted that the application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0112] Additionally, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0113] It should be noted that the application may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing the information or retrieving the information (e.g., from a memory). Furthermore, "receiving" typically involves in some way an operation, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0114] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first enumerated alternative (A), or the selection of only the second enumerated alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first enumerated alternative (A), or the selection of only the second enumerated alternative (B), or the selection of only the third enumerated alternative (C), or the selection of only the first and second enumerated alternatives (A and B), or the selection of only the first and third enumerated alternatives (A and C), or the selection of only the second and third enumerated alternatives (B and C), or the selection of all three alternatives (A and B and C). This can be expanded as many times as the items listed, as would be apparent to one of ordinary skill in this and related arts.

[0115] Also, as used herein, the term "signaling" means to indicate something, especially to a corresponding decoder. For example, in a particular embodiment, the encoder signals a quantization matrix for dequantization. Thus, in an embodiment, the same parameters are used at both the encoder and decoder sides. Thus, for example, the encoder can transmit a particular parameter to the decoder so that the decoder can use the same particular parameter (explicit signaling). In contrast, if the decoder already has the particular parameter as well as other parameters, a non-transmitting signaling (implicit signaling) can be used to simply allow the decoder to know and select the particular parameter. By avoiding the transmission of any actual function, bit savings are realized in various embodiments. It will be understood that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used in various embodiments to signal information to the corresponding decoder. The above relates to the verb form of the word "signal", which may also be used as a noun in this specification.

[0116] As will be apparent to one of ordinary skill in the art, implementations may generate a variety of signals formatted to carry information that may be, for example, stored or transmitted. Information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bit stream of the described embodiments. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information that the signal carries may be, for example, analog information or digital information. The signal may be transmitted by a variety of different wired or wireless links, as is well known. The signal may be stored in a processor-readable medium.

[0117] Furthermore, the disclosed concepts are based on the number of MAC operations used as a complexity metric. Those skilled in the art will clearly derive from this disclosure other complexity metrics such as number of operations (OPs), number of floating point operations (FLOPs), number of uses per sample, etc.

[0118] Several embodiments are described. The features of these embodiments may be provided alone or in any combination across various claim categories and types. Furthermore, the embodiments may include one or more of the following features, devices, or aspects across various claim categories and types, alone or in any combination. The bitstream or signal includes one or more of the described syntax elements, constraint files, or variations thereof. The bitstream or signal includes a syntax carrying information generated according to any of the described embodiments. · Inserting it into a syntax element or constraint file that allows the decoder to process it in a way that matches the way the encoder uses it. · Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the described syntax elements, constraint files, or variations thereof. · Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal according to any of the described embodiments. Any of the described embodiments provide a method, process, apparatus, medium for storing instructions, medium for storing data or signals. A TV, set-top box, mobile phone, tablet, or other electronic device that performs a capabilities constraint checking process according to any of the described embodiments. A TV, set-top box, mobile phone, tablet, or other electronic device that performs a capability constraint check according to any of the described embodiments and displays the resulting images (e.g., using a monitor, screen, or other type of display). A TV, set-top box, mobile phone, tablet, or other electronic device that selects a channel (e.g., using a tuner) to receive a signal containing encoded images and performs a capability constraint check according to any of the described embodiments. A TV, set-top box, mobile phone, tablet, or other electronic device that receives (e.g., using an antenna) an over-the-air signal containing an encoded image and performs a capability constraint check according to any of the described embodiments.

Claims

1. 1. A method implemented in a decoder, comprising determining a capability of the decoder to decode a video stream, the video stream comprising coded data representing a sequence of images, an indication of a profile to which the coded data conforms from a plurality of profiles, and an indication of a level to which the coded data conforms from a plurality of levels, the plurality of profiles and levels specifying capacity constraints for decoding conforming coded data, the capacity constraints including an indication as to the number of multiply-add operations required to decode the coded data; determining the capability of a decoder to decode the video stream; parsing the indication of the profile to which the coded data matches; parsing said indication of said level to which coded data conforms; The method implemented in a decoder, further comprising: in response to determining that the profile is supported by the decoder and that the level is supported by the decoder, determining that the decoder is capable of performing certain multiply-and-accumulate operations required to decode coded data.

2. The method of claim 1 , wherein each level of the plurality of levels defines at least a maximum multiply-add operation rate.

3. The method of claim 1 , wherein each level of the plurality of levels defines at least a ratio of maximum multiply-add operations to a maximum sample rate.

4. 2. The method of claim 1, wherein the video stream further includes an indication of which of a plurality of multiply-accumulate levels the coded data conforms to.

5. determining the capability of a decoder to decode the video stream; parsing said indication of a multiply-accumulate level to which the coded data fits; 5. The method of claim 4, further comprising: in response to determining that the multiply-accumulate operation level is supported by the decoder, determining that the decoder is capable of performing a number of multiply-accumulate operations required to decode coded data.

6. 6. The method of claim 5, further comprising decoding from the video stream the indication of a multiply-accumulate level to which coded data fits.

7. 7. The method of claim 5 or 6, wherein determining that the profile to which the coded data fits is supported by the decoder, determining that the level to which the coded data fits is supported by the decoder, and determining that the multiply-accumulate level to which the coded data fits is supported by the decoder are performed independently and in any order.

8. The method according to any one of claims 4 to 7, wherein each multiply-add operation level of the plurality of multiply-add operation levels defines at least a maximum multiply-add operation rate.

9. The method of any one of claims 4 to 7, wherein each multiply-add level of the plurality of multiply-add levels defines at least a ratio of maximum multiply-add operations to a maximum sample rate.

10. The method of any one of claims 4 to 7, wherein each multiply-add operation level of the plurality of multiply-add operation levels defines at least a maximum multiply-add operation per sample.

11. A method according to any one of claims 2 to 10, wherein said indication of the number of multiply-accumulate operations required to decode coded data applies to some or all of the decoding steps.

12. A method according to any one of claims 2 to 10, wherein the indication of the number of multiply-accumulate operations required to decode coded data is applied to a part of the decoding step which implements neural network processing.

13. 11. The method of claim 2, wherein the indication on the number of multiply-and-accumulate operations required to decode the coded data comprises a first indication on the number of multiply-and-accumulate operations required to decode the coded data applied to a part of the decoding step that implements neural network processing, and a second indication on the number of multiply-and-accumulate operations required to decode the coded data applied to a part of the decoding step that does not implement neural network processing.

14. A method according to any one of claims 2 to 10, wherein said indication of the number of multiply-accumulate operations required to decode coded data is applied in a post-processing part of the decoding step.

15. A method according to any one of claims 2 to 14, wherein the capacity constraints include an indication as to the number of multiply-add operations required for NN-based display processing.

16. 16. The method of claim 15, wherein the video stream further includes an indication of a maximum multiply-add operation rate required for NN-based display processing.

17. The method of claim 16 , wherein the indication of a maximum multiply-add operation rate required for NN-based display processing is embedded in an SEI message.

18. 1. A method comprising: receiving a video stream, the video stream including coded data representing a sequence of images, an indication of a profile from a plurality of profiles to which the coded data conforms, and an indication of a level from a plurality of levels to which the coded data conforms, the plurality of profiles and levels specifying capacity constraints for decoding the conforming coded data, the capacity constraints including an indication regarding a number of multiply-add operations required to decode the coded data.

19. 1. A method comprising: transmitting a video stream, the video stream including coded data representing a sequence of images, an indication of a profile among a plurality of profiles to which the coded data conforms, and an indication of a level among a plurality of levels to which the coded data conforms, the plurality of profiles and levels specifying capacity constraints for decoding the conforming coded data, the capacity constraints including an indication regarding a number of multiply-add operations required to decode the coded data.

20. 1. A video stream comprising: coded data representing a sequence of images; an indication of a profile among a plurality of profiles to which the coded data conforms; and an indication of a level among a plurality of levels to which the coded data conforms, the plurality of profiles and levels specifying capacity constraints for decoding the conforming coded data, the capacity constraints including an indication as to the number of multiply-add operations required to decode the coded data.

21. 1. A method comprising decoding coded data representing a sequence of images from a video stream, the method comprising: Determining the capabilities of a decoder for decoding a video stream according to a method according to any one of claims 1 to 17; in response to the decoder determining that it is capable of decoding the video stream, decoding the video stream.

22. 1. A method comprising encoding data representing a sequence of images into a video stream, the method comprising: obtaining a profile of a plurality of profiles to which the coded data should conform; obtaining a level among a plurality of levels to which the coded data should conform; encoding data representing a sequence of images in accordance with said profile and said level; encoding an indication of the profile to which the coded data conforms; and and encoding an indication of the level to which the coded data conforms; A method according to claim 1, wherein the plurality of profiles and levels specify capacity constraints for decoding conforming coded data, the capacity constraints including an indication as to the number of multiply-add operations required to decode the coded data.

23. 1. An apparatus comprising a decoder for decoding a video stream comprising coded data representing a sequence of images, the decoder comprising: Determining the capabilities of the decoder for decoding the video stream according to a method according to any one of claims 1 to 17, In response to the decoder determining that it is able to decode the video stream, the apparatus is configured to decode the video stream.

24. 1. An apparatus comprising an encoder for encoding data representing a sequence of images into a video stream, the encoder comprising: Obtaining a profile among a plurality of profiles to which the coded data should conform; Obtaining a level among a plurality of levels to which the coded data should conform; encoding data representing a sequence of images according to said profile and said level; encoding an indication of the profile to which the coded data conforms; the coded data is configured to encode an indication of a level of conformance; An apparatus, wherein the plurality of profiles and levels specify capacity constraints for decoding conforming coded data, the capacity constraints including an indication as to the number of multiply-add operations required to decode the coded data.

25. A computer program comprising program code instructions for carrying out the steps of the method according to any one of claims 1 to 19, 21 and 22, when the computer program is executed by a processor.

26. A non-transitory program storage device readable by a computer, tangibly embodying a program of instructions executable by said computer to perform the method of any one of claims 1, 18, 19, 20 and 22.

Images