EVC decoding complexity metric

Abstract

An apparatus includes a communication interface and a processor operably coupled to the communication interface. The processor is configured to receive a bitstream comprising complexity metric (CM) metadata via the communication interface. The processor is configured to, for each CU size of the plurality of CU sizes, extract information in CM metadata from the bitstream, where the information includes a direct indication of a number of coding units (CUs) having non-zero transform coefficients in the bitstream. The processor is configured to predict a decoding complexity of the bitstream based on information in the CM metadata.

Description

Technical Field

[0001] The present disclosure relates generally to media devices and processes. More specifically, the present disclosure relates to elementary video coding (EVC) decoding complexity metrics. Background Art

[0002] EVC Baseline and Main profiles do not share many tools and use different methods to partition pictures. The lengths of the syntax elements indicating the number of samples and coding units need to be decided based on their maximum values ​​to minimize the size of the Software Engineering Institute (SEI) messages that carry complexity metrics. Summary of the invention

[0003] Technical Solutions

[0004] The present disclosure provides an EVC decoding complexity metric.

[0005] In a first embodiment, an apparatus includes a communication interface and a processor operably coupled to the communication interface. The processor is configured to receive a bitstream including complexity metric (CM) metadata via the communication interface. The processor is configured to extract information in the CM metadata from the bitstream for each CU size in a plurality of CU sizes, wherein the information includes a direct indication of the number of coding units (CUs) having non-zero transform coefficients in the bitstream. The processor is configured to predict a decoding complexity of the bitstream based on the information in the CM metadata.

[0006] In a second embodiment, a method includes receiving a bitstream including complexity metric (CM) metadata. The method includes extracting information in the CM metadata from the bitstream for each CU size in a plurality of CU sizes, wherein the information includes a direct indication of a number of coding units (CUs) in the bitstream having non-zero transform coefficients. The method includes predicting a decoding complexity of the bitstream based on the information in the CM metadata.

[0007] Other technical features may be apparent to those skilled in the art from the following drawings, descriptions and claims.

[0008] Before the following detailed description, it may be advantageous to set forth the definitions of certain words and phrases used throughout this patent document. The term "coupling" and its derivatives refer to any direct or indirect communication between two or more elements, whether or not these elements are in physical contact with each other. The terms "send", "receive" and "communication" and their derivatives include both direct and indirect communication. The terms "include" and "comprise" and their derivatives mean including but not limited to this. The term "or" is inclusive, meaning and / or. The phrase "associated with..." and its derivatives mean including, included in, interconnected with, included in, connected to or connected with, coupled to or coupled with, communicable with, collaborate with, interweave, juxtapose, approach, bind to or bind with, have, have the property of, have a relationship with, etc. The term "controller" means any device, system or part thereof that controls at least one operation. Such a controller can be implemented in hardware or a combination of hardware and software and / or firmware. The functionality associated with any particular controller may be centralized or distributed, whether local or remote. When used with a list of items, the phrase "at least one of" means that different combinations of one or more of the listed items may be used, and only one of the items in the list may be required. For example, "at least one of A, B, and C" includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A and B and C.

[0009] In addition, the various functions described below may be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, related data, or a portion thereof, suitable for implementation in a suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disk (CD), a digital video disk (DVD), or any other type of memory. "Non-transitory" computer-readable media do not include wired, wireless, optical, or other communication links that transmit temporary electrical or other signals. Non-transitory computer-readable media include media that can store data persistently and media that can store data and rewrite data later, such as rewritable optical disks or erasable memory devices.

[0010] Definitions for certain other words and phrases are provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals represent like parts:

[0012] Figure 1 An example communication system according to an embodiment of the present disclosure is shown;

[0013] Figure 2 An example electronic device according to an embodiment of the present disclosure is shown;

[0014] Figure 3 An example electronic device according to an embodiment of the present disclosure is shown;

[0015] Figure 4 A block diagram illustrating an example environment architecture according to an embodiment of the present disclosure; and

[0016] Figure 5 An example method for EVC decoding complexity metric according to the present disclosure is shown. DETAILED DESCRIPTION

[0017] Described below Figures 1 to 5 The various embodiments used to describe the principles of the present disclosure are for illustration only and should not be interpreted in any way as limiting the scope of the present disclosure. It will be understood by those skilled in the art that the principles of the present disclosure can be implemented in any type of appropriately arranged device or system.

[0018] As discussed above, the EVC baseline profile and the main profile do not share many tools and use different methods to segment pictures. Therefore, the profile to which the Concurrent Versions System (CVS) is conformed is used to determine the set of syntax elements used to describe the complexity metric applied to each CVS. The maximum number of pixels and coding units depends on the size of the picture to which the complexity metric is applied. Since the maximum size of the picture is indicated by the level to which the CVS conforms, the length of the syntax elements indicating the number of pixels and coding units is determined by the level. In addition, when the length of the syntax element indicating the number of coding units of a CVS conforming to the baseline profile becomes smaller as the width and height of the coding unit become larger than the maximum number of coding units, the width and height of the coding unit are taken into account.

[0019] Figure 1 An example communication system 100 is shown in accordance with an embodiment of the present disclosure. Figure 1The embodiment of the communication system 100 shown is for illustration only. Other embodiments of the communication system 100 may be used without departing from the scope of the present disclosure.

[0020] like Figure 1 As shown, the communication system 100 includes a network 102 that facilitates communication between various components in the communication system 100. For example, the network 102 can transmit IP packets, frame relay frames, asynchronous transfer mode (ATM) cells, or other information between network addresses. The network 102 includes all or a portion of one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), a global network (such as the Internet), or any other one or more communication systems at one or more locations.

[0021] In an example, the network 102 facilitates communication between the server 104 and various client devices 106-116. The client devices 106-116 may be, for example, smart phones, tablet computers, laptop computers, personal computers, TVs, interactive displays, wearable devices, HMDs, etc. The server 104 may represent one or more servers. Each server 104 includes any suitable computing or processing device that can provide computing services to one or more client devices (such as client devices 106-116). Each server 104 may, for example, include one or more processing devices, one or more memories storing instructions and data, and one or more network interfaces that facilitate communication through the network 102. As described in more detail below, the server 104 may send a compressed bitstream representing a point cloud or mesh to one or more display devices, such as client devices 106-116. In an embodiment, each server 104 may implement an EVC decoding complexity metric.

[0022] Each client device 106-116 represents any suitable computing or processing device that interacts with at least one server (such as server 104) or other computing device through network 102. Client devices 106-116 include desktop computers 106, mobile phones or mobile devices 108 (such as smart phones), PDAs 110, laptop computers 112, tablet computers 114, and HMDs 116. However, any other or additional client devices may be used in the communication system 100. Smart phones represent a class of mobile devices 108 that are handheld devices with mobile operating systems and integrated mobile broadband cellular network connections for voice, short message service (SMS), and Internet data communications. HMD 116 can display a 360° scene including one or more dynamic or static 3D point clouds. In an embodiment, any one of the client devices 106-116 may include an encoder, a decoder, or both. For example, the mobile device 108 may record a 3D volumetric video and then encode the video so that the video can be sent to one of the client devices 106-116. In an example, laptop computer 112 may be used to generate a 3D point cloud or mesh, which is then encoded and sent to one of client devices 106 - 116 .

[0023] In the example, some of the client devices 108-116 communicate indirectly with the network 102. For example, the mobile device 108 and the PDA 110 communicate via one or more base stations 118, such as cellular base stations or eNodeBs (eNBs). In addition, the laptop 112, the tablet computer 114, and the HMD 116 communicate via one or more wireless access points 120, such as IEEE 802.11 wireless access points. Note that these are for illustration only, and each client device 106-116 can communicate directly with the network 102 or indirectly with the network 102 via any suitable intermediate device or network. In an embodiment, the server 104 or any of the client devices 106-116 can be used to compress the point cloud or mesh, generate a bitstream representing the point cloud or mesh, and send the bitstream to another client device, such as any of the client devices 106-116.

[0024] In an embodiment, any of the client devices 106-114 securely and efficiently sends information to another device, such as the server 104. In addition, any of the client devices 106-116 can trigger the transmission of information between itself and the server 104. Any of the client devices 106-114 can be used as a VR display when attached to a headset via a bracket and function similarly to the HMD 116. For example, the mobile device 108 can function similarly to the HMD 116 when attached to the bracket system and worn on the user's eyes. The mobile device 108 (or any other client device 106-116) can trigger the transmission of information between itself and the server 104.

[0025] In an embodiment, any of the client devices 106-116 or the server 104 may create a 3D point cloud or mesh, compress a 3D point cloud or mesh, send a 3D point cloud or mesh, receive a 3D point cloud or mesh, decode a 3D point cloud or mesh, render a 3D point cloud or mesh, or a combination thereof. For example, the server 104 may then compress the 3D point cloud or mesh to generate a bitstream and then send the bitstream to one or more of the client devices 106-116. For example, one of the client devices 106-116 may compress the 3D point cloud or mesh to generate a bitstream and then send the bitstream to another of the client devices 106-116 or the server 104. In an embodiment, the client devices 106-116 implement the EVC decoding complexity metric.

[0026] although Figure 1 One example of a communication system 100 is shown, but may be Figure 1 Various changes may be made. For example, communication system 100 may include any number of each component in any suitable arrangement. In general, computing and communication systems have a wide variety of configurations, and Figure 1 The scope of the present disclosure is not limited to any particular configuration. Figure 1 One operating environment is shown in which the various features disclosed in this patent document may be used, but these features may be used in any other suitable system.

[0027] Figure 2 An example electronic device according to an embodiment of the present disclosure is shown. In particular, Figure 2 An example server 200 is shown, and the server 200 may represent Figure 1 Server 200 may represent one or more encoders, decoders, local servers, remote servers, cluster computers, and components that act as a single seamless resource pool, cloud-based servers, etc. Server 200 may be composed of Figure 1 One or more of the client devices 106-116 or another server accesses the server.

[0028] like Figure 2 As shown, server 200 may represent one or more local servers, one or more compression servers, or one or more encoding servers, such as encoders. In an embodiment, the encoder may perform decoding. Figure 2 As shown, server 200 includes a bus system 205 that supports communications between at least one processing device (such as processor 210 ), at least one storage device 215 , at least one communication interface 220 , and at least one input / output (I / O) unit 225 .

[0029] Processor 210 executes instructions that may be stored in memory 230. Processor 210 may include any suitable number and type of processors or other devices in any suitable arrangement. Example types of processor 210 include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuits.

[0030] In an embodiment, the processor 210 may encode the 3D point cloud or mesh stored in the storage device 215. In an embodiment, the 3D point cloud is encoded and the 3D point cloud or mesh is decoded to ensure that when the point cloud or mesh is reconstructed, the reconstructed 3D point cloud or mesh matches the 3D point cloud or mesh before encoding. In an embodiment, the processor 210 may implement an EVC decoding complexity metric.

[0031] Memory 230 and persistent storage 235 are examples of storage devices 215, which represent any structure capable of storing and facilitating retrieval of information, such as temporary or permanent data, program code, or other suitable information. Memory 230 may represent random access memory or any other suitable volatile or non-volatile storage device. Persistent storage 235 may include one or more components or devices that support long-term storage of data, such as read-only memory, a hard drive, flash memory, or an optical disk.

[0032] Communication interface 220 supports communication with other systems or devices. For example, communication interface 220 may include interfaces that facilitate communication via Figure 1 The communication interface 220 may include a network interface card or a wireless transceiver for communicating with the network 102. The communication interface 220 may support communication via any suitable physical or wireless communication link. For example, the communication interface 220 may send a bitstream including a 3D point cloud to another device, such as one of the client devices 106-116.

[0033] I / O unit 225 allows input and output of data. For example, I / O unit 225 can provide a connection for user input through a keyboard, mouse, keypad, touch screen or other suitable input device. I / O unit 225 can send output to a display, printer or other suitable output device. However, it is noted that I / O unit 225 can be omitted when I / O interaction with server 200 occurs via a network connection, for example.

[0034] Note that although Figure 2 Described as indicating Figure 1 104, but the same or similar structure may be used in one or more of the various client devices 106-116. For example, a desktop computer 106 or a laptop computer 112 may have a Figure 2 The same or similar structure as shown in .

[0035] Figure 3 An example electronic device according to an embodiment of the present disclosure is shown. Figure 3 An example electronic device 300 is shown and may represent Figure 1 The electronic device 300 may be a mobile communication device, such as, for example, a mobile station, a subscriber station, a wireless terminal, a desktop computer (similar to a Figure 1 desktop computer 106), portable electronic devices (similar to Figure 1 In an embodiment, Figure 1 One or more of the client devices 106-116 may include the same or similar configuration as the electronic device 300. In an embodiment, the electronic device 300 is an encoder, a decoder, or both. For example, the electronic device 300 may be used for data transmission, image or video compression, image or video decompression, encoding, decoding, and media rendering applications.

[0036] like Figure 3 As shown, the electronic device 300 includes an antenna 305, a radio frequency (RF) transceiver, a transmission (TX) processing circuit, a microphone 320, and a reception (RX) processing circuit 325. The RF transceiver may include, for example, an RF transceiver, a Bluetooth transceiver, a WI-FI transceiver, a ZIGBEE transceiver, an infrared transceiver, and various other wireless communication signals. The electronic device 300 includes a speaker 330, a processor 340, an input / output (I / O) interface (IF) 345, an input 350, a display 355, a memory 360, and a sensor 365. The memory 360 includes an operating system (OS) 361 and one or more applications 362.

[0037] The RF transceiver receives an incoming RF signal from the antenna 305 transmitted from an access point (such as a base station, a WI-FI router, or a Bluetooth device) or other devices of the network 102 (such as WI-FI, Bluetooth, mobile phones, 5G, LTE, LTE-A, WiMAX, or any other type of wireless network). The RF transceiver down-converts the incoming RF signal to generate an intermediate frequency or baseband signal. The intermediate frequency or baseband signal is sent to the RX processing circuit 325, which generates a processed baseband signal by filtering, decoding, and / or digitizing the baseband or intermediate frequency signal. The RX processing circuit 325 sends the processed baseband signal to the speaker 330 (such as for voice data) or to the processor 340 for further processing (such as for web browsing data).

[0038] The TX processing circuit receives analog or digital voice data from microphone 320, or receives other outgoing baseband data from processor 340. The outgoing baseband data may include network data, email, or interactive video game data. The TX processing circuit encodes, multiplexes, and / or digitizes the outgoing baseband data to generate a processed baseband or intermediate frequency signal. The RF transceiver receives the outgoing processed baseband or intermediate frequency signal from the TX processing circuit and up-converts the baseband or intermediate frequency signal to an RF signal that is transmitted via antenna 305.

[0039] The processor 340 may include one or more processors or other processing devices. The processor 340 may execute instructions stored in the memory 360 (such as OS 361) to control the overall operation of the electronic device 300. For example, the processor 340 may control the reception of the forward channel signal and the transmission of the reverse channel signal through the RF transceiver 310, the RX processing circuit 325 and the TX processing circuit 315 according to well-known principles. The processor 340 may include any suitable number and type of processors or other devices of any suitable arrangement. For example, in an embodiment, the processor 340 includes at least one microprocessor or microcontroller. Example types of the processor 340 include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application-specific integrated circuits, and discrete circuits.

[0040] The processor 340 is capable of executing other processes and programs resident in the memory 360, such as operations of receiving and storing data. The processor 340 can move data into or out of the memory 360 as required by the execution process. In an embodiment, the processor 340 is configured to execute one or more applications 362 based on the OS 361 or in response to a signal received from an external source or operator. For example, the application 362 may include an encoder, a decoder, a VR or AR application, a camera application (for still images and videos), a video phone call application, an email client, a social media client, an SMS messaging client, a virtual assistant, etc. In an embodiment, the processor 340 is configured to receive and send media content. In an embodiment, the processor 340 may implement an EVC decoding complexity metric.

[0041] Processor 340 is coupled to I / O interface 345 , which provides electronic device 300 with the ability to connect to other devices, such as client devices 106 - 114 . I / O interface 345 is the communication path between these accessories and processor 340 .

[0042] Processor 340 is coupled to input 350 and display 355. An operator of electronic device 300 can use input 350 to enter data or input into electronic device 300. Input 350 can be a keyboard, touch screen, mouse, trackball, voice input or other device that can act as a user interface to allow the user to interact with electronic device 300. For example, input 350 can include voice recognition processing, thereby allowing the user to enter voice commands. In an example, input 350 can include a touch panel, a (digital) pen sensor, a key or an ultrasonic input device. The touch panel can, for example, recognize touch input in at least one scheme, such as a capacitive scheme, a pressure-sensitive scheme, an infrared scheme or an ultrasonic scheme. By providing additional input to processor 340, input 350 can be associated with sensor 365 and / or camera. In an embodiment, sensor 365 includes one or more inertial measurement units (IMUs) (such as accelerometers, gyroscopes and magnetometers), motion sensors, optical sensors, cameras, pressure sensors, heart rate sensors, altimeters, etc. Input 350 may include a control circuit. In a capacitive scheme, input 350 can recognize touch or proximity.

[0043] Display 355 may be a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED), an active matrix OLED (AMOLED), or other display capable of rendering text and / or graphics, such as from a website, video, game, image, etc. Display 355 may be resized to fit the HMD. Display 355 may be a single display screen or multiple display screens capable of creating a stereoscopic display. In an embodiment, display 355 is a heads-up display (HUD). Display 355 may display 3D objects, such as a 3D point cloud or mesh.

[0044] Memory 360 is coupled to processor 340. A portion of memory 360 may include RAM, and another portion of memory 360 may include flash memory or other ROM. Memory 360 may include a persistent storage device (not shown), which represents any structure capable of storing and facilitating retrieval of information (such as data, program code, and / or other suitable information). Memory 360 may include one or more components or devices that support long-term storage of data, such as a read-only memory, a hard drive, flash memory, or an optical disk. Memory 360 may include media content. Media content may include various types of media, such as images, videos, three-dimensional content, VR content, AR content, 3D point clouds, meshes, etc.

[0045] The electronic device 300 includes one or more sensors 365, which can measure physical quantities or detect the activation state of the electronic device 300 and convert the measured or detected information into electrical signals. For example, the sensor 365 may include one or more buttons for touch input, a camera, a gesture sensor, an IMU sensor (such as a gyroscope or gyro sensor and an accelerometer), an eye tracking sensor, an air pressure sensor, a magnetic sensor or a magnetometer, a grip sensor, a proximity sensor, a color sensor, a biophysical sensor, a temperature / humidity sensor, an illumination sensor, an ultraviolet (UV) sensor, an electromyogram (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an IR sensor, an ultrasonic sensor, an iris sensor, a fingerprint sensor, a color sensor (such as a red, green, and blue (RGB) sensor), etc. The sensor 365 may include a control circuit for controlling any sensor included therein.

[0046] As discussed in more detail below, one or more of these sensors 365 can be used to control a user interface (UI), detect UI input, determine the orientation of the user and the direction facing the user for three-dimensional content display recognition, etc. Any of these sensors 365 can be located within the electronic device 300, within an auxiliary device operably connected to the electronic device 300, within a headset configured to have the electronic device 300, or within a single device in which the electronic device 300 includes the headset.

[0047] The electronic device 300 can create media content, such as generating a virtual object or capturing (or recording) content through a camera. The electronic device 300 can encode the media content to generate a bitstream so that the bitstream can be transmitted directly to another electronic device, or such as through Figure 1 The electronic device 300 may directly receive the bit stream from another electronic device, or may transmit the bit stream to another electronic device indirectly through the network 102. Figure 1 The network 102 indirectly receives the bit stream.

[0048] although Figure 2 and Figure 3 An example of an electronic device is shown, but the Figure 2 and Figure 3 Make various changes. For example, Figure 2 and Figure 3 The various components in the can be combined, subdivided, or omitted, and additional components can be added as needed. As a specific example, processor 340 can be divided into multiple processors, such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs). In addition, as with computing and communications, electronic devices and servers can have a variety of configurations, and Figure 2 and Figure 3 The present disclosure is not limited to any particular electronic device or server.

[0049] Figure 4 A block diagram of an example environment architecture 400 for storing EVC decoder configuration information is shown according to an embodiment of the present disclosure. Figure 4 The embodiments shown are for illustration only. Other embodiments may be used without departing from the scope of the present disclosure.

[0050] like Figure 4 As shown, the example environment architecture 400 includes an encoder 410 and a decoder 450 communicating over a network.

[0051] In an embodiment, the encoder 410 and the decoder 450 may represent the server 104, Figure 1 The encoder 410 and the decoder 450 may include a processor similar to Figure 2 Server 200 and Figure 3The encoder 410 and the decoder 450 may be an internal component of the electronic device 300. In an embodiment, the encoder 410 and the decoder 450 may be a "cloud" of computers interconnected by one or more networks, wherein each network is a computing system that utilizes clustered computers and components to act as a single seamless resource pool when accessed through a network. In some embodiments, a portion of the components included in the encoder 410 or the decoder 450 may be included in different devices, such as multiple servers 104 or 200, multiple client devices 106-116, or other combinations of different devices. In an embodiment, the encoder 410 is operably connected to an electronic device or server, and the decoder 450 is operably connected to an electronic device. In an embodiment, the encoder 410 and the decoder 450 are the same device or are operably connected to the same device.

[0052] The encoder 410 receives data from a server such as a Figure 1 Server 104, Figure 2 The media content is received by another device such as a server 200 or an information repository such as a database. Encoder 410 sends frames representing the media content as an encoded bitstream. Bitstream 420 may be sent to an information repository such as a database or an electronic device including a decoder such as decoder 450 or decoder 450 itself.

[0053] Decoder 450 may receive a bitstream representing media content. The bitstream may include data representing EVC decoder configuration information. In an embodiment, decoder 450 may decode a bitstream including multiple frames and auxiliary information used in reconstructing media content. EVC decoder configuration information may be associated with green metadata 422.

[0054] The media preprocessor 412 can analyze and filter the content source, and the media encoder 414 can encode the content into a bitstream 420 for delivery. The bitstream 420 is delivered to the decoder 450 and decoded by the media decoder 452, where the output is rendered on the presentation subsystem 454 that implements the display process.

[0055] Green metadata 422 is extracted from the media encoder 414 or the media preprocessor 412. In both cases, the green metadata 422 is multiplexed or encapsulated in the conforming bitstream 420. Such green metadata 422 is used at the decoder 450 to reduce power consumption for video decoding and presentation. The bitstream 420 can be packetized and delivered to the decoder 450 for decoding and presentation. At the decoder 450, the metadata extractor 456 processes the packets and sends the green metadata 422 to the power optimization module 458 for effective power control. For example, the power optimization module 458 can interpret the green metadata 422, and then can apply appropriate operations to reduce the power consumption of the media decoder 452 when decoding the video and reduce the power consumption of the presentation subsystem 454 when rendering the video. In addition, the power optimization module 458 can collect receiver information, such as the remaining battery capacity, and send it to the encoder 410 as green feedback 424 to adjust the encoder operation to reduce power consumption. The EVC decoder can use complexity metric (CM) metadata to predict the decoding complexity of the bitstream 420 based on the changing operating frequency or voltage, and adjust the decoder power consumption accordingly. The syntax of EVC CM is shown in Table 1.

[0056]

Table 1

[0057]

[0058] The semantics of the EVC CM is defined below. The segmen t type variable specifies the type of upcoming segmen t to which the complexity metric applies and is defined in Table 2.

[0059]

Table 2

[0060]

[0061] The num_non_zero_4_cus field indicates the number of coding units whose width and height are 4 samples and have non-zero transform coefficient values ​​in the period where complexity metric is applied. The num_non_zero_8_cus field indicates the number of coding units whose width and height are 8 samples and have non-zero transform coefficient values ​​in the period where complexity metric is applied. The num_non_zero_16_cus field indicates the number of coding units whose width and height are 16 samples and have non-zero transform coefficient values ​​in the period where complexity metric is applied. The num_non_zero_32_cus field indicates the number of coding units whose width and height are 32 samples and have non-zero transform coefficient values ​​in the period where complexity metric is applied. The num_non_zero_64_cus field indicates the number of coding units whose width and height are 32 samples and have non-zero transform coefficient values ​​in the period where complexity metric is applied. The portion_fractional_prediction_sample field indicates the portion of the sample that needs to apply the fractional sample interpolation process when the complexity metric is applied. It is defined as follows:

[0062]

[0063] The num_count field indicates the number of pictures, slices, or NAL units to which the complexity metric applies when period_type is 1, 3, or 5. When period_type is 1, then num_count indicates the number of pictures in decoding order. When period_type is 3, then num_count indicates the number of slices in decoding order. When period_type is 5, then num_count indicates the number of NAL units in reception order.

[0064] The num_non_zero_4_cus[t] field indicates the number of coding units whose width and height are 4 samples and have non-zero transform coefficient values ​​in the period to which the (t+1)th complexity metric is applied. When period_type is 1, the period to which the metric applies is the tth picture in decoding order. When period_type is 3, the period to which the metric applies is the tth slice in decoding order. When period_type is 5, the period to which the metric applies is the tth NAL unit in reception order.

[0065] The num_non_zero_8_cus[t] field indicates the number of coding units whose width and height are 8 samples and have non-zero transform coefficient values ​​in the period to which the (t+1)th complexity metric is applied. When period_type is 1, the period to which the metric applies is the (t+1)th picture in decoding order. When period_type is 3, the period to which the metric applies is the (t+1)th slice in decoding order. When period_type is 5, the period to which the metric applies is the (t+1)th NAL unit in reception order.

[0066] The num_non_zero_16_cus[t] field indicates the number of coding units whose width and height are 16 samples and have non-zero transform coefficient values ​​in the period to which the (t+1)th complexity metric is applied. When period_type is 1, the period to which the metric applies is the (t+1)th picture in decoding order. When period_type is 3, the period to which the metric applies is the (t+1)th slice in decoding order. When period_type is 5, the period to which the metric applies is the (t+1)th NAL unit in reception order.

[0067] The num_non_zero_32_cus[t] field indicates the number of coding units whose width and height are 32 samples and have non-zero transform coefficient values ​​in the period to which the (t+1)th complexity metric is applied. When period_type is 1, the period to which the metric applies is the (t+1)th picture in decoding order. When period_type is 3, the period to which the metric applies is the (t+1)th slice in decoding order. When period_type is 5, the period to which the metric applies is the (t+1)th NAL unit in reception order.

[0068] num_non_zero_64_cus[t] indicates the number of coding units whose width and height are 64 samples and have non-zero transform coefficient values ​​in the period to which the (t+1)th complexity metric is applied. When period_type is 1, the period to which the metric applies is the (t+1)th picture in decoding order. When period_type is 3, the period to which the metric applies is the (t+1)th slice in decoding order. When period_type is 5, the period to which the metric applies is the (t+1)th NAL unit in reception order.

[0069] The portion_fractional_prediction_sample[t] field indicates the portion of the sample that requires the application of the fractional sample interpolation process when applying the (t+1)th complexity metric. When period_type is 1, the period to which the metric applies is the (t+1)th picture in decoding order. When period_type is 3, the period to which the metric applies is the (t+1)th slice in decoding order. When period_type is 5, the period to which the metric applies is the (t+1)th NAL unit in reception order.

[0070] The syntax of EVC CM can be expressed as shown in Table 3.

[0071]

Table 3

[0072]

[0073]

[0074] The maximum number of pixels and coding units depends on the size of the picture to which the complexity metric is applied. Since the maximum size of the picture is indicated by the level to which the CVS conforms, the length of the syntax element indicating the number of pixels and coding units is determined by the level. In addition, when the length of the syntax element indicating the number of coding units conforming to the CVS of the baseline profile becomes smaller as the width and height of the coding unit become larger than the maximum number of coding units, the width and height of the coding unit are taken into account.

[0075] The -uk(v) field is an unsigned integer and its length is determined by the value of the level_idc field in the SPS used by the CVS to which this SEI message is applied and the value of k assigned to each field based on the size of the unit of count. The length of the field according to each value of both level_idc and k is shown in Table 4.

[0076]

Table 4

[0077]

[0078] The profile_idc field indicates the profile of the CVS associated with this SEI message.

[0079] The period_type field specifies the type of the upcoming period to which the complexity metric applies and is defined in Table 5.

[0080] Table 5

[0081]

[0082] The num_non_zero_4_cus field indicates the number of coding units whose width and height are not greater than 4 samples and have non-zero transform coefficient values ​​in the period to which the complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS used by the CVS of the SEI message to which this field is applied, and the value of k is equal to 4.

[0083] The num_non_zero_8_cus field indicates the number of coding units whose width and height are 8 samples and have non-zero transform coefficient values ​​in the period to which the complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS used by the CVS of the SEI message to which this field is applied, and the value of k is equal to 8.

[0084] The num_non_zero_16_cus field indicates the number of coding units whose width and height are 16 samples and have non-zero transform coefficient values ​​in the period to which the complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS used by the CVS of the SEI message to which this field is applied, and the value of k is equal to 16.

[0085] The num_non_zero_32_cus field indicates the number of coding units whose width and height are 32 samples and have non-zero transform coefficient values ​​in the period to which the complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS used by the CVS of the SEI message to which this field is applied, and the value of k is equal to 32.

[0086] The num_non_zero_64_cus field indicates the number of coding units whose width and height are 32 samples and have non-zero transform coefficient values ​​in the period to which the complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS used by the CVS of the SEI message to which this field is applied, and the value of k is equal to 64.

[0087] The portion_fractional_prediction_sample field indicates the portion of the sample that requires the fractional sample interpolation process to be applied when applying the complexity metric. It is defined as follows:

[0088]

[0089] The num_count field indicates the number of pictures, slices, or NAL units to which the complexity metric applies when period_type is 1, 3, or 5. When period_type is 1, then num_count indicates the number of pictures in decoding order. When period_type is 3, then num_count indicates the number of slices in decoding order. When period_type is 5, then num_count indicates the number of NAL units in reception order.

[0090] The num_non_zero_4_cus[t] field indicates the number of coding units whose width and height are not greater than 4 samples and which have non-zero transform coefficient values ​​in the period to which the (t+1)th complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS used by the CVS of the SEI message to which this field is applied, and the value of k is equal to 4. When period_type is 1, the period to which the metric applies is the tth picture in decoding order. When period_type is 3, the period to which the metric applies is the tth slice in decoding order. When period_type is 5, the period to which the metric applies is the tth NAL unit in reception order.

[0091] The num_non_zero_8_cus[t] field indicates the number of coding units whose width and height are 8 samples and have non-zero transform coefficient values ​​in the period to which the (t+1)th complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS used by the CVS of the SEI message to which this field applies, and the value of k is equal to 8. When period_type is 1, the period to which this metric applies is the (t+1)th picture in decoding order. When period_type is 3, the period to which this metric applies is the (t+1)th slice in decoding order. When period_type is 5, the period to which this metric applies is the (t+1)th NAL unit in reception order.

[0092] The num_non_zero_16_cus[t] field indicates the number of coding units whose width and height are 16 samples and have non-zero transform coefficient values ​​in the period to which the (t+1)th complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS used by the CVS of the SEI message to which this field applies, and the value of k is equal to 16. When period_type is 1, the period to which the metric applies is the (t+1)th picture in decoding order. When period_type is 3, the period to which the metric applies is the (t+1)th slice in decoding order. When period_type is 5, the period to which the metric applies is the (t+1)th NAL unit in reception order.

[0093] The num_non_zero_32_cus[t] field indicates the number of coding units whose width and height are 32 samples and have non-zero transform coefficient values ​​in the period to which the (t+1)th complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS used by the CVS of the SEI message to which this field applies, and the value of k is equal to 32. When period_type is 1, the period to which this metric applies is the (t+1)th picture in decoding order. When period_type is 3, the period to which this metric applies is the (t+1)th slice in decoding order. When period_type is 5, the period to which this metric applies is the (t+1)th NAL unit in reception order.

[0094] The num_non_zero_64_cus[t] field indicates the number of coding units whose width and height are 64 samples and have non-zero transform coefficient values ​​in the period to which the (t+1)th complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS used by the CVS of the SEI message to which this field is applied, and the value of k is equal to 64. When period_type is 1, the period to which the metric applies is the (t+1)th picture in decoding order. When period_type is 3, the period to which the metric applies is the (t+1)th slice in decoding order. When period_type is 5, the period to which the metric applies is the (t+1)th NAL unit in reception order.

[0095] The portion_fractional_prediction_sample[t] field indicates the portion of samples in the period for which the (t+1)th complexity metric is applied that requires the application of the fractional sample interpolation process. When period_type is 1, the period for which the metric is applied is the (t+1)th picture in decoding order. When period_type is 3, the period for which the metric is applied is the (t+1)th slice in decoding order. When period_type is 5, the period for which the metric is applied is the (t+1)th NAL unit in reception order.

[0096] The num_non_zero_samples field indicates the number of non-zero transform coefficients in the period to which the complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS of the CVS to which the SEI message associated with this field is applied, this field is associated with the SEI message, and the value of k is equal to 1.

[0097] The num_affine_samples field indicates the number of pixels to which affine model-based motion compensation is applied in a period in which complexity metrics are applied. The length of this field is determined by the value of the level_idc field in the SPS of the CVS of the SEI message to which this field is associated, and the value of k is equal to 1.

[0098] The num_dmvr_samples field indicates the number of pixels to which decoder-side motion vector refinement is applied in the period in which the complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS of the CVS to which the field is associated with the SEI message, and the value of k is equal to 1.

[0099] The num_alf_samples field indicates the number of pixels to which the adaptive loop filter is applied in the period in which the complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS of the CVS of the SEI message to which this field is associated, and the value of k is equal to 1.

[0100] The num_deblocking_filter_samples field indicates the number of pixels to which the deblocking filter is applied in the period in which the complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS of the CVS of the SEI message to which this field is associated, and the value of k is equal to 1.

[0101] The num_htdf_samples field indicates the number of pixels to which the hadamard transform domain filter is applied in the period in which the complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS of the CVS of the SEI message to which this field is associated, and the value of k is equal to 1.

[0102] The num_non_zero_samples[t] field indicates the number of non-zero transform coefficients in the period to which the (t+1)th complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS used by the CVS of the SEI message to which this field is applied, and the value of k is equal to 1. When period_type is 1, the period to which the metric applies is the (t+1)th picture in decoding order. When period_type is 3, the period to which the metric applies is the (t+1)th slice in decoding order. When period_type is 5, the period to which the metric applies is the (t+1)th NAL unit in reception order.

[0103] The num_affine_samples[t] field indicates the number of pixels to which affine model-based motion compensation is applied in the period in which the (t+1)th complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS used by the CVS of the SEI message to which this field is applied, and the value of k is equal to 1. When period_type is 1, the period in which the metric is applied is the (t+1)th picture in decoding order. When period_type is 3, the period in which the metric is applied is the (t+1)th slice in decoding order. When period_type is 5, the period in which the metric is applied is the (t+1)th NAL unit in reception order.

[0104] The num_dmvr_samples[t] field indicates the number of pixels to which decoder-side motion vector refinement is applied in the period over which the (t+1)th complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS used by the CVS of the SEI message to which this field applies, and the value of k is equal to 1. When period_type is 1, the period over which the metric applies is the (t+1)th picture in decoding order. When period_type is 3, the period over which the metric applies is the (t+1)th slice in decoding order. When period_type is 5, the period over which the metric applies is the (t+1)th NAL unit in reception order.

[0105] The num_alf_samples[t] field indicates the number of pixels to which the adaptive loop filter is applied in the period over which the (t+1)th complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS used by the CVS of the SEI message to which this field is applied, and the value of k is equal to 1. When period_type is 1, the period over which the metric is applied is the (t+1)th picture in decoding order. When period_type is 3, the period over which the metric is applied is the (t+1)th slice in decoding order. When period_type is 5, the period over which the metric is applied is the (t+1)th NAL unit in reception order.

[0106] The num_deblocking_filter_samples[t] field indicates the number of pixels to which the deblocking filter is applied in the period in which the (t+1)th complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS used by the CVS of the SEI message to which this field is applied, and the value of k is equal to 1. When period_type is 1, the period in which the metric applies is the (t+1)th picture in decoding order. When period_type is 3, the period in which the metric applies is the (t+1)th slice in decoding order. When period_type is 5, the period in which the metric applies is the (t+1)th NAL unit in reception order.

[0107] The num_htdf_samples[t] field indicates the number of pixels to which the hadamard transform domain filter is applied in the period over which the (t+1)th complexity metric is applied. The length of this field is determined by the value of the level_idc field in the SPS used by the CVS of the SEI message to which this field is applied, and the value of k is equal to 1. When period_type is 1, the period over which the metric is applied is the (t+1)th picture in decoding order. When period_type is 3, the period over which the metric is applied is the (t+1)th slice in decoding order. When period_type is 5, the period over which the metric is applied is the (t+1)th NAL unit in reception order.

[0108] although Figure 4 An example of an encoder and a decoder is shown, but the Figure 4 Make various changes. For example, Figure 4 The various components in the can be combined, subdivided, or omitted, and additional components can be added as needed. As a specific example, encoder 410 or decoder 450 can be divided into multiple components. In addition, as with computing and communication, encoders and decoders can have a variety of configurations, and Figure 4The present disclosure is not limited to any particular encoder or decoder.

[0109] Figure 5 An example method 500 for EVC decoding complexity metric according to the present disclosure is shown. For ease of explanation, Figure 5 The method 500 is described as using Figure 1 The method 500 is performed by the electronic device 101. However, the method 500 may be used with any other suitable system and any other suitable electronic device.

[0110] like Figure 5 As shown, the electronic device 101 may receive a bitstream including complexity metric metadata at step 502. The CM metadata may include information about the number of non-zero coding units, a number of pixels to which affine model-based motion compensation is applied, a number of pixels to which decoder-side motion vector refinement is applied, a number of pixels to which an adaptive loop filter is applied, a number of pixels to which a deblocking filter is applied, and a number of pixels to which a hadamard transform domain filter is applied.

[0111] At step 504, the electronic device may extract information in the complexity metric metadata from the bitstream. For each coding unit size in a plurality of coding unit sizes, the information may include a direct indication of the number of coding units in the bitstream having non-zero transform coefficients. The direct indication of each CU size may be included in a corresponding field that varies in length based on the CU size and the level indicated in the sequence parameter set. For a given level indicated in the sequence parameter set, the length of the field for a smaller CU size is greater than the length of the field for a larger CU size. For a given CU size, the length of the field for a lower level is greater than the length of the field for a higher level.

[0112] At step 506, the electronic device may predict the decoding complexity of the bitstream based on the CM metadata. The prediction of the coding complexity may be based on the changed operating frequency or the changed voltage. The prediction of the coding complexity may be used to adjust the decoder power consumption.

[0113] although Figure 5 An example of a method 500 for EVC decoding complexity metric is shown, but may be used for Figure 5 For example, although shown as a series of steps, Figure 5 The various steps in may overlap, occur in parallel, or occur any number of times.

[0114] Although the present disclosure has been described using exemplary embodiments, various changes and modifications may be suggested to those skilled in the art. The present disclosure is intended to cover these changes and modifications that fall within the scope of the appended claims. The description in this application should not be interpreted as implying that any particular element, step or function is an essential element that must be included within the scope of the claims. The scope of the patent subject matter is defined by the claims.

[0115] According to an embodiment of the present disclosure, a device is provided. The device may include a communication interface and a processor operably coupled to the communication interface. The processor may be configured to receive a bitstream including complexity metric (CM) metadata via the communication interface. The processor may be configured to extract information in the CM metadata from the bitstream for each CU size in a plurality of CU sizes, wherein the information includes a direct indication of the number of coding units (CUs) having non-zero transform coefficients in the bitstream. The processor may be configured to predict the decoding complexity of the bitstream based on the information in the CM metadata. The direct indication of each CU size may be included in a corresponding field, the length of which varies based on the CU size and the level indicated in the sequence parameter set.

[0116] According to an embodiment of the present disclosure, for a given level indicated in a sequence parameter set, the length of the field for a smaller CU size may be greater than the length of the field for a larger CU size.

[0117] According to an embodiment of the present disclosure, for a given CU size, the length of a field for a lower level may be greater than the length of a field for a higher level.

[0118] According to an embodiment of the present disclosure, CM metadata may include information about the number of non-zero coding units.

[0119] According to an embodiment of the present disclosure, the CM metadata may include a ratio of fractional prediction samples.

[0120] According to an embodiment of the present disclosure, CM metadata may include a plurality of pixels to which motion compensation based on an affine model is applied.

[0121] According to an embodiment of the present disclosure, CM metadata may include a plurality of pixels where decoder-side motion vector refinement is applied.

[0122] According to an embodiment of the present disclosure, CM metadata may include a plurality of pixels in which an adaptive loop filter is applied.

[0123] According to an embodiment of the present disclosure, CM metadata may include a plurality of pixels in which a deblocking filter is applied.

[0124] According to an embodiment of the present disclosure, the CM metadata may include a plurality of pixels in which a Hadamard transform domain filter is applied.

[0125] According to an embodiment of the present disclosure, a method is provided. The method may include receiving a bitstream including complexity metric (CM) metadata. The method may include extracting information in the CM metadata from the bitstream for each CU size in a plurality of CU sizes, wherein the information includes a direct indication of the number of coding units (CUs) having non-zero transform coefficients in the bitstream. The method may include predicting the decoding complexity of the bitstream based on the information in the CM metadata. The method may include wherein the direct indication for each CU size is included in a corresponding field, the length of which varies based on the CU size and the level indicated in the sequence parameter set.

[0126] According to an embodiment of the present disclosure, for a given level indicated in a sequence parameter set, the length of the field for a smaller CU size may be greater than the length of the field for a larger CU size.

[0127] According to an embodiment of the present disclosure, for a given CU size, the length of a field for a lower level may be greater than the length of a field for a higher level.

[0128] According to an embodiment of the present disclosure, CM metadata may include information about the number of non-zero coding units.

[0129] According to an embodiment of the present disclosure, the CM metadata may include a ratio of fractional prediction samples.

[0130] According to an embodiment of the present disclosure, CM metadata may include a plurality of pixels to which motion compensation based on an affine model is applied.

[0131] According to an embodiment of the present disclosure, CM metadata may include a plurality of pixels where decoder-side motion vector refinement is applied.

[0132] According to an embodiment of the present disclosure, CM metadata may include a plurality of pixels in which an adaptive loop filter is applied.

[0133] According to an embodiment of the present disclosure, CM metadata may include a plurality of pixels in which a deblocking filter is applied.

[0134] According to an embodiment of the present disclosure, the CM metadata may include a plurality of pixels in which a Hadamard transform domain filter is applied.

[0135] According to an embodiment of the present disclosure, a computer-readable medium storing a bitstream is provided. The bitstream may be decoded by the above method. The bitstream may include complexity metric (CM) metadata.

Claims

1. A device comprising: Communication interface; as well as A processor operably coupled to the communication interface, the processor configured to: receiving, via a communication interface, a bitstream including complexity metric (CM) metadata; For each CU size in a plurality of CU sizes, extracting information in CM metadata from a bitstream, wherein the information includes a direct indication of a number of coding units (CUs) having non-zero transform coefficients in the bitstream; and Predict the decoding complexity of the bitstream based on the information in the CM metadata; Therein, a direct indication of each CU size is included in a corresponding field, the length of which varies based on the CU size and the level indicated in the sequence parameter set.

2. The device according to claim 1, wherein: For a given level indicated in a sequence parameter set, the length of the field for smaller CU sizes is greater than the length of the field for larger CU sizes.

3. The device according to any one of claims 1 to 2, wherein: For a given CU size, the length of the fields for the lower levels is larger than the length of the fields for the higher levels.

4. The device according to any one of claims 1 to 3, wherein: The CM metadata includes information about the number of non-zero coding units.

5. The device according to any one of claims 3 to 4, wherein: CM metadata includes the ratio of the score prediction samples.

6. The device according to any one of claims 1 to 5, wherein: The CM metadata includes a number of pixels where motion compensation based on an affine model is applied.

7. The device according to any one of claims 1 to 6, wherein: The CM metadata includes a number of pixels where decoder-side motion vector refinement is applied.

8. The device according to any one of claims 1 to 7, wherein: The CM metadata includes a number of pixels where the adaptive loop filter is applied.

9. The device according to any one of claims 1 to 8, wherein: The CM metadata includes a number of pixels where the deblocking filter is applied.

10. The device according to any one of claims 1 to 9, wherein: The CM metadata includes a number of pixels where the Hadamard transform domain filter is applied.

11. A method comprising: receiving a bitstream including complexity measure (CM) metadata; For each CU size in a plurality of CU sizes, extracting information in CM metadata from a bitstream, wherein the information includes a direct indication of a number of coding units (CUs) having non-zero transform coefficients in the bitstream; and Predict the decoding complexity of the bitstream based on the information in the CM metadata; Therein, a direct indication of each CU size is included in a corresponding field, the length of which varies based on the CU size and the level indicated in the sequence parameter set.

12. The method according to claim 11, wherein: For a given level indicated in a sequence parameter set, the length of the field for smaller CU sizes is greater than the length of the field for larger CU sizes.

13. The method according to any one of claims 11 to 12, wherein: For a given CU size, the length of the fields for the lower levels is larger than the length of the fields for the higher levels.

14. The method according to any one of claims 11 to 13, wherein: The CM metadata includes information about the number of non-zero coding units.

15. The method according to any one of claims 13 to 14, wherein: CM metadata includes the ratio of the score prediction samples.

Images