Fine-tuning video data in streaming applications

Abstract

A method for fine-tuning video data in a low-latency streaming application, the method comprising: obtaining a quality of experience identifier defining a group of constraint flags involved in obtaining a quality of experience, each constraint flag being used to activate or deactivate an associated encoding tool of a video encoder, the identifier specifying a value for each constraint flag that indicates activation or deactivation of the associated encoding tool; sending the identifier to a server; and receiving from the server a video stream that complies with the identifier.

Description

[Technical field]

[0001] At least one of the present embodiments generally relates to methods, apparatus, and signals that enable fine-tuning of video data in low-latency streaming applications. [Background technology]

[0002] Streaming applications have grown tremendously in recent years: initially, they addressed pure audio and / or video applications, but they are now reaching new areas, such as that of gaming applications.

[0003] Gaming applications are indeed migrating from on-device to cloud-hosted applications. Cloud gaming allows the game rendering process to be partially offloaded to several remote game servers located in the cloud.

[0004] Fig. 1A represents a schematic representation of a cloud gaming system. Basically, a game engine 10 and a 3D graphics rendering 11, which require expensive and power-consuming devices, are implemented by a server 1 in the cloud. The generated pictures are then classically encoded into a video stream using a normal / standard video encoder 12 and transmitted over a network 3 to a user gaming system 2, which is for example a PC or a game console. Here, we assume that the user gaming system comprises an input device (such as a joypad or a keyboard). The video stream is then decoded on the user gaming system 2 side using a normal / standard video decoder 20 for rendering on a display device. An additional lightweight module 21, hereafter called a registration module, is responsible for managing gamer interaction commands (i.e. for registering user actions).

[0005] One important factor for user comfort in gaming applications is the so-called motion-to-photon latency, i.e. the latency between a user action (motion) and the display of the result of this action on a display device (photons).

[0006] FIG. 1B illustrates a schematic of a typical motion-to-photon path in a cloud gaming application.

[0007] The steps described in relation to FIG. 1B are implemented by the cloud gaming system of FIG. 1A and require coordination between the server 1 and the user gaming system 2 (ie, the client system).

[0008] In step 100, on the game system 2 side, user actions are registered by the input device and sent to a main processing module.

[0009] In step 101 , information representative of a user action is transmitted over the network 3 to the server 1 .

[0010] In step 102, the registered actions are used by game engine 10 to calculate the next game state (or next game states). The game state includes the user state (such as position) and any other entity state that may be calculated by game engine 10, or the external state in the case of a multiplayer game.

[0011] In step 103, from the game state, the picture rendering is calculated.

[0012] The rendering is followed in step 104 by video encoding of the rendered pictures in a video stream by video encoder 12 .

[0013] The video stream generated by the video encoder 12 is then transmitted, in step 105 , over the network 3 to the user game system 2 and decoded, in step 106 , by the video decoder 20 .

[0014] The resulting picture is then displayed on a display device in step 107.

[0015] Each of the above steps introduces a processing latency. In FIG. 1B, boxes with a dotted background represent steps that introduce a latency due to hardware computation. Generally, this latency is fixed, small, and can be easily changed. Boxes with a white background represent steps that introduce a latency due to software computation. Generally, this latency is longer and can be dynamically adapted.

[0016] In the motion-to-photon path of Figure 1B, the encoding and decoding processes introduce significant latencies, but these latencies are typical latencies that can be dynamically adapted according to application constraints or user desires.

[0017] Gaming is an area where users (gamers) are used to tune any parameters made available to them to enjoy the game with the highest possible quality or to get the most responsive and fast user experience. Gamers generally trade visual quality for less game latency. However, in cloud gaming, a very limited amount of parameters are made available to the end user. Tools aimed at fine-grained control of the computational cost, latency, and rendering of video codecs in cloud gaming applications would be beneficial. Similarly, other applications such as head mounted display (HMD) based applications, or augmented reality / virtual reality (AR / VR) glasses based applications, as well as any low latency streaming applications, would benefit from this fine-grained control / tuning of video codecs.

[0018] It is desirable to propose a solution that makes it possible to overcome the above problems, in particular a solution that allows fine-tuning of the codec in streaming applications, which solution will be especially adapted to the cloud gaming context in order to allow the gamer to find the best compromise between the responsiveness of the game and the quality of the display that best suits him. Summary of the Invention

[0019] In a first aspect, one or more of the embodiments of the present invention provide a method comprising: obtaining a desired quality of experience identifier defining a group of constraint flags involved in obtaining a desired quality of experience, each constraint flag being used to activate or deactivate an associated encoding tool of a video encoder, the identifier specifying a value for each constraint flag that indicates activation or deactivation of the associated encoding tool; transmitting the identifier to a remote server; and receiving a video stream from the remote server that complies with the identifier.

[0020] In one embodiment, the identifier is further used to define a decoding decision that allows the video decoder to adjust the video decoder to skip at least one decoding process specified in the video stream or to add at least one additional decoding process to the decoding processes specified in the video stream.

[0021] In one embodiment, the identifier is transmitted using the Session Description Protocol.

[0022] In one embodiment, the identifier is obtained after a stage of capabilities exchange with the remote server, which includes receiving a Session Description Protocol message from the remote server specifying a set of encoding tools that may be activated or deactivated, or a quality of experience that may be selected.

[0023] In one embodiment, before obtaining the identifier, the method includes obtaining a description of a set of qualities of experience that may be selected in the SEI message.

[0024] In one embodiment, each quality of experience in the set is associated with a corresponding group of constraint flags and / or at least one encoding decision and / or at least one decoding decision.

[0025] In a second aspect, one or more of the embodiments of the present invention provide a method including obtaining a desired quality of experience identifier defining a group of constraint flags involved in obtaining a desired quality of experience, each constraint flag being used to activate or deactivate an associated encoding tool of a video encoder, the identifier specifying a value for each constraint flag that indicates activation or deactivation of the associated encoding tool; obtaining an encoded video stream that conforms to the identifier; and transmitting the encoded video stream to a remote system.

[0026] In one embodiment, the encoded video stream is obtained by adjusting a video encoder based on the identifier.

[0027] In one embodiment, adjusting includes activating or deactivating coding tools associated with the group of constraint flags depending on the value specified by the identifier.

[0028] In one embodiment, the identifier is further used to specify at least one encoding decision that allows defining a specific implementation of a video encoder, independent of the constraint flags and any profile.

[0029] In one embodiment, the identifier is obtained from the remote system in a Session Description Protocol message.

[0030] In one embodiment, the identifier is obtained after a stage of capabilities exchange with the remote system, which includes sending a Session Description Protocol message to the remote system specifying a set of encoding tools that may be activated or deactivated, or a quality of experience that may be selected.

[0031] In one embodiment, before obtaining the identifier, the method includes sending to the remote system a description of a set of quality of experience that may be selected in the SEI message.

[0032] In one embodiment, each quality of experience in the set is associated with a corresponding group of constraint flags and / or at least one encoding decision and / or at least one decoding decision.

[0033] In a third aspect, one or more of the embodiments of the present invention provide a signal including a desired quality of experience identifier defining a group of constraint flags involved in obtaining a desired quality of experience, each constraint flag being used to activate or deactivate an associated encoding tool of a video encoder, and the identifier specifying a value for each constraint flag indicating the activation or deactivation of the associated encoding tool.

[0034] In a fourth aspect, one or more of the embodiments of the present invention provide a device comprising electronic circuitry adapted to: obtain a desired quality of experience identifier defining a group of constraint flags involved in obtaining the desired quality of experience, each constraint flag being used to activate or deactivate an associated encoding tool of a video encoder, the identifier specifying a value for each constraint flag that indicates activation or deactivation of the associated encoding tool; transmit the identifier to a remote server; and receive from the remote server a video stream that complies with the identifier.

[0035] In one embodiment, the identifier is further used to define a decoding decision that allows the video decoder to adjust the video decoder to skip at least one decoding process specified in the video stream or to add at least one additional decoding process to the decoding processes specified in the video stream.

[0036] In one embodiment, the identifier is transmitted using the Session Description Protocol.

[0037] In one embodiment, the identifier is obtained after a stage of capabilities exchange with the remote server, which includes receiving a Session Description Protocol message from the remote server specifying a set of encoding tools that may be activated or deactivated, or a quality of experience that may be selected.

[0038] In one embodiment, the electronic circuitry is further adapted to obtain, prior to obtaining the identifier, a description of a set of qualities of experience that can be selected in the SEI message.

[0039] In one embodiment, each quality of experience in the set is associated with a corresponding group of constraint flags and / or at least one encoding decision and / or at least one decoding decision.

[0040] In a fifth aspect, one or more of the embodiments of the present invention provide a device comprising electronic circuitry adapted to: obtain a desired quality of experience identifier defining a group of constraint flags involved in obtaining the desired quality of experience, each constraint flag being used to activate or deactivate an associated encoding tool of a video encoder, the identifier specifying a value for each constraint flag that indicates activation or deactivation of the associated encoding tool; obtain an encoded video stream that complies with the identifier; and transmit the encoded video stream to a remote system.

[0041] In one embodiment, the encoded video stream is obtained by adjusting a video encoder based on the identifier.

[0042] In one embodiment, adjusting includes activating or deactivating coding tools associated with the group of constraint flags depending on the value specified by the identifier.

[0043] In one embodiment, the identifier is further used to specify at least one encoding decision that allows defining a specific implementation of a video encoder, independent of the constraint flags and any profile.

[0044] In one embodiment, the identifier is obtained from the remote system in a Session Description Protocol message.

[0045] In one embodiment, the identifier is obtained after a stage of capabilities exchange with the remote system, which includes sending a Session Description Protocol message to the remote system specifying a set of encoding tools that may be activated or deactivated, or a quality of experience that may be selected.

[0046] In one embodiment, before obtaining the identifier, the method includes sending to the remote system a description of a set of quality of experience that may be selected in the SEI message.

[0047] In one embodiment, each quality of experience in the set is associated with a corresponding group of constraint flags and / or at least one encoding decision and / or at least one decoding decision.

[0048] In a sixth aspect, one or more of the present embodiments provide a computer program comprising program code instructions for implementing a method according to the first or second aspect.

[0049] In a seventh aspect, one or more of the present embodiments provide a non-transitory information storage medium storing program code instructions for implementing a method according to the first or second aspect. [Brief description of the drawings]

[0050] [Figure 1A] 1 illustrates an example of a context in which some embodiments may be implemented. [Figure 1B] Illustrates the concept of motion-to-photon path. [Diagram 2]1 illustrates generally a first embodiment of a method for enabling codec fine-tuning in streaming applications. [Diagram 3] 2 illustrates a schematic representation of a second embodiment of a method for enabling codec fine-tuning in streaming applications. [Figure 4] 4 illustrates a third embodiment of a method for enabling codec fine-tuning in streaming applications; [Figure 5A] 1 illustrates generally an example of a hardware architecture of a processing module in which various aspects and embodiments may be implemented; [Figure 5B] 1 illustrates a block diagram of an example of a gaming system in which various aspects and embodiments may be implemented. [Figure 5C] 1 illustrates a block diagram of an example of a server in which various aspects and embodiments may be implemented. [Figure 6] 2 illustrates an example of the division that a picture of pixels of an original video may undergo. [Figure 7] 2 illustrates generally a method for encoding a video stream executed by an encoding module; [Figure 8] 2 illustrates generally a method for decoding a video stream executed by a decoding module; [Figure 9] 1 illustrates a streaming session establishment process between a client and a server using RTP / RTSP session-based streaming according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0051] In the following, we present examples of embodiments that allow fine-tuning of codecs in streaming applications, these solutions being particularly adapted to low-latency streaming applications such as cloud gaming applications. In the context of streaming and / or gaming, the following tuning solutions can be mentioned: ● Adjusting device and network settings: Examples of device settings that are commonly adjusted by gamers are updating the graphics card drivers, disabling the native graphics card when a more powerful graphics card is available on the device, optimizing the hardware (PC and router), connecting to Ethernet instead of WiFi, adjusting router parameters, changing network settings, and selecting a server closer to the user. ● Adjusting Game Settings: Examples of in-game settings that can be adjusted (typically to increase frames per second (fps)): ○ Reducing the resolution, ○ Reducing the amount of detail in a picture; o Disable or change anti-aliasing, motion blur modes, and other advanced rendering parameters that may be implemented on the device. ● Developer Settings: Tools are emerging to help game developers monitor the gamer experience in order to adapt their games and development. FPS and CPU usage on the device are tracked and aggregated based on annotations (anchors) built into the gameplay by the developer. One example of such a tool is the Android Performance Tuner, described at https: / / developer.android.com / games / sdk / performance-tuner. ● Video Codec Profile: A video codec profile defines a fixed set of tools that may be used by a video encoder and shall be supported by a video decoder. These profiles are useful to ensure interoperability of devices across multiple services. In a closed OTT (over-the-top) ecosystem, a private codec or a private profile may be loaded along with the application on the endpoint. However, interoperability with other services is not a requirement, which may be an inhibitor or even a technical burden for demanding applications such as cloud gaming, as it is computationally expensive on end devices that do not benefit from a hardware implementation. ● VVC flags: In recent video codecs called VVC (H.266, ISO / IEC 23090-3, MPEG-I Part 3 (Versatile Video Coding)), coding tools can be enabled / disabled by high-level syntax elements (called constraint flags) to indicate that certain coding tools were not used by the encoder for a given bitstream. In VVC, constraint flags are collected in a data structure called general_constraints_info, according to which a bitstream can be labeled as not using certain tools, which among other things allows resource allocation in the decoder implementation. ● Capability Exchange: For streaming services and file downloads, an exchange of multimedia capability information takes place between end devices and servers. This allows adapting the content to the device and / or network capabilities by using Quality Of Experience (QoE). These device capability information are usually specified as part of the handshake protocol and transport protocol format. Example: o 3GPP TS 23.234 "Transparent end-to-end Packet-switched Streaming Service (PSS)" specifies a session establishment process where device capabilities are exchanged followed by SDP (Session Description Protocol) based communication that enables the server to provide a wide range of devices with content suitable for these devices. However, these exchanges of capability information are not related to VVC restriction flags nor to desired user experience quality settings. o The 3GPP SA4 MTSI specification (TS26.114) uses RTP (Real-Time Transport Protocol) for transporting media data, as well as the SDP mechanisms defined in RTP / AVPF (RFC4585) and SDPCapNeg (RFC5939). Very little information is exchanged related to the desired user quality of experience. o The 3GPP 5GMS specification (5G Multimedia Streaming) relies on MPEG DASH (Dynamic Adaptive Streaming over HTTP / ISO / IEC 23009-1:201x) or ISO file formats to transmit media over HTTP. Therefore, while content selection can be tailored, the use of these transport protocols in the 5GMS specification is not currently suitable for low codec fine-tuning in latency applications such as gaming applications.

[0052] Solutions exist for controlling codecs, as seen above with respect to video codec profiles and constraint flags, however, in their current form, these solutions are not adapted to allow users to fine-tune the codec in low-latency streaming applications, such as cloud gaming applications.

[0053] FIG. 5A illustrates generally an example of a hardware architecture of a processing module 500 that may implement modules of the server 1, such as, for example, the video encoder 12 and / or the game engine 10 and / or the 3D graphics renderer 11, or modules of the game system 2, such as, for example, the video decoder 20 and / or the registration module 21.

[0054] The processing module 500 includes, as non-limiting examples, a processor or CPU (Central Processing Unit) 5000, including one or more microprocessors, general purpose computers, special purpose computers, and processors based on multi-core architectures, connected by a communication bus 5005; a random access memory (RAM) 5001; a read only memory (ROM) 5002; and a memory array (RAM) 5003, including an electrically erasable programmable read-only memory (EEPROM), a read only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash, a magnetic disk drive, and / or an optical disk drive, or a secure digital (SD) card reader and / or a hard disk drive. The system includes a storage unit 5003, which may include non-volatile and / or volatile memory, including but not limited to a storage media reader and / or a network accessible storage device, such as a storage drive (HDD), and at least one communication interface 5004 for exchanging data with other modules, devices, or equipment. The communication interface 5004 may include, but is not limited to, a transceiver configured to transmit and receive data via a communication channel (or network) 3. The communication interface 5004 may include, but is not limited to, a modem or a network card.

[0055] If the processing module 500 implements the video decoder 20, the communications interface 5004 may, for example, enable the processing module 500 to receive an encoded video stream and provide a sequence of decoded pictures.

[0056] If the processing module 500 implements the registration module 21, the communication interface 5004 enables, for example, the processing module 500 to receive user actions or parameters selected by a user and provide data representing these user actions and parameters to the game engine 10 and / or the video decoder 20.

[0057] If the processing module 500 implements a video encoder 12, the communications interface 5004 may, for example, enable the processing module 500 to receive pictures generated by a 3D graphics renderer 11 and provide an encoded video stream representing these pictures.

[0058] If the processing module 500 implements a game engine 10, the communications interface 5004 may, for example, enable the processing module 500 to receive data representing user actions and selected parameters, and provide this data to a 3D graphics renderer and / or a video encoder 12.

[0059] If the processing module 500 implements a 3D graphics renderer 11, the communications interface 5004 may, for example, enable the processing module 500 to receive data representing user actions and / or selected parameters and to transmit the generated pictures to the video encoder 12 so that the processing module 500 can generate corresponding pictures.

[0060] The processor 5000 can execute instructions loaded into the RAM 5001 from the ROM 5002, an external memory (not shown), a storage medium, or a communication network. When the processing module 500 is powered on, the processor 5000 can read instructions from the RAM 5001 and execute them. These instructions form computer programs that cause the implementation by the processor 5000 of, for example, the decoding methods, the encoding methods, the processes performed by the game engine 10, the 3D graphics renderer 11, the registration module 21, and parts of the processes described in connection with Figures 2, 3, and 4 described herein below.

[0061] All or part of the algorithms and steps of the encoding or decoding methods may be implemented in software form by execution of a set of instructions by a programmable machine such as a DSP (digital signal processor) or a microcontroller, or in hardware form by a machine or dedicated component such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit).

[0062] FIG. 5C illustrates a block diagram of an example of a system 2 in which various aspects and embodiments are implemented. The gaming system 2 may be embodied as a device including the various components described above (i.e., the registration module 21 and the video decoder 20) and configured to implement one or more of the aspects and embodiments described herein. Examples of such devices include various electronic devices, such as, but not limited to, personal computers, laptop computers, smartphones, tablet computers, head-mounted displays, and game consoles. The elements of the gaming system 2 may be embodied, alone or in combination, in an integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the gaming system 2 comprises a processing module 500 that implements the video decoder 20 and the registration module 21. In various embodiments, the gaming system 2 is communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, the gaming system 2 is configured to implement one or more of the aspects described herein.

[0063] Input to processing module 500 may be provided via various input modules, as shown in block 531. Such input modules include, but are not limited to, (i) a radio frequency (RF) module that receives, for example, wirelessly transmitted RF signals;

[0064] In various embodiments, the input modules of block 531 have associated respective input processing elements as known in the art. For example, the RF module may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band, which in certain embodiments may be referred to (for example) as a channel, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF module of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF portion may include, for example, a tuner that performs various of these functions, including down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency close to baseband) or to baseband. In one embodiment, the RF module and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and refiltering to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF module includes an antenna.

[0065] The various elements of the gaming system 2 may be provided within a one-piece housing, where the various elements may be interconnected and transmit data between them using any suitable connection arrangement, such as an internal bus known in the art, including an Inter-IC (I2C) bus, wiring, and printed circuit boards. For example, in the gaming system 2, the processing module 500 is interconnected to the other elements of the gaming system 2 by a bus 5005.

[0066] The communication interface 5004 of the processing module 500 enables the gaming system 2 to communicate over a communication channel 3. As already mentioned above, the communication channel 3 may be implemented, for example, in a wired and / or wireless medium.

[0067] Data is streamed or otherwise provided to the gaming system 2 in various embodiments using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received via a communication channel 3 and a communication interface 5004 that are adapted for Wi-Fi communication. The communication channel 3 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, the RF connection of the input block 531 is used to provide the streamed data to the gaming system 2. Additionally, various embodiments use wireless networks other than Wi-Fi, e.g., cellular networks.

[0068] The gaming system 2 can provide output signals to various output devices including a display system 55, speakers 56, and other peripheral devices 57. The display system 55 of various embodiments includes, for example, one or more of a touch screen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display system 55 can be for a television, a tablet, a laptop, a mobile phone, a head mounted display, or other device. The display system 55 can also be integrated with other components, such as a smartphone, or can be separate, such as an external monitor for a laptop. The other peripheral devices 57 include, in various examples of embodiments, one or more of a stand-alone digital video disc (or digital versatile disc, DVR, for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 57 that provide functionality based on the output of the gaming system 2. For example, a disc player performs the functionality of playing the output of the gaming system 2.

[0069] In various embodiments, control signals are communicated between the gaming system 2 and the display system 55, speakers 56, or other peripheral devices 57 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that allow control between devices with or without user intervention. The output devices may be communicatively coupled to the gaming system 2 via dedicated connections through respective interfaces 532, 533, and 534. Alternatively, the output devices may be connected to the gaming system 2 using communication channel 3 via communication interface 5004, or using a dedicated communication channel via communication interface 5004. The display system 55 and speakers 56 may be integrated into one unit with other components of the gaming system 2 in an electronic device such as, for example, a gaming console. In various embodiments, the display interface 532 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0070] The display system 55 and the speakers 56 may alternatively be separate from one or more of the other components. In various embodiments in which the display system 55 and the speakers 56 are external components, the output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0071] FIG. 5B illustrates a block diagram of an example of a server 1 in which various aspects and embodiments are implemented. The server 1 is very similar to the game system 2. The server 1 may be embodied as a device including the various components described above (i.e., the game engine 10, the 3D graphics renderer 11, and the video encoder 12) and configured to perform one or more of the aspects and embodiments described herein. Examples of such devices include various electronic devices, such as, but not limited to, personal computers, laptop computers, and servers. The elements of the server 1 may be embodied, alone or in combination, in an integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the server 1 comprises a processing module 500 that implements the video encoder 12, the 3D graphics renderer 11, and the game engine 10. In various embodiments, the server 1 is communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, the server 1 is configured to implement one or more of the aspects described herein.

[0072] Input to the processing module 500 may be provided via various input modules as shown in block 531 already described with respect to FIG. 5C.

[0073] The various elements of system 1 may be provided within a unitary housing, where the various elements may be interconnected and transmit data between them using any suitable connection arrangement, such as an internal bus known in the art, including an Inter-IC (I2C) bus, wires, and printed circuit boards. For example, in system 1, processing module 500 is interconnected to other elements of server 1 by bus 5005.

[0074] The communication interface 5004 of the processing module 500 enables the system 1 to communicate over a communication channel 3 .

[0075] Data is streamed or otherwise provided to server 1 in various embodiments using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received via communication channel 3 and communication interface 5004, which are adapted for Wi-Fi communication. Communication channel 3 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. Other embodiments provide data to server 1 using an RF connection of input block 531.

[0076] Various embodiments use wireless networks other than Wi-Fi, for example, cellular networks.

[0077] The data provided to the server 1 is data representative of actions performed by a user (or users) or parameters selected by a user (or users).

[0078] The server 1 provides the encoded video stream to the gaming system 2 in the form of an output signal.

[0079] Various implementations include decoding, as used in this application, includes applying a decoding process to the encoded video stream in response to encoding tools that are activated or deactivated in the encoded video stream, and in some embodiments in response to tuning parameters that define a particular implementation of the decoding process.

[0080] Various implementations involve encoding. "Encoding" as used in this application includes applying an encoding process in response to encoding tools selected by a user and, in some embodiments, tuning parameters that define a particular implementation of the encoding process.

[0081] It should be noted that the syntax element names used herein are descriptive terms, and therefore they do not exclude the use of other syntax element names.

[0082] Where a figure is presented as a flow chart, it should be understood that the figure also provides a block diagram of the corresponding apparatus. Similarly, where a figure is presented as a block diagram, it should be understood that the figure also provides a flow chart of the corresponding method / process.

[0083] The implementations and aspects described herein may be implemented, for example, in a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed in the context of only a single implementation form (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). For example, an apparatus may be implemented in suitable hardware, software, and firmware. A method may be implemented, for example, in a processor, where a processor refers to a general processing device including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. A processor also includes, for example, a communication device, such as a computer, a mobile phone, a portable / personal digital assistant, and other devices that facilitate communication of information between end users.

[0084] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation," as well as other variations thereof, means that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "an embodiment" or "in one implementation," as well as other variations thereof, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0085] Additionally, the application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, retrieving information from a memory, or retrieving information from, for example, another device, module, or user.

[0086] Additionally, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, obtaining information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0087] Additionally, the application may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" generally involves in some way, for example, an operation such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0088] Use of any of the terms " / ", "and / or", "at least one of", "one or more", e.g., "A / B", "A and / or B", "at least one of A and B", "one or more of A and B" is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, "A, B, and / or C" and "at least one of A, B, and C", "one or more of A, B, and C" are intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This may be expanded to include as many of the items listed as would be apparent to one of ordinary skill in this and related arts.

[0089] Also, as used herein, the term "signaling" means to indicate something, especially to a corresponding decoder. For example, in certain embodiments, an encoder signals the use of some coding tools. In this way, in embodiments, the same parameters can be used at both the encoder and decoder sides. Thus, for example, an encoder can transmit a particular parameter to a decoder (explicit signaling) so that the decoder can use the same particular parameter. In contrast, if the decoder already has the particular parameter as well as other parameters, a non-transmitting signaling (implicit signaling) can be used to simply allow the decoder to know and select the particular parameter. By avoiding the transmission of any actual functionality, bit savings are realized in various embodiments. It will be appreciated that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used in various embodiments to signal information to a corresponding decoder. While the above relates to the verb form of the word "signal", the word "signal" may also be used as a noun in this specification.

[0090] As will be apparent to one skilled in the art, implementations can result in a variety of signals formatted to carry information that can be, for example, stored or transmitted. Information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry an encoded video stream that includes constraint flags in a data structure general_constraints_info. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding the encoded video stream and modulating a carrier wave with the encoded video stream. The information that the signal carries can be, for example, analog or digital information. As is known, the signal can be transmitted over a variety of different wired or wireless links. The signal can be stored in a processor-readable medium.

[0091] The following examples of embodiments are described in the context of video formats similar to VVC. However, these embodiments are not limited to video coding / decoding methods corresponding to VVC. These embodiments are adapted to any video format, in particular as soon as encoding tools can be activated, deactivated or modified, or a specific implementation of an encoder or decoder can be selected. Such formats include, for example, the standard EVC (Essential Video Coding / MPEG-5), AV1, and VP9.

[0092] 6, 7 and 8 introduce examples of video formats.

[0093] 6 illustrates an example of a division that a picture 61 of pixels of an original picture sequence 60 undergoes. Here, a pixel is considered to consist of three components, namely a luminance component and two chrominance components. However, other types of pixels may contain fewer or more components, such as only a luminance component or an additional depth component.

[0094] A picture is divided into multiple coding entities. First, as represented by reference number 63 in FIG. 6, a picture is divided into a grid of blocks called coding tree units (CTUs). A CTU consists of an N×N block of luminance samples and two corresponding blocks of chrominance samples. N is generally a power of two with a maximum value of, for example, "128". Second, a picture is divided into one or more groups of CTUs. For example, a picture can be divided into one or more tile rows and tile columns, where a tile is a sequence of CTUs that covers a rectangular area of ​​a picture. In some cases, a tile can be divided into one or more bricks, each of which consists of at least one CTU row in the tile. Certain types of tiles prevent spatial and temporal prediction from samples of other tiles. These tiles are called subpictures. On top of the concept of tiles and bricks, there is another coding entity called a slice, which can include at least one tile of a picture or at least one brick of a tile.

[0095] In the example of Figure 6, as represented by reference numeral 62, a picture 61 is divided into three slices S1, S2, and S3 in raster scan slice mode, each containing multiple tiles (not shown), with each tile containing only one brick.

[0096] As represented by reference numeral 64 in Figure 6, a CTU may be divided into a hierarchical tree of one or more sub-blocks called coding units (CUs). The CTU is the root (i.e., parent node) of the hierarchical tree and may be divided into multiple CUs (i.e., child nodes). Each CU becomes a leaf of the hierarchical tree if it has not been further divided into smaller CUs, or becomes a parent node of the smaller CUs (i.e., child nodes) if it has been further divided.

[0097] In the example of FIG. 6, CTU14 is first divided into "4" rectangular CUs using a quadtree type division. The top left CU is a leaf of the hierarchical tree, i.e., it is not a parent node of other CUs, since it is not further divided. The top right CU is further divided into "4" smaller square CUs, also using a quadtree type division. The bottom right CU is divided vertically into "2" rectangular CUs using a binary tree type division. The bottom left CU is divided vertically into "3" rectangular CUs using a ternary tree type division.

[0098] During coding of a picture, the partitioning is adaptive, and each CTU is partitioned to optimize the compression efficiency of the CTU criteria.

[0099] In HEVC, the concept of prediction unit (PU) and transform unit (TU) has been introduced. Indeed, in HEVC, the coding entities used for prediction (i.e., PU) and transformation (i.e., TU) may be parts of a CU. For example, as shown in FIG. 6, a CU of size 2N×2N may be divided into PUs 6411 of size N×2N or size 2N×N. The CU may further be divided into “4” TUs 6412 of size N×N or PUs 6413 of size N×N.

[0100]

number

[0101] It may be noted that in VVC, except for some specific cases, the frontiers of TUs and PUs are aligned with the frontier of a CU. Thus, a CU generally contains one TU and one PU.

[0102] In this application, the term "block" or "picture block" may be used to refer to any one of a CTU, a CU, a PU, and a TU. Additionally, the term "block" or "picture block" may be used to refer to a macroblock, a partition, and a sub-block as specified in H.264 / AVC or other video coding standards, and more generally to an array of samples of multiple sizes.

[0103] In this application, the terms "reconstructed" and "decoded" can be used interchangeably, the terms "pixel" and "sample" can be used interchangeably, and the terms "image," "picture," "subpicture," "slice," and "frame" can be used interchangeably. Typically, but not necessarily, the term "reconstructed" is used on the encoder side, while the term "decoded" is used on the decoder side.

[0104] Fig. 7 illustrates in a schematic manner a method for encoding a video stream executed by an encoding module. Variations of this method for encoding are contemplated, but for the sake of clarity, the method for encoding in Fig. 7 is described below without describing all possible variations. All steps described in relation to Fig. 7 are executed, for example, by the processing module 500 when this processing module implements the video encoder 12.

[0105] Before being encoded, the current original image of the original video sequence may undergo pre-processing. For example, in step 701, a color transformation is applied to the current original picture (e.g., from RGB 4:4:4 to YCbCr 4:2:0) or a remapping is applied to the current original picture components to obtain a signal distribution more resilient to compression (e.g., using histogram equalization of one of the color components). In addition, pre-processing 601 may include resampling (down-sampling or up-sampling). Resampling may be applied to some pictures so that the generated bitstream may contain a picture in the original resolution and a picture in another resolution (or at least pictures in at least two different resolutions). Resampling generally consists of down-sampling and is used to reduce the bitrate of the generated bitstream. Nevertheless, up-sampling is also possible. The pictures obtained by pre-processing are referred to as pre-processed pictures in the following.

[0106] Encoding the preprocessed picture begins with the division of the preprocessed picture during step 702, as described in connection with Figure 6. Thus, the preprocessed picture is divided into CTUs, CUs, PUs, TUs, etc. For each block, the encoding module determines a coding mode between intra prediction and inter prediction.

[0107] Intra prediction consists of predicting pixels of the current block from a predictive block derived from pixels of a reconstructed block located in the causal neighborhood of the current block to be coded, during step 703, according to an intra prediction method. The result of intra prediction is a prediction direction indicating which pixel of the neighboring block to use, and a residual block resulting from the calculation of the difference between the current block and the predictive block. Recently, new intra prediction modes have been proposed and introduced into VVC. These new intra prediction modes include: - MIP (Matrix weighted Intra Prediction), which consists in using a matrix to generate an intra predictor from the reconstructed neighboring boundary samples to the top left of the block to be predicted. ● ISP (Intra Sub-Partition) that divides a luma intra prediction block into 2 or 4 sub-partitions vertically or horizontally depending on the block size. • CCLM (Cross-component linear model) prediction, where the chroma samples of a CU are predicted based on the reconstructed luminance samples of the same CU by using a linear model. ● IBC (Intra Block Copy), which consists of predicting a block in a picture from another block of the same picture. Reference sample filtering in the intra area, which consists of filtering the reference samples used for intra prediction.

[0108] Inter prediction consists of predicting the pixels of the current block from a block of pixels (called a reference block) of a picture preceding or following the current picture (this picture is called a reference picture). During the coding of the current block by the inter prediction method, the block of the reference picture that is closest to the current block according to a similarity criterion is determined by a motion estimation step 704. During step 704, a motion vector is determined that indicates the position of the reference block in the reference picture. Motion estimation is generally performed with sub-pixel accuracy, i.e. the current picture and the reference picture are interpolated. The motion vector determined by motion estimation is used during a motion compensation step 305, during which a residual block is calculated in the form of the difference between the current block and the reference block.

[0109] In the first video compression standard, the unidirectional inter prediction mode described above was the only inter mode available. As video compression standards have evolved, the family of inter modes has grown significantly and now includes many different inter modes. These inter prediction modes include, for example: In bi-prediction, refined motion vectors are searched around each initial motion vector: DMVR (decoder side motion vector refinement). The refinement is performed symmetrically by the encoder and the decoder. ● Bi-directional optical flow (BDOF) is based on the concept of optical flow and assumes smooth object motion. BDOF is used to refine the bi-directional prediction signal of a CU at the 4x4 sub-block level. BDOF is only applied to the luma component. ● PROF (prediction refinement with optical flow): Sub-block-based affine motion compensation can save memory access bandwidth and reduce computational complexity compared to pixel-based motion compensation at the expense of a prediction accuracy penalty. To achieve finer granularity of motion compensation, prediction refinement with optical flow (PROF) is used to refine sub-block-based affine motion compensation prediction without increasing memory access bandwidth for motion compensation. - CIIP (Combined inter and intra prediction), which combines the inter prediction signal with the intra prediction signal. ● GPM (geometric partitioning mode) which divides a CU into two parts by a geometrically located line. Each part of the geometric partition within a CU is inter predicted using its own motion, and for each partition, only uni-prediction is allowed.

[0110] During a selection step 706, the encoding module selects a prediction mode that optimizes compression performance from among the tested prediction modes (e.g., intra prediction modes, inter prediction modes) according to a rate / distortion optimization criterion (i.e., RDO criterion).

[0111] Once a prediction mode is selected, the residual block is transformed during step 707 and quantized during step 709. The inverse transform has also evolved and new tools have been proposed recently. These new tools include: • Chroma residuals are coded jointly, JCCR (Joint coding of chroma residuals). - MTS (multiple transform selection), where a selection is made between DCT-2, DCT-7 and DCT-8 for horizontal and vertical transforms. ● LFNST (Low-frequency non-separable transform): LFNST is applied between the forward linear transform and the quantization (at the encoder) and between the inverse quantization and the inverse linear transform (at the decoder side). A 4x4 non-separable transform or an 8x8 non-separable transform is applied according to the block size. ● BDPCM (Block differential pulse coded modulation). BDPCM can be considered as a competitor of the normal intra mode. When BDPCM is used, a BDPCM prediction direction flag is transmitted to indicate whether the prediction is horizontal or vertical. The block is then predicted using the normal horizontal or vertical intra prediction process with unfiltered reference samples. The residuals are quantized and the difference between each quantized residual and its predictor, i.e. the residual previously coded in the horizontal or vertical neighbor (depending on the BDPCM prediction direction), is coded. ● SBT (Subblock transform), where only a sub-portion of the residual block is coded for a CU.

[0112] It should be noted that the encoding module can skip the transform and directly apply quantization to the untransformed residual signal. If the current block is coded according to an intra prediction mode, the prediction direction and the transformed and quantized residual block are coded by the entropy encoder during step 710. If the current block is coded according to inter prediction, the motion vector of the block is predicted, if appropriate, from a prediction vector selected from a set of motion vectors corresponding to reconstructed blocks located near the block to be coded. Then, motion information is coded by the entropy encoder during step 710 in the form of a motion residual and an index for identifying the prediction vector. The transformed and quantized residual block is coded by the entropy encoder during step 710. It should be noted that the encoding module can bypass both the transform and the quantization, i.e. the entropy coding is applied to the residual without applying a transform or quantization process. The result of the entropy coding is inserted into the coded video stream 711.

[0113] After the quantization step 709, the current block is reconstructed so that the pixels corresponding to said block can be used for future prediction. This reconstruction phase is also called a prediction loop. Thus, inverse quantization is applied to the transformed and quantized residual block during step 712 and an inverse transform is applied during step 713. Depending on the prediction mode used for the block obtained during step 714, a prediction block of the block is reconstructed. If the current block is coded according to an inter prediction mode, the coding module applies, if appropriate, motion compensation using the motion vector of the current block to identify a reference block of the current block during step 716. If the current block is coded according to an intra prediction mode, the prediction direction corresponding to the current block is used to reconstruct the reference block of the current block during step 715. The reference block and the reconstructed residual block are added to obtain a reconstructed current block.

[0114] After reconstruction, during step 717, in-loop post-filtering intended to reduce coding artifacts is applied to the reconstructed block. This filtering is called in-loop post-filtering because it is performed in the prediction loop to obtain the same reference image at the encoder as the decoder, thus avoiding drift between the encoding and decoding processes. As mentioned above, in-loop filtering tools include deblocking filtering, SAO (Sample Adaptive Offset), ALF (Adaptive Loop Filter), and CC-ALF (Cross Component ALF). CC-ALF refines each chroma component by applying an adaptive linear filter to the luma channel using luma sample values, and then using the output of this filtering operation to refine the chroma. A new tool called LMCS (Luma Mapping with Chroma Scaling) can also be considered as in-loop filtering. LMCS is added as a new processing block before the other loop filters. LMCS has two main components: an in-loop mapping of the luma component based on an adaptive piecewise linear model, and a luma-dependent chroma residual scaling applied to the chroma component.

[0115] Once a block is reconstructed, it is inserted during step 718 into a reconstructed picture stored in a memory of reconstructed pictures 719, commonly called a Decoded Picture Buffer (DPB). The reconstructed picture so stored can serve as a reference picture for other pictures to be coded.

[0116] A new tool of VVC called Reference Picture Resampling (RPR) allows changing the resolution of coded pictures on the fly. Pictures are stored in the DPB at their actual encoding / decoding resolution, which may be lower than the video spatial resolution signaled in the high-level syntax (HLS) of the bitstream. When a picture being coded at a given resolution uses a reference picture for temporal prediction that is not of the same resolution, texture reference picture resampling is applied (represented by step 720 in FIG. 7) so that the predicted picture and the reference picture have the same resolution. It should be noted that, depending on the implementation, the resampling process is not necessarily applied to the whole reference picture (whole reference picture resampling), but may be applied only to the blocks identified as reference blocks when performing the decoding and reconstruction of the current picture (block-based reference picture resampling). In this case, when a current block in a current picture uses a reference picture having a different resolution than the current picture, the samples in the reference picture used for temporal prediction of the current block are resampled according to a resampling ratio calculated as the ratio between the current picture resolution and the reference picture resolution.

[0117] Metadata, such as a Supplemental Enhancement Information (SEI) message, may be added to the encoded video stream 311. A Supplemental Enhancement Information (SEI) message, as defined in standards such as AVC, HEVC, or VVC, is a data container or data structure that includes metadata associated with and providing information about a video stream.

[0118] Fig. 8 illustrates diagrammatically a method for decoding an encoded video stream 711, encoded according to the method described in relation to Fig. 8, performed by a decoding module. Variations of this method for decoding are contemplated, but for the sake of clarity, the method for decoding of Fig. 8 is described below without describing all possible variations. All steps described in relation to Fig. 8 are performed, for example, by processing module 500 when this processing module implements video decoder 20.

[0119] The decoding is performed block by block. For the current block, the decoding starts with entropy decoding of the current block during step 810. The entropy decoding makes it possible to obtain the prediction mode of the block.

[0120] If the block is coded according to an inter prediction mode, entropy decoding makes it possible to obtain, where appropriate, a prediction vector index, a motion residual, and a residual block. During step 808, a motion vector is reconstructed for the current block using the prediction vector index and the motion residual.

[0121] If the block is coded according to an intra prediction mode, the entropy decoding allows to obtain the prediction direction and the residual block. Steps 812, 813, 814, 815, 816 and 817 implemented by the decoding module are all identical to steps 712, 713, 714, 715, 716 and 717 implemented by the coding module, respectively. In step 818, the decoded block is saved in a decoded picture, and the decoded picture is stored in a DPB 819. When the decoding module decodes a given picture, the picture stored in the DPB 819 is identical to the picture stored in the DPB 719 by the coding module during the coding of the given picture. The decoded picture may also be output by the decoding module, for example for display. When RPR is activated, the samples of (at least part of) the picture used as a reference picture are resampled in step 820 to the resolution of the predicted picture. The resampling step (820) and the motion compensation step (816) may be combined into one single-sample interpolation step in some implementations.

[0122] The decoded image may further undergo post-processing in step 821. Post-processing may include inverse color conversion (e.g., YCbCr 4:2:0 to RGB 4:4:4 conversion), inverse mapping to perform the inverse of the remapping process performed in the pre-processing of step 701, post-filtering to improve the reconstructed picture based on the filter parameters provided in the SEI message, and / or resampling, for example, to adjust the output image to display constraints.

[0123] As mentioned above, many of the encoding tools defined in the VVC encoding method can be activated or deactivated using constraint flags stored in the data structure general_constraint_info as represented in Table TAB1.

[0124] [Table 1-1]

[0125] [Table 1-2]

[0126] [Table 1-3]

[0127] example: The constraint flag gci_no_mip_constraint_flag activates or deactivates MIP. ● The constraint flag gci_no_isp_constraint_flag activates or deactivates the ISP. The constraint flag gci_no_dmvr_constraint_flag activates or deactivates the DMVR. ● The constraint flag gci_no_bdof_constraint_flag activates or deactivates BDOF. The constraint flag gci_no_mts_constraint_flag activates or deactivates MTS. ● The constraint flag gci_no_lfnst_constraint_flag activates or deactivates LFNST. ● The constraint flag gci_no_sao_constraint_flag activates or deactivates SAO. ● etc.

[0128] FIG. 2 illustrates generally a first embodiment of a method for enabling codec fine-tuning in streaming applications.

[0129] The method of Figure 2 is implemented by a server 1 and a gaming system 2. However, the method may be implemented outside of a gaming context by any system that implements low latency streaming applications.

[0130] 2, only the modules involved in fine-tuning the codecs are represented: game engine 10, video encoder 12, video decoder 20, and game registration module 21. For simplicity, the 3D graphics renderer 11 is not represented, which is only involved in generating the original pictures provided to the video encoder 12. However, all pictures encoded by the video encoder 12 and decoded by the video decoder 20 were originally generated by the 3D graphics renderer 11 under the control of the game engine 10.

[0131] The game engine 10 is a typical module for gaming applications. In other types of low-latency streaming applications (video conferencing, head-mounted display (HMD)-based applications, or augmented reality / virtual reality (AR / VR) glasses-based applications XR on glasses, any immersive and interactive applications), the game engine 10 is replaced by a simpler module that is responsible at least for collecting data representing encoding parameters and for configuring the video encoder 12 based on these encoding parameters.

[0132] Similarly, in other types of low-latency streaming applications, the registration module 21 collects parameters representative of the desired quality of experience (QoE) resulting from the user and, in some embodiments, converts these parameters representative of the desired QoE into encoding parameters and forwards the parameters representative of the desired QoE or the encoding parameters to the server 1.

[0133] When starting a gaming application or any low latency streaming application, a user can access parameters from a user interface and select some settings according to the user's desired QoE. These settings can be described by general labels or can be very detailed.

[0134] 2, for simplicity, we assume that a processing module 500 implements all modules of the server 1, and another processing module 500 implements all modules of the game system 2. For example, on the server 1 side, the processing module 500 implements the game engine 10 and the video encoder 12. On the game system 2 side, the processing module 500 implements the video decoder 20 and the registration module 21. However, in another embodiment, the independent processing module 500 may independently implement each module of the server 1 and the game system 2.

[0135] In step 201, the processing module 500 of the gaming system 2 obtains settings selected by a user. From these settings, the processing module 500 of the gaming system 2 obtains an identifier representing a desired QoE. In one embodiment, the user directly selects the identifier representing the desired QoE.

[0136] In step 202 , the processing module 500 of the game system 2 transmits the identifier to the server 1 .

[0137] In step 203, the processing module 500 of the server 1 receives an identifier representative of a desired QoE.

[0138] In step 204, the processing module 500 of the server 1 obtains the encoding parameters based on the identifier.

[0139] In step 205, the processing module 500 of the server 1 adjusts the video encoder so that it produces an encoded video stream that complies with the identifier, i.e. complies with the desired QoE.

[0140] From step 205, the video encoder 12 in step 206 encodes the pictures provided by the 3D graphics renderer 11 in an encoded video stream that complies with the identifier.

[0141] In step 207 , the processing module 500 of the server 1 transmits the encoded video stream conforming to the identifier to the gaming system 2 , ie the decoder 20 .

[0142] In step 208, the processing module 500 of the gaming system 2 (ie, the video decoder 20 implemented by the processing module 500) decodes the received encoded video stream.

[0143] In step 209 , the processing module 500 of the gaming system 2 transmits the decoded picture to the display system 55 .

[0144] In an embodiment adapted for low-latency applications where pictures are not generated live, the server 1 stores all (or at least the most likely or most required) QoE-compliant pre-encoded video streams. The processing module then selects in step 205 the pre-encoded video stream that complies with the quality of experience corresponding to the identifier and transmits the selected pre-encoded video stream.

[0145] FIG. 3 illustrates diagrammatically a second embodiment of a method for enabling codec fine-tuning in streaming applications.

[0146] In the embodiment of FIG. 3, the identifier of the desired QoE is not only used to adjust the encoding process (and the decoding process through the encoded video stream it receives), but is also used to directly adjust the decoding process.

[0147] All steps of the method of Figure 2 are maintained in the method of Figure 3. Two additional steps 301 and 302 are added.

[0148] In step 301, the processing module 500 of the gaming system 2 obtains decoder parameters from an identifier of a desired QoE. For example, a first value of the identifier induces decoder parameters indicating to decode only INTRA pictures or to skip some in-loop post-filters, such as SAO. A second value of the identifier induces decoder parameters leading to the application by the video decoder 20 of post-processing to the decoded pictures. As can be seen, the identifier adjusts the video decoder 20 to skip some decoding processes specified in the encoded video stream and activates only the necessary decoding functions according to the parameters set for the desired QoE from the video encoder, or the video decoder 20 adds decoding processes not specified in the encoded video stream, i.e. adds additional decoding processes in addition to the processes specified in the encoded video stream.

[0149] In step 302, the processing module 500 of the gaming system 2 configures a video decoder based on the decoder parameters.

[0150] FIG. 4 illustrates diagrammatically a third embodiment of a method for enabling codec fine-tuning in streaming applications.

[0151] The method of Figure 4 is another variation of the method of Figure 2. Steps 202 to 209 are maintained.

[0152] Step 201 is replaced by two steps 401 and 402 in which a capability exchange takes place.

[0153] In step 401 , the processing module 500 of the server 1 transmits information to the gaming system 2 representative of a set of encoding tools and / or configurations supported by the video encoder 12 .

[0154] In step 402, the processing module 500 of the gaming system 2 receives this information and determines a set of coding tools that may be activated or deactivated depending on this information. This set of coding tools is then presented to the user who defines a desired QoE depending on this set.

[0155] Another feature of the method of FIG. 4 is the ability to update the desired QoE, ie, to send a new identifier of the desired QoE, if required.

[0156] Therefore, in step 410, the processing module 500 of the game system 2 sends the new identifier to the server 1. Then, the processing module 500 of the server 1 executes step 203.

[0157] So far, we have not described in detail what is represented by the desired quality of experience identifier. In the following, we describe some embodiments of the data represented by the identifier.

[0158] In a first embodiment of the data represented by the identifier, the identifier represents a profile of a video compression standard.

[0159] In certain applications, such as cloud gaming, the content may vary greatly from a highly graphic representation to a more natural look. However, the color space may be significantly extended and the textures may be significantly enriched. Because such video coding tools were typically developed for natural content, they may not be as efficient as coding tools developed for computer graphics pictures and graphics. Thus, in some embodiments, the video coding tools may be complemented by tools developed for computer graphics pictures and graphics.

[0160] In a first embodiment, a new profile adapted to low-latency streaming applications is defined. In the following, a profile called Low Latency Graphic (LLG) profile adapted to VVC is proposed. The LLG profile disables coding tools that have low coding performance and high computational cost. Similar LLG profiles can be defined for other video coding standards such as EVC, HEVC, AVC, VP9, ​​and AV1.

[0161] Table TAB2 provides an example of a definition of an LLG profile for VVC (called the Main 10 LLG profile). A bitstream conforming to the Main 10 LLG profile shall obey at least some of the following constraints:

[0162] [Table 2]

[0163] In a second embodiment of the data represented by the identifier, the identifier represents a group of constraint flags.

[0164] Below are examples of how constraint flags as described in VVC can be related to user QoE and how they can be mapped to user friendly preference settings.

[0165] The following exemplary qualities of experience may be defined: High visual quality with latency increase (HVQLI), when the tool being deactivated increases latency but visual quality improves (either by disabling the tool or by using an alternative tool); o Fast (F): The deactivated tools will reduce the encoding / decoding processing time while impacting the visual quality. Deactivation of a tool may or may not induce a significant loss of compression gain. Intermediate (I): Deactivated tools reduce encoding / decoding processing times, but less than the fast settings, while not affecting visual quality too much. Chroma (C): Tools that apply specific processing to chroma are deactivated. Only processes derived from luma are applied. Alternatively, a gray-level QoE can be defined where only the luma component is coded / rendered.

[0166] Each of these identifiers is associated with a group of constraint flags that make it possible to achieve this QoE. Table TAB3 associates a subset of the constraint flags defined in VVC to QoEs. Other constraint flags not mentioned in this table are considered not affected by any QoE that may be selected, and the corresponding tools may be used freely. When only a subset of the QoEs defined above are mentioned for constraint flags, the tools corresponding to the constraint flags may be used freely for the QoEs not mentioned.

[0167] [Table 3]

[0168] In table TAB3: o QoE HVQLI is associated with a group of constraint flags including: gci_no_lmcs_constraint_flag, gci_no_sao_constraint_flag, gci_no_joint_cbcr_constraint_flag, gci_no_transform_skip_constraint_flag, gci_no_gpm_constraint_flag, gci_no_ciip_constraint_flag, gci_no_bcw_constraint_flag, gci_no_dmvr_constraint_flag, gci_no_smvd_constraint_flag, constraint_flag, gci_no_bdof_constraint_flag, gci_no_gdr_constraint_flag, gci_no_cra_constraint_flag, gci_intra_only_constraint_flag. 〇 QoE F is gci_no_lmcs_constraint_flag, gci_no_sao_constraint_flag, gci_no_transform_skip_constraint_flag, gci_no_gpm_constraint_flag, gci_no_ciip_constraint_ flag, gci_no_bcw_constraint_flag, gci_no_dmvr_constraint_flag, gci_no_smvd_constraint_flag, constraint_flag, gci_no_bdof_constraint_flag, gci_no_idr_con The constraint flags are associated with a group of constraint flags including: gci_no_cra_constraint_flag, gci_no_intra_only_constraint_flag, gci_no_palette_constraint_flag, gci_no_isp_constraint_flag, gci_no_mrl_constraint_flag, gci_no_mip_constraint_flag, gci_no_dep_quant_constraint_flag, gci_no_alf_constraint_flag, and gci_no_ccalf_constraint_flag. o QoE I is associated with a group of constraint flags including: gci_no_lmcs_constraint_flag, gci_no_sao_constraint_flag, gci_no_transform_skip_constraint_flag, gci_no_gpm_constraint_flag, gci_no_ciip_constraint_flag, gci_no_bcw_constraint_flag, gci_no_dmvr_constraint_flag, gci_no_smvd_constraint_flag, constraint_flag, gci_no_bdof_constraint_flag, gci_no_gdr_constraint_flag, gci_no_cra_constraint_flag, gci_intra_only_constraint_flag, gci_no_dep_quant_constraint_flag. o QoE C is associated with a group of constraint flags including: gci_no_lmcs_constraint_flag, gci_no_sao_constraint_flag, gci_no_joint_cbcr_constraint_flag, gci_no_transform_skip_constraint_flag, gci_no_gpm_constraint_flag, gci_no_ciip_constraint_flag, gci_no_bcw_constraint_flag, gci_no_dmvr_constraint_flag, gci_no_smvd_constraint_flag, constraint_flag, gci_no_bdof_constraint_flag, gci_no_gdr_constraint_flag, gci_no_cra_constraint_flag, gci_no_idr_constraint_flag, gci_intra_only_constraint_flag.

[0169] When a user selects a QoE, the user indirectly selects a group of constraint flags and the values ​​for the constraint flags within this set.

[0170] In a first variant of the first and second embodiments of the data represented by the identifier, additional encoding decisions may be made on the server 1 side depending on the identifier. The encoding decisions define a specific implementation of the video encoder, independent of the constraint flags and the profile.

[0171] For example, if F (fast) QoE is selected, video encoder 12 will not use tools that require any iterative process (also known as when parallel operation is not supported). One example is to prevent the use of RDOQ (rate-distortion optimized quantization) if it is not parallelizable.

[0172] If HVQLI QoE is selected, video encoder 12 performs additional operations to improve the subjective quality or overall RD performance of the bitstream, for example by applying pre-processing.

[0173] In addition, an increased latency mode (for games with a slower pace, or for video interruptions during the game, such as video ads, pauses, etc.) may be used and may be achieved by increasing the GOP (Group Of Pictures) size to a value greater than 1, depending on the tradeoff between quality and desired latency. The increased GOP size will significantly increase the rate-distortion performance of the video encoder.

[0174] In a second variant of the first and second embodiments of the data represented by the identifier, additional decoding decisions may be made on the game system 2 side depending on the identifier. The decoding decisions define a specific implementation of the video decoder 20, independent of the constraint flags and the profile.

[0175] This second variant is based on the method of FIG.

[0176] For example, if F (fast) QoE is selected, the processing module 500 of the game system 2 tasks the video decoder 20 with reducing the frame rate of the decoded video, for example, by skipping the decoding of some pictures that are not used as reference pictures for temporal prediction of other pictures.

[0177] The selection of the HVQLI QoE leads to the application of post-processing (de-noising filters, contour enhancement, etc.) to the picture output by the video decoder.

[0178] In one embodiment, the QoE identifiers (HVQLI, F, I, C, etc.) are provided via specific SEI messages.

[0179] An exemplary SEI message is provided.

[0180] [Table 4]

[0181] ue_qoe_type identifies the type of desired end-user QoE as specified in Table TAB5. The value of ue_qoe_type shall be in the range 0 to y inclusive.

[0182] [Table 5]

[0183] If ue_qoe_type value=0, all constraint flags defined above are set to their corresponding values ​​for HVQLI (same for F when value is 1, I when value is 2, and C when value is 3).

[0184] In another embodiment, the mapping between QoE identifiers (HVQLI, F, I, C, etc.) and their corresponding sets of constraint flags / encoding decisions / decoding decisions is provided out-of-band via specific SEI messages described in Table TAB6.

[0185] [Table 6]

[0186] Contains all flags that are set according to the paragraph above, or contains only all flags that are required to be set to 1.

[0187] In another embodiment, the gaming system 2 receives in an SEI message a description of a set of QoEs that may be selected by a user.

[0188] When end-to-end latency is a critical factor, some low-latency applications, such as cloud gaming, use the Real-time Transport Protocol (RTP), designed for end-to-end real-time transport of streaming media over IP networks. These applications require timely delivery of information and can tolerate some packet loss. An RTP session is typically initiated between two or more endpoints using a signaling protocol. A Session Description Protocol (SDP) may be used to specify parameters for the session and describe the session, timing, and media within the session. Note that even though some of the following embodiments are described as using RTP, RTP may be replaced by other transport protocols, such as SRTP, webrtc, etc.

[0189] As can be seen in the methods of Figures 2, 3 and 4, each method comprises a step 202 of transmitting an identifier of a desired QoE. In addition, the method of Figure 4 comprises a stage of capabilities exchange having steps 401 and 402. Various protocols can be adapted to perform these steps.

[0190] When an RTP (Real-time Transport Protocol: RFC-1889) session is used to transmit the video stream, information representing the QoE desired by the client can be signaled as an extension of the Session Description Protocol (SDP: RFC4566) mechanisms, such as those defined in RTP / AVPF (RFC4585), SDP CapNeg (RFC5939), and Codec Control Message (RFC5104).

[0191] The VVC RTP payload definition (https: / / tools.ietf.org / html / draft-ietf-avtcore-rtp-vvc-07#page-50) provides an example SDP that looks like this: m=video 49170 RTP / AVP 98 a=rtpmap:98 H266 / 90000 a=fmtp:98 profile-id=1;sprop-vps= <video parameter sets data>

[0192] Since the RTP payload is to be fully specified, the line a=fmtp will contain more parameters. For example, in one embodiment, the parameter sprop-constraint-field=<constraint flag sets data> is introduced in the line a=fmtp. This parameter is used to convey any information that represents a constraint flag, multiple constraint flags, a group of constraint flags, or a profile, such as the Main 10 LLG profile.

[0193] In a variant, the parameter sprop-constraint-field=<constraint flag sets data> is used for capability exchange. During capability exchange, the server 1 transmits to the game system 2 a parameter sprop-constraint-field=<constraint flag sets data> In response, the game system 2 sends to the server 1 a parameter sprop-constraint-field=<constraint flag sets data> We use {overscore (x)} to specify identifiers representing the desired QoE, i.e., the coding tools to be activated and deactivated selected by the user.

[0194] As already mentioned, this parameter sprop-constraint-field may contain an identifier of a list of constraint flags or a group of constraint flags corresponding to the desired QoE. As explained by steps 203 and 204 of figures 2 to 4, when receiving the identifier, the server 1 determines the constraint flags associated with this identifier and the values ​​of these constraint flags and, if relevant, makes an encoding decision (aka applying RDOQ, applying an iterative process, ...) as in the first variant of the first and second embodiments of the data represented by the identifier. In parallel, in the second variant of the first and second embodiments of the data represented by the identifier, the game system 2 determines, based on the identifier, a decoding decision that allows to adjust the decoding process, if relevant.

[0195] 9 illustrates a streaming session establishment process between a client and a server using RTP / RTSP session-based streaming according to one embodiment, where the client is a gaming system 2 and the server is a server 1.

[0196] In step 901, the processing module 500 of the game system 2 sends an RTSP DESCRIBE request to the server 1. The RTSP DESCRIBE request allows obtaining a description of the content or media object identified by the request URL from the server. The Accept header may be used to specify the description format that the game system 2 understands.

[0197] In step 902, the processing module 500 of Server 1 receives an RTSP DESCRIBE request.

[0198] In step 903, the processing module 500 of the server 1 responds with an SDP message including a description of the requested content in SDP format. A new optional constraint flag set data parameter is included in the SDP message to signal at least the support of parameters related to control of coding constraints, but also coding restriction information related to the requested content corresponding to a different QoE.

[0199] In step 904, the processing module 500 of the gaming system 2 receives an SDP message including parameter constraint flag set data. In one embodiment, the parameter constraint flag set data only informs the gaming system 2 that the server 1 supports parameters related to coding constraints. In another embodiment, the parameter constraint flag set data includes syntax elements ue_qoe_F_flag and / or syntax elements corresponding to QoE_F, such as gci_no_isp_constraint_flag, gci_no_idr_constraint_flag, gci_no_dep_quant_constraint_flag, gci_no_alf_constraint_flag, etc. Thus, in step 904, the processing module 500 of the gaming system 2 receives information that allows it to control the coding configuration consistent with its desired QoE.

[0200] In step 905, the processing module 500 of the gaming system 2 sends an RTSP SETUP request to the server 1. The RTSP SETUP request specifies the transport mechanism to be used for the streamed content. In addition, the RTSP SETUP request specifies an indication of the QoE expected by the gaming system 2 for the encoding tools to be activated or deactivated when decoding the stream corresponding to the requested content. For example, the gaming system 2 requests Fast (F) quality of experience for the most complex version of the requested content. As can be seen, in step 905, the game system 2 sets gci_no_lmcs_constraint_flag, gci_no_sao_constraint_flag, gci_no_transform_skip_constraint_flag, gci_no_gpm_constraint_flag, gci_no_ciip_constraint_flag, gci_no_bcw_constraint_flag, gci_no_dmvr_constraint_flag, gci_no_smvd_constraint_flag, constraint_flag, gci_no_bdof_constraint_flag, gci_no_idr_constraint_flag

[0113] A stream may be requested that conforms to the expected QoE_F corresponding to the encoding tools that are activated or deactivated from gci_no_gdr_constraint_flag, gci_no_cra_constraint_flag, gci_intra_only_constraint_flag, gci_no_palette_constraint_flag, gci_no_isp_constraint_flag, gci_no_mrl_constraint_flag, gci_no_mip_constraint_flag, gci_no_dep_quant_constraint_flag, gci_no_alf_constraint_flag, and gci_no_ccalf_constraint_flag.

[0201] In a variation, in step 905, gaming system 2 may request an expected QoE and specify the coding tools to be activated and deactivated. For example, gaming system 2 may request a fast QoE for the most complex version of the requested content and request a version of the content in which the adaptive loop filter is deactivated.

[0202] When the parameter constraint flag set data indicates only to the game system 2 that the server 1 supports parameters related to quality of experience control (without specifying which parameters related to QoE control are supported), the game system 2 may specify any of the parameters, e.g., gci_no_lmcs_constraint_flag, gci_no_sao_constraint_flag, gci_no_joint_cbcr_constraint_flag, gci_no_transform_skip_constraint_flag, gci_no_gpm_constraint_flag, gci_no_ciip_constraint It may be noted that it is understood that any parameter in the "HVQLI" set is supported, including constraint flags such as t_flag, gci_no_bcw_constraint_flag, gci_no_dmvr_constraint_flag, gci_no_smvd_constraint_flag, constraint_flag, gci_no_bdof_constraint_flag, gci_no_gdr_constraint_flag, gci_no_cra_constraint_flag, gci_no_idr_constraint_flag, and gci_intra_only_constraint_flag.

[0203] In step 906, the processing module 500 of Server 1 receives the RTSP SETUP request.

[0204] In step 907, the processing module 500 of Server 1 sends an RTSP SETUP response including the transport parameters and the session identifier selected by the processing module of Server 1.

[0205] In step 908, the processing module 500 of the game system 2 receives the RTSP SETUP response.

[0206] In step 909, the processing module 500 of the game system 2 sends an RTSP PLAY request. The RTSP PLAY request tells the server 1 to start sending data corresponding to the requested content version via the mechanism specified in the RTSP SETUP request.

[0207] In step 910, the processing module 500 of Server 1 receives the RTSP PLAY request.

[0208] In step 911, the processing module 500 of the server 1 sends an RTSP PLAY response confirming the start of data transmission.

[0209] In step 912, the processing module 500 of the gaming system 2 receives the RTSP PLAY response confirming the start of data transmission.

[0210] In step 913, the transmission of data by the processing module 500 of the server 1 starts using the RTP session. The transmitted data corresponds to the version of the content expected by the gaming system and corresponding to the QoE or characteristics for the activated and deactivated encoding tools specified in the RTSP SETUP request sent in step 905.

[0211] In step 914, the game system 2 begins receiving the data.

[0212] In step 915, during the transmission of data, the processing module 500 of the gaming system 2 periodically sends RTCP (Real Time Control Protocol) requests to provide information about the ongoing RTP session to the server 1. Receipt of the RTCP requests by the server 1 is represented by step 916. The RTCP requests may include different QoE levels (HVQLI, Fast) or may include codec control messages such as RRPR or GDRR (RRPR and GDRR control messages are described later in this specification).

[0213] In step 917, the processing module 500 of the game system 2 sends an RTSP PAUSE request to the server 1. The RTSP PAUSE request temporarily halts the stream distribution.

[0214] In step 918, the processing module 500 of Server 1 receives the RTSP PAUSE request.

[0215] In step 919, the processing module 500 of the server 1 sends an RTSP PAUSE response to the game system 2 confirming the pause.

[0216] In step 920, the processing module 500 of the gaming system 2 receives the RTSP PAUSE response.

[0217] In step 921, the processing module 500 of the game system 2 sends an RTSP TEARDOWN request to the server 1. The RTSP TEARDOWN request stops the stream delivery and releases the resources associated with it.

[0218] In step 922, the processing module 500 of Server 1 receives the RTSP TEARDOWN request.

[0219] In step 923, the processing module 500 of the server 1 sends an RTSP TEARDOWN response to the gaming system 2 confirming the stop.

[0220] In step 924, the processing module 500 of the gaming system 2 receives the RTSP TEARDOWN response.

[0221] It may be noted that during an ongoing streaming session, each time the game system 2 attempts to modify the QoE of the requested content, it can loop back to step 905 and send a new RTSP SETUP request to the server 1 including the new QoE requirements.

[0222] Alternatively, or in addition to the parameter sprop-constraint-field, in one embodiment, the SDP attribute ACAP (Attribute CAPability) and SDP attribute PCFG (Potential ConFiGuration) as defined in RFC5939: Session Description Protocol (SDP) Capability Negotiation may be included in the offer / answer to indicate which capabilities or configurations are supported. The ACAP attribute defines how to enumerate the attribute names and their associated values ​​(if any) as capabilities. The PCFG attribute enumerates the potential configurations that are supported.

[0223] For example, the attributes ACAP and PCFG for the following encoding setting (ES) are included in the offer provided via SDP: m=video 49170 RTP / AVP 98 a=rtpmap:98 H266 / 90000 a=tcap:1 RTP / SAVPF a=acap:1 ES:HVQLI a=acap:2 ES:Fast a=pcfg:1 t:1 a:2 a=pcfg:8 a:1 In this example, the offer proposes RTP / AVP (RFC3551: RTP Profile for Audio and Video Conferencing with Minimal Control) sessions over m lines, and one transport option, tcap, with secure RTP / SAVP (Real-time Transport Protocol / Secure Audio Video Profile). The offer proposes two potential attribute capabilities (acap:1 and acap:2), with encoder capabilities for HVQLI and FAST. The preferred potential configuration is indicated by pcfg:1, which contains secure transport (t:1) and ES Fast (a:2). The least preferred potential configuration is indicated by pcfg:8, which contains ES HVQLI (a:1).

[0224] In the above embodiments, the codec was controlled primarily based on an identifier representing a profile, or an identifier representing a constraint flag or a group of constraint flags, which may be used to control features of the codec that cannot be controlled by a profile or a constraint flag.

[0225] Other solutions exist for controlling codecs. For example, in document RFC5104 (https: / / tools.ietf.org / html / rfc5104) a set of codec control messages is defined, some of which are included in the HEVC RTP RFC and the VVC RTP RFC. In particular, these control messages define the FIR (Full Intra Request) command. When a FIR command is provided to an encoder, the encoder is expected to send an instantaneous decoding refresh (IDR) picture as soon as possible. An IDR picture is a coded picture in which all slices are I-slices. When decoding an IDR picture, the decoding process marks all reference pictures as "unused for reference" immediately after decoding the IDR picture. Upon receiving the FIR command, the sender must transmit an IDR picture. One limitation of IDR pictures is that their transmission creates temporary high peaks in the bitrate that are difficult to manage in streaming applications.

[0226] In VVC, a Gradual Decoder Refresh (GDR) function is normatively defined, making it possible to create gradual recovery points by transmitting a sequence of intra-coded blocks. It is therefore beneficial to define a new codec control message to provide more flexibility than FIR commands and to make it possible to obtain smoother bitrate changes than when transmitting IDR pictures.

[0227] According to section 7 of the document RFC5104, which defines the SDP procedures for indicating and negotiating support for codec control messages (CCM) in SDP, a Gradual Decoder Refresh request (GDRR) may be defined as follows: rtcp-fb-ccm-param= / SP"GDRR", Gradual Decoder Refresh Request

[0228] The purpose of the GDRR command is to make the encoder send gradual decoder refresh points as soon as possible. In the example of Figures 1 to 4, upon receiving the GDRR command, the server 1 must start sending GDR points.

[0229] RFC5104 states that for a particular use case, "the use of the FIR command to recover from errors is explicitly not permitted; instead, the PLI messages defined in AVPF [RFC4585] should be used. The PLI messages report lost pictures, and have been included in AVPF precisely for that purpose."

[0230] However, sending a GDR command may occur prior to a picture loss or slice loss indication (e.g., upon monitoring of accumulating bandwidth congestion) and would still be permitted for other use cases. In addition, RFC4585 notes that while receipt of a PLI message typically triggers the sending of a full intra picture, the goal is precisely to allow for gradual refresh and not send a full intra picture.

[0231] As specified by RFC4585, Payload-Specific feedback (PSBF) messages are identified by the RTCP packet type value PSFB. GDRR messages are identified by the RTCP packet type values ​​PT=PSFB and FMT=xxx. The value will need to be attributed to the GDRR FMT.

[0232] In VVC, RPR is defined and can be advantageously used to address some bandwidth or congestion issues. AOM AV1 has similar functionality and the proposed commands will apply to both codecs.

[0233] In one embodiment, according to section 7 of RFC5104, an RPR request (RRPR) is defined as follows: rtcp-fb-ccm-param= / SP"RRPR", reference picture resampling request

[0234] The purpose of the RRPR request is to instruct the encoder to transmit a picture with a lower spatial resolution as soon as possible. In the example of Fig. 1-4, upon receiving the RRPR, the server 1 must transmit a resampled picture. When no additional parameters are transmitted, the encoder / server is free to decide which spatial resolution to transmit. For example, additional parameters in the FCI (Feedback Control Information) field can be used to request horizontal or vertical scaling, or incremental down / upscaling size, or an indicator for the desired down / upscaling size.

[0235] As specified by section 6.1 of RFC4585, payload-specific feedback messages are identified by the RTCP packet type value PSFB. GDRR messages are identified by the RTCP packet type values ​​PT=PSFB and FMT=xxx. A value will need to be attributed to that GDRR FMT.

[0236] In one example, we extend the example in section 7.3 of RFC5104 with support for RRPR and GDRR capability exchange as part of the offer / answer for codec control messages, including audio (e.g., G.711) and video (H.263) codecs. The offerer wishes to support "tstr (Temporal Spatial Trade-off)", "fir (Full Intra Request)", "gdrr" and "rrpr". The offered SDP is as follows: ------------->Offer v=0 o=alice 3203093520 3203093520 IN IP4 host.example.com s=Offer / Answer c=IN IP4 192.0.2.124 m=audio 49170 RTP / AVP 0 a=rtpmap:0 PCMU / 8000 m=video 51372 RTP / AVPF 98 a=rtpmap:98 H263-1998 / 90000 a=rtcp-fb:98 ccm tstr a=rtcp-fb:98 ccm fir a=rtcp-fb:*ccm tmmbr smaxpr=120 a=rtcp-fb:98 ccm gdrr a=rtcp-fb:98 ccm rrpr

[0237] The answerer wishes to support only the GDRR message as part of its additional capabilities, and the answered SDP is: <----------------Answer v=0 o=alice 3203093520 3203093524 IN IP4 otherhost.example.com s=Offer / Answer c=IN IP4 192.0.2.37 m=audio 47190 RTP / AVP 0 a=rtpmap:0 PCMU / 8000 m=video 53273 RTP / AVPF 98 a=rtpmap:98 H263-1998 / 90000 a=rtcp-fb:98 ccm gdrr

[0238] Several embodiments have been described above. The features of these embodiments may be provided alone or in any combination. Furthermore, the embodiments may include one or more of the following features, devices, or aspects, alone or in any combination, across various claim categories and types. - A bitstream or signal including one or more of the syntax elements described, or variations thereof. - Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the described syntax elements, or variations thereof. A television, game console, mobile phone, tablet, or other electronic device running at least one of the described embodiments. ● A television, games console, mobile phone, tablet, or other electronic device that performs at least one of the described embodiments and displays the resulting picture (e.g., using a monitor, screen, or other type of display). ● A television, game console, mobile phone, tablet, or other electronic device that tunes a channel (e.g., using a tuner) to receive a signal including an encoded video stream and executes at least one of the described embodiments. ● A television, set-top box, mobile phone, tablet, or other electronic device that receives a signal containing an encoded video stream wirelessly (e.g., using an antenna) and executes at least one of the described embodiments.< / video>

Claims

1. 1. A method in a client device, comprising: selecting a quality of experience identifier, the identifier being associated with a group of constraint flag values, each constraint flag value of the group indicating whether to activate or deactivate an associated encoding tool of a video encoder; sending said identifier to a server; and receiving from the server video data encoded with an encoding tool that has been activated or deactivated in response to the group of constraint flag values ​​associated with the transmitted identifier.

2. 2. The method of claim 1 , wherein the identifier is used to define a decoding decision that enables the video decoder to adjust the video decoder to skip at least one decoding process specified in the video data or to add at least one additional decoding process to the decoding processes specified in the video data.

3. The method of claim 1 , wherein the identifier is transmitted using a Session Description Protocol.

4. 4. The method of claim 3, wherein the identifier is selected after a stage of capabilities exchange with the server, including receipt of a Session Description Protocol message from the server specifying a set of encoding tools that may be activated or deactivated, or a quality of experience that may be selected.

5. The method of claim 1 , wherein prior to selecting the identifier, the method comprises obtaining a description of a set of quality of experience options that may be selected in an SEI message.

6. The method of claim 5 , wherein each quality of experience in the set is associated with a corresponding group of constraint flags and / or at least one encoding decision and / or at least one decoding decision.

7. 1. A method in a server device, comprising: receiving, from a remote system, an identifier of a quality of user experience, the identifier being associated with a group of constraint flag values, each constraint flag value of the group indicating whether to activate or deactivate an associated encoding tool of a video encoder; encoding video data using an activated or deactivated encoding tool in response to the group of constraint flag values ​​associated with the received identifier; transmitting the encoded video data to the remote system.

8. The method of claim 7 , wherein the video data is encoded by a video encoder adjusted based on the identifier.

9. The method of claim 8 , wherein the adjusting comprises activating or deactivating an encoding tool associated with the group of constraint flags depending on the constraint flag value specified by the identifier.

10. The method of claim 7 , wherein the identifier is used to specify at least one encoding decision that allows defining a specific implementation of a video encoder independent of the constraint flag value and any profile.

11. The method of claim 7 , wherein the identifier is received from the remote system in a Session Description Protocol message.

12. 8. The method of claim 7, wherein the identifier is received after a stage of capabilities exchange with the remote system, the stage including sending a Session Description Protocol message to the remote system specifying a set of coding tools that may be activated or deactivated, or a quality of experience that may be selected.

13. The method of claim 7 , wherein prior to receiving the identifier, the method comprises sending to the remote system a description of a set of quality of experience options that may be selected in an SEI message.

14. The method of claim 13 , wherein each quality of experience in the set is associated with a corresponding group of constraint flags and / or at least one encoding decision and / or at least one decoding decision.

15. A non-transitory information storage medium that stores a signal including an identifier of a user experience quality, the identifier being associated with a group of constraint flag values, each constraint flag value of the group indicating whether to activate or deactivate an associated encoding tool of a video encoder.

16. A device comprising an electronic circuit, the electronic circuit comprising: selecting a quality of experience identifier, the identifier being associated with a group of constraint flag values, each constraint flag value of the group indicating whether to activate or deactivate an associated encoding tool of a video encoder; sending said identifier to a server; and receiving from the server video data encoded with an activated or deactivated encoding tool in response to the group of constraint flag values ​​associated with the transmitted identifier.

17. 17. The device of claim 16, wherein the identifier is used to define a decoding decision that enables the video decoder to adjust the video decoder to skip at least one decoding process specified in the video data or to add at least one additional decoding process to a decoding process specified in the video data.

18. The device of claim 16 , wherein the identifier is transmitted using a Session Description Protocol.

19. 20. The device of claim 18, wherein the identifier is selected after a stage of capabilities exchange with the server including receipt of a Session Description Protocol message from the server specifying a set of encoding tools that may be activated or deactivated or a quality of experience that may be selected.

20. The device of claim 16 , wherein the electronic circuitry is adapted to obtain, before selecting the identifier, a description of a set of quality of experience that can be selected in an SEI message.

21. The device of claim 20 , wherein each quality of experience in the set is associated with a corresponding group of constraint flags and / or at least one encoding decision and / or at least one decoding decision.

22. A device comprising an electronic circuit, the electronic circuit comprising: receiving, from a remote system, an identifier of a quality of user experience, the identifier being associated with a group of constraint flag values, each constraint flag value of the group indicating whether to activate or deactivate an associated encoding tool of a video encoder; encoding video data using an activated or deactivated encoding tool in response to the group of constraint flag values ​​associated with the received identifier; and transmitting the encoded video data to the remote system.

23. The device of claim 22 , wherein the video data is encoded by a video encoder adjusted based on the identifier.

24. The device of claim 23 , wherein the adjusting comprises activating or deactivating an encoding tool associated with the group of constraint flags depending on the constraint flag value specified by the identifier.

25. 23. The device of claim 22, wherein the identifier is used to specify at least one encoding decision that enables defining a particular implementation of a video encoder independent of the constraint flag value and any profile.

26. 23. The device of claim 22, wherein the identifier is received from the remote system in a Session Description Protocol message.

27. 23. The device of claim 22, wherein the identifier is received after a stage of capabilities exchange with the remote system including sending a Session Description Protocol message to the remote system specifying a set of coding tools that may be activated or deactivated or a quality of experience that may be selected.

28. 23. The device of claim 22, wherein the electronic circuitry is configured to send to the remote system a description of a set of quality of experience that may be selected in an SEI message prior to receiving the identifier.

29. The device of claim 28 , wherein each quality of experience in the set is associated with a corresponding group of constraint flags and / or at least one encoding decision and / or at least one decoding decision.

30. A non-transitory information storage medium storing program code instructions for implementing the method of claim 1.

31. A non-transitory information storage medium storing program code instructions for implementing the method of claim 7.

Images