Indication of importance of video codec and configuration of awareness of radio access network

Abstract

An apparatus, method, and system for video codec importance indication and RAN-aware configuration is disclosed. The apparatus (1200) includes a processor (1205) for detecting a plurality of video coded network abstraction layer ("NAL") units of a video coded stream, extracting semantic information associated with the plurality of NAL units, combining the extracted semantic information associated with the plurality of NAL units to form a plurality of feature sets correspondingly synchronized with the plurality of NAL units containing the extracted semantic information, determining an information vs. importance value for each of the plurality of NAL units based on the plurality of feature sets without performing video decoding of the video coded stream, and indicating the determined information vs. importance value for each of the plurality of NAL units to a video coded traffic aware transceiver for scheduling video traffic based on the indicated information vs. importance value.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE The subject matter disclosed herein relates generally to wireless communications, and more particularly to indication of importance of video codecs and configuration awareness of radio access networks. [Background technology]

[0002] In wireless networks, emerging applications such as augmented reality ("AR") / virtual reality ("VR") / extended reality ("XR"), cloud gaming ("CG"), device tele-operation (e.g., vehicle tele-operation, robotic arm tele-operation, etc.), 3D video conferencing, smart distance education, etc. are expected to drive the growth of video traffic. Even though the aforementioned applications may require different quantitative constraints and configurations in terms of rate, reliability, latency, and quality of service ("QoS"), such constraint sets are expected to challenge current and future communication networks in providing high fidelity quality of experience ("QoE") at ever-increasing resolutions. As the quality of rendering end devices improves and their costs fall over time, such applications are expected to steadily spread and further raise the bar of QoE for end applications. Therefore, there is a high interest in providing scalable and reliable solutions from the perspective of communication networks for next-generation media content delivery systems and their immersive digital reality applications. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] 3GPP Technical Report TR 26.928 (v16.0.0 - November 2020), 5G; Extended Reality (XR) in 5G

Non-Patent Document 2

Non-Patent Document 3

Non-Patent Document 4

Non-Patent Document 5

Non-Patent Document 6

[0004] Disclosed are procedures for video codec importance indication and radio access network ("RAN") aware configuration, which may be implemented by an apparatus, system, method, and / or computer program product.

[0005] In one embodiment, a first device includes a processor that detects a plurality of video coded network abstraction layer ("NAL") units of a video coded stream according to defined syntax elements of a determined video codec specification. In one embodiment, the processor extracts semantic information associated with the plurality of NAL units according to syntax elements and a semantic knowledge base defined within the determined video codec specification. In one embodiment, the processor combines the extracted semantic information associated with the plurality of NAL units to form a plurality of feature sets that are correspondingly synchronized with the plurality of NAL units that encompass the extracted semantic information. In one embodiment, the processor determines an information-to-importance value of each of the plurality of NAL units based on the plurality of feature sets without performing video decoding of the video coded stream. In one embodiment, the processor indicates the determined information-to-importance value of each of the plurality of NAL units of the determined video codec specification to a video coded traffic-aware transceiver for scheduling video traffic based on the indicated information-to-importance value.

[0006] In one embodiment, the first method includes detecting a plurality of video coded NAL units of a video coded stream according to defined syntax elements of a determined video codec specification. In one embodiment, the first method includes extracting semantic information associated with the plurality of NAL units according to defined syntax elements and a semantic knowledge base within the determined video codec specification. In one embodiment, the first method includes combining the extracted semantic information associated with the plurality of NAL units to form a plurality of feature sets that are correspondingly synchronized with the plurality of NAL units encompassing the extracted semantic information. In one embodiment, the first method includes determining an information vs. importance value of each of the plurality of NAL units based on the plurality of feature sets without performing video decoding of the video coded stream. In one embodiment, the first method includes indicating the determined information vs. importance value of each of the plurality of NAL units of the determined video codec specification to a video coded traffic aware transceiver for scheduling video traffic based on the indicated information vs. importance value.

[0007] In one embodiment, the second device includes a processor that encodes and compresses the uncompressed video sequence into a video coded stream formed from a plurality of NAL units using a selected video codec specification. In one embodiment, the processor extracts semantic information associated with the plurality of NAL units according to syntax elements and a semantic knowledge base defined within the determined video codec specification. In one embodiment, the processor combines the extracted semantic information associated with the plurality of NAL units to form a plurality of feature sets that are correspondingly synchronized with the plurality of NAL units encompassing the extracted semantic information. In one embodiment, the processor determines an information versus importance value for each of the plurality of NAL units based on the plurality of feature sets without performing video decoding of the video coded stream. In one embodiment, the processor annotates the plurality of NAL units with importance values ​​to form a plurality of application data units ("ADUs") of the video coded stream for a packet-switched communication network. In one embodiment, the processor signals the importance values ​​of the plurality of ADUs and the plurality of underlying NAL units to a video coded traffic aware transceiver.

[0008] In one embodiment, the second method includes encoding and compressing the uncompressed video sequence into a video coded stream formed from a plurality of NAL units using a selected video codec specification. In one embodiment, the second method includes extracting semantic information associated with the plurality of NAL units according to syntax elements and a semantic knowledge base defined within the determined video codec specification. In one embodiment, the second method includes combining the extracted semantic information associated with the plurality of NAL units to form a plurality of feature sets that are correspondingly synchronized with the plurality of NAL units encompassing the extracted semantic information. In one embodiment, the second method includes determining an information versus importance value for each of the plurality of NAL units based on the plurality of feature sets without performing video decoding of the video coded stream. In one embodiment, the second method includes annotating the plurality of NAL units with importance values ​​to form a plurality of ADUs of the video coded stream for a packet switched communication network. In one embodiment, the second method includes signaling the importance values ​​of the plurality of ADUs and the plurality of underlying NAL units to a video coded traffic aware transceiver. In one embodiment, the second method includes processing video-coded traffic awareness information to determine optimized radio scheduling and control procedures for transmission and reception operations for the ADUs of the video-coded stream, and transmitting the ADUs of the video-coded stream to the RAN based on the video-coded traffic aware optimization.

[0009] A more particular description of the embodiments briefly described above will be made by reference to specific embodiments that are illustrated in the accompanying drawings, in which the embodiments will be described and explained with greater specificity and detail, with the understanding that these drawings illustrate only some embodiments and therefore should not be considered limiting in scope. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic block diagram illustrating an embodiment of a wireless communication system for video codec importance indication and RAN awareness configuration. [Diagram 2] A diagram illustrating a split rendering architecture for mobile networks based on edge / cloud video application servers and XR UE devices. [Diagram 3] FIG. 1 is a simplified block diagram of a generic video codec that performs both spatial and temporal (motion) compression of a video source. [Figure 4] FIG. 1 illustrates one embodiment of an overview of a communications system architecture. [Diagram 5] A diagram showing a video coded elementary stream and its partitioning into multiple NAL units. [Figure 6] FIG. 2 is a block diagram of syntax parsing and semantic extraction of information from a video encoded elementary stream. [Figure 7] FIG. 2 illustrates a hierarchical-P GoP structure, a hierarchical-B GoP structure, and various conventional frame-to-slice partitionings. [Figure 8] A diagram showing a graph model of directed information flow for NAL units of a hybrid modern video coded elementary stream having a hierarchical multi-level structure including a video layer level, a sequence layer level, a video frame level, and a video slice level, utilizing interrelationships and references of video coded data at the slice level to determine the individual importance of NAL units without decoding. [Figure 9] FIG. 1 shows a graph representation of a directed information flow model for a video coded stream with two temporal layers. [Figure 10]FIG. 1 illustrates a set of embodiments of a video codec-aware RAN configuration. [Figure 11] FIG. 2 is a block diagram illustrating an embodiment of a user equipment device that may be used for video codec importance indication and RAN awareness configuration. [Figure 12] FIG. 2 is a block diagram illustrating an embodiment of a network device that may be used for video codec importance indication and RAN awareness configuration. [Figure 13] 1 is a flow diagram illustrating one embodiment of a method for video codec importance indication and RAN awareness configuration. [Figure 14] 1 is a flow diagram illustrating an embodiment of another method for video codec importance indication and RAN awareness configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] As will be appreciated by one skilled in the art, aspects of the embodiments may be embodied as a system, apparatus, method, or program product. Accordingly, the embodiments may take the form of an all-hardware embodiment, an all-software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects.

[0012] For example, the disclosed embodiments may be implemented as a hardware circuit comprising custom very large scale integrated ("VLSI") circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. The disclosed embodiments may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, etc. As another example, the disclosed embodiments may include one or more physical or logical blocks of executable code which may be organized, for example, as an object, procedure, or function.

[0013] Furthermore, embodiments may take the form of a program product embodied in one or more computer readable storage devices storing machine readable code, hereinafter referred to as code, computer readable code, and / or program code. The storage devices may be tangible, non-transitory, and / or non-transmitting. The storage devices may not embody signals. In certain embodiments, the storage devices employ only signals to access the code.

[0014] Any combination of one or more computer readable media may be utilized. The computer readable medium may be a computer readable storage medium. The computer readable storage medium may be a storage device that stores the code. The storage device may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, holographic, micro-mechanical, or semiconductor system, apparatus, or device, or any suitable combination thereof.

[0015] More specific examples (non-exhaustive list) of storage devices include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory ("RAM"), a read-only memory ("ROM"), an erasable programmable read-only memory ("EPROM" or flash memory), a portable compact disk read-only memory ("CD-ROM"), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the context of this specification, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0016] The code for carrying out the operations of the embodiments may be any number of lines and may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Python, Ruby, Java, Smalltalk, C++, and conventional procedural programming languages ​​such as the "C" programming language, and / or machine code such as assembly language. The code may run entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the last scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network ("LAN"), wireless LAN ("WLAN"), or wide area network ("WAN"), or a connection to an external computer may be made (e.g., via the Internet using an Internet Service Provider ("ISP").

[0017] Furthermore, the described features, structures, or characteristics of the embodiments may be combined in any suitable manner. In the following description, numerous specific details, such as examples of programming, software modules, user selections, network transactions, database queries, database structures, hardware modules, hardware circuits, hardware chips, etc., are provided to provide a thorough understanding of the embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other cases, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the embodiments.

[0018] Reference throughout this specification to "one embodiment," "an embodiment," or similar language means that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment. Thus, throughout this specification, the appearances of the phrases "in one embodiment," "in an embodiment," and similar language may, but do not necessarily, all refer to the same embodiment and may mean "one or more, but not all, embodiments" unless otherwise specified. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specified. An enumerated list of items does not imply that any or all of the items are mutually exclusive, unless otherwise specified. Additionally, the terms "a," "an," and "the" refer to "one or more," unless otherwise specified.

[0019] As used herein, a list using the conjunction "and / or" includes any single item in the list or a combination of items in the list. For example, a list of A, B, and / or C includes only A, only B, only C, a combination of A and B, a combination of B and C, a combination of A and C, or a combination of A, B, and C. As used herein, a list using the term "one or more of" includes any single item in the list or a combination of items in the list. For example, one or more of A, B, and C includes only A, only B, only C, a combination of A and B, a combination of B and C, a combination of A and C, or a combination of A, B, and C. As used herein, a list using the term "one of" includes only one of any single item in the list. For example, "one of A, B, and C" includes only A, only B, or only C, and excludes the combination of A, B, and C. As used herein, "a member selected from the group consisting of A, B, and C" includes only one of A, B, or C, and excludes the combination of A, B, and C. As used herein, "a member selected from the group consisting of A, B, and C and combinations thereof" includes only A, only B, only C, the combination of A and B, the combination of B and C, the combination of A and C, or the combination of A, B, and C.

[0020] Aspects of the embodiments are described below with reference to schematic flow charts and / or schematic block diagrams of methods, apparatus, systems, and program products according to the embodiments. It will be understood that each block of the schematic flow charts and / or schematic block diagrams, and combinations of blocks in the schematic flow charts and / or schematic block diagrams, may be implemented by code. This code may be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing apparatus to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus produce means for performing the functions / acts specified in the flow charts and / or block diagrams.

[0021] The code may be stored in a storage device that can instruct a computer, other programmable data processing apparatus, or other device to function in a particular manner, such that the instructions stored in the storage device produce an article of manufacture that includes instructions that perform the functions / acts specified in the flowcharts and / or block diagrams.

[0022] The code may be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to produce a computer-implemented process such that the code executing on the computer or other programmable apparatus provides a process for performing the functions / acts specified in the flow charts and / or block diagrams.

[0023] The flowcharts and / or block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and program products according to various embodiments. In this regard, each block in the flowcharts and / or block diagrams may represent a module, segment, or portion of code that includes one or more executable instructions of code for implementing the specified logical function(s).

[0024] It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may be executed in the reverse order, depending on the functionality involved. Other steps and methods may be conceived that are equivalent in function, logic, or effect to one or more blocks of the illustrated figures, or portions thereof.

[0025] Various arrow types and line types may be used in the flow diagrams and / or block diagrams, but they are understood not to limit the scope of the corresponding embodiment. Indeed, some arrows or other connectors may be used to indicate only the logical flow of the illustrated embodiment. For example, arrows may indicate waiting or monitoring periods of unspecified duration between recited steps of the illustrated embodiment. It is also noted that each block of the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or acts, or a combination of dedicated hardware and code.

[0026] The description of the elements in each figure may refer to the elements in the procedure figures. Like numbers refer to like elements in all figures, including alternative embodiments of like elements.

[0027] Generally, this disclosure describes systems, methods, and apparatus for video codec importance indication and RAN awareness configuration. In certain embodiments, the methods may be performed using computer code embodied in a computer readable medium. In certain embodiments, the apparatus or system may include a computer readable medium including computer readable code that, when executed by a processor, causes the apparatus or system to perform at least a portion of the solutions described below.

[0028] Emerging applications such as Augmented Reality ("AR") / Virtual Reality ("VR") / Extended Reality ("XR"), Cloud Gaming ("CG"), device remote teleoperation (e.g., vehicle teleoperation, robotic arm teleoperation, etc.), 3D video conferencing, smart distance education, etc. are expected to drive the growth of video traffic. Even though the aforementioned applications may require different quantitative constraints and configurations in terms of rate, reliability, latency, and quality of service ("QoS"), such constraint sets are expected to challenge current and future communication networks in providing high fidelity Quality of Experience ("QoE") at ever-increasing resolutions. As the quality of rendering end devices improves and their cost drops over time, such applications are expected to steadily spread and further raise the bar of QoE for end applications. Hence, there is a high interest in providing scalable and reliable solutions from a communication network perspective for next-generation media content delivery systems and their immersive digital reality applications.

[0029] Communication networks are one crucial component of such applications. Another key technology in scaling the deployment of these immersive media experiences is video encoding and compression of source video information. This is crucial in reducing the size of raw image data to a point where communication systems can reliably transmit video content over a variety of challenging network conditions associated with mobile and wireless data systems and applications. Currently, the communication plane is completely separate from the encoding plane of the video source, which makes optimizing transmission strategies for reliable QoE for such video-intensive applications difficult and / or limited. Despite current advances in video codec development (e.g., H.266 standard release), the data growth rate of high-resolution multiview / AR / XR / 3D applications outpaces the compression gain. Thus, there is interest in developing mechanisms to assist communication and access networks in understanding codec-specific semantics and leveraging the latter in designing and configuring optimized transmission strategies for the aforementioned video applications.

[0030] In one embodiment, the present disclosure proposes a method for deriving the importance of data units of a video codec without decoding the video coded data stream, based on video codec syntax analysis and syntax element extraction, semantic analysis and extraction of information as universal features of video coded data units, directed information flow processing through a graph of importance per video coded data unit according to kernel information-to-importance function metric and cumulative importance considering the hierarchical and encapsulation structure of modern hybrid video codecs, and decoration of video data units with importance information for further processing by other network and RAN level blocks. In a further embodiment, a system for configuring RAN functionality based on video codec recognition is disclosed, subject to extraction of importance of video coded data units with reduced latency without decoding the video elementary stream.

[0031] FIG. 1 illustrates a wireless communication system 100 for configuration of video codec importance indication and RAN awareness according to an embodiment of the disclosure. In one embodiment, the wireless communication system 100 includes at least one remote unit 105, a fifth generation radio access network ("5G-RAN") 115, and a mobile core network 140. The 5G-RAN 115 and the mobile core network 140 form a mobile communication network. The 5G-RAN 115 may be comprised of a 3rd Generation Partnership Project ("3GPP") access network 120 including at least one cellular base unit 121, and / or a non-3GPP access network 130 including at least one access point 131. The remote unit 105 communicates with the 3GPP access network 120 using a 3GPP communication link 123 and / or with the non-3GPP access network 130 using a non-3GPP communication link 133. Although a particular number of remote units 105, 3GPP access networks 120, cellular base units 121, 3GPP communication links 123, non-3GPP access networks 130, access points 131, non-3GPP communication links 133, and mobile core networks 140 are shown in FIG. 1, those skilled in the art will recognize that any number of remote units 105, 3GPP access networks 120, cellular base units 121, 3GPP communication links 123, non-3GPP access networks 130, access points 131, non-3GPP communication links 133, and mobile core networks 140 may be included in the wireless communications system 100.

[0032] In one implementation, the RAN 120 complies with a 5G system defined in 3GPP specifications. For example, the RAN 120 may be a NextGen RAN ("NG-RAN") that implements a New Radio ("NR") radio access technology ("RAT") and / or a Long Term Evolution ("LTE") RAT. In another example, the RAN 120 may include a non-3GPP RAT (e.g., Wi-Fi or a WLAN compliant with the Institute of Electrical and Electronics Engineers ("IEEE") 802.11 family). In another implementation, the RAN 120 complies with an LTE system defined in 3GPP specifications. However, more broadly, the wireless communication system 100 may implement any other open or proprietary communication network, such as the Worldwide Interoperability for Microwave Access ("WiMAX") or IEEE 802.16 family of standards, among other networks. This disclosure is not intended to be limited to any particular wireless communications system architecture or protocol implementation.

[0033] In one embodiment, the remote unit 105 may include a computing device such as a desktop computer, a laptop computer, a personal digital assistant ("PDA"), a tablet computer, a smartphone, a smart television (e.g., a television connected to the Internet), a smart appliance (e.g., an appliance connected to the Internet), a set-top box, a game console, a security system (including security cameras), an in-vehicle computer, a network device (e.g., a router, a switch, a modem), etc. In some embodiments, the remote unit 105 includes a wearable device such as a smart watch, a fitness band, an optical head mounted display, etc. In one embodiment, the remote unit 105 includes a device for presenting a virtual reality environment, an augmented reality environment, and / or an extended reality environment, e.g., a head mounted display unit.

[0034] Additionally, the remote unit 105 may be referred to as a user equipment (UE) device, a subscriber unit, a mobile phone, a mobile station, a user, a terminal, a mobile terminal, a fixed terminal, a subscriber station, a user terminal, a wireless transmit / receive unit ("WTRU"), a device, or other terminology used in the art. In various embodiments, the remote unit 105 includes a subscriber identity and / or identification module ("SIM") and a mobile equipment ("ME") that provides mobile termination functions (e.g., wireless transmission, handover, voice encoding and decoding, error detection and correction, signaling, and access to the SIM). In certain embodiments, the remote unit 105 may include terminal equipment ("TE") and / or may be incorporated into a home appliance or device (e.g., a computing device as described above).

[0035] The remote unit 105 may communicate directly with one or more of the cellular base units 121 in the 3GPP access network 120 by uplink ("UL") and downlink ("DL") communication signals. Furthermore, the UL and DL communication signals may be carried on a 3GPP communication link 123. Similarly, the remote unit 105 may communicate with one or more access points 131 in the non-3GPP access network 130 by UL and DL communication signals carried on a non-3GPP communication link 133. Here, the access networks 120 and 130 are intermediate networks that provide the remote unit 105 with access to the mobile core network 140.

[0036] In some embodiments, the remote unit 105 communicates with a remote host (e.g., in the data network 150 or the data network 160) via a network connection with the mobile core network 140. For example, an application 107 (e.g., a web browser, a media client, a telephony and / or a voice over Internet protocol ("VoIP") application) in the remote unit 105 may trigger the remote unit 105 to establish a protocol data unit ("PDU") session (or other data connection) with the mobile core network 140 via the 5G-RAN 115 (e.g., via the 3GPP access network 120 and / or the non-3GPP network 130). The mobile core network 140 then relays traffic between the remote unit 105 and the remote host using the PDU session. The PDU session represents a logical connection between the remote unit 105 and the user plane function ("UPF") 141.

[0037] In order to establish a PDU session (or packet data network ("PDN") connection), the remote unit 105 must register with the mobile core network 140 (also referred to as "attached to the mobile core network" in the context of fourth generation ("4G") systems). It should be noted that the remote unit 105 may establish one or more PDU sessions (or other data connections) with the mobile core network 140. Thus, the remote unit 105 may have at least one PDU session for communicating with the packet data network 150. Additionally--or alternatively--the remote unit 105 may have at least one PDU session for communicating with the packet data network 160. The remote unit 105 may establish additional PDU sessions for communicating with other data networks and / or other communication peers.

[0038] In the context of 5G systems ("5GS"), the term "PDU session" refers to a data connection that provides end-to-end ("E2E") user plane ("UP") connectivity between a remote unit 105 and a particular data network ("DN") via the UPF 141. A PDU session supports one or more QoS flows. In certain embodiments, there may be a one-to-one mapping between QoS flows and QoS profiles such that all packets belonging to a particular QoS flow have the same 5G QoS Identifier ("5QI").

[0039] In the context of a 4G / LTE system, such as the Evolved Packet System ("EPS"), a PDN connection (also called an EPS session) provides an E2E UP connection between a remote unit and a PDN. The PDN connection procedure establishes an EPS bearer, e.g., a tunnel between the remote unit 105 and a Packet Gateway ("P-GW"), not shown, in the Evolved Packet Core Network ("EPC"). In a particular embodiment, there is a one-to-one mapping between EPS bearers and QoS profiles, such that all packets belonging to a particular EPS bearer have the same QoS Class Identifier ("QCI").

[0040] As described in further detail below, the remote unit 105 may use the first data connection (e.g., PDU session) established with the first mobile core network, EPC, to establish a second data connection (e.g., part of a second PDU session) with the second mobile core network 140. When establishing the data connection (e.g., PDU session) with the second mobile core network 140, the remote unit 105 registers with the second mobile core network 140 using the first data connection.

[0041] The cellular base units 121 may be distributed in a geographical region. In certain embodiments, the cellular base units 121 may be referred to as an access terminal, a base, a base station, a Node B ("NB"), an evolved Node B (abbreviated as eNodeB or "eNB" and also known as an evolved Universal Terrestrial Radio Access Network ("E-UTRAN") Node B), a 5G / NR Node B ("gNB"), a home Node B, a relay node, a device, or any other terminology used in the art. The cellular base units 121 are generally part of a RAN, such as a 3GPP access network 120, which may include one or more controllers communicatively coupled to one or more corresponding cellular base units 121. These and other elements of a radio access network are not shown, but are generally familiar to those skilled in the art. The cellular base units 121 connect to a mobile core network 140 via the 3GPP access network 120.

[0042] The cellular base unit 121 may serve multiple remote units 105 in a serving area, e.g., a cell or a sector of a cell, via a 3GPP wireless communication link 123. The cellular base unit 121 may directly communicate with one or more of the remote units 105 by communication signals. In general, the cellular base unit 121 transmits DL communication signals to serve the remote units 105 in the time, frequency, and / or space domains. Furthermore, the DL communication signals may be carried on the 3GPP communication link 123. The 3GPP communication link 123 may be any suitable carrier in a licensed or unlicensed radio spectrum. The 3GPP communication link 123 facilitates communication between one or more of the remote units 105 and / or one or more of the cellular base units 121. It should be noted that during NR operation over an unlicensed spectrum (referred to as “NR-U”), the base unit 121 and the remote unit 105 communicate over an unlicensed (e.g., shared) radio spectrum.

[0043] The non-3GPP access networks 130 may be distributed in a geographical region. Each non-3GPP access network 130 may serve a number of remote units 105 within a serving area. An access point 131 of the non-3GPP access network 130 may directly communicate with one or more remote units 105 by receiving UL communication signals and transmitting DL communication signals to serve the remote units 105 in the time, frequency, and / or spatial domains. Both DL and UL communication signals are carried on a non-3GPP communication link 133. The 3GPP communication link 123 and the non-3GPP communication link 133 may employ different frequencies and / or different communication protocols. In various embodiments, the access point 131 may communicate using an unlicensed radio spectrum. The mobile core network 140 may serve the remote units 105 via the non-3GPP access networks 130 as described in more detail herein.

[0044] In some embodiments, the non-3GPP access network 130 connects to the mobile core network 140 via an interworking entity 135. The interworking entity 135 provides interworking between the non-3GPP access network 130 and the mobile core network 140. The interworking entity 135 supports connections via "N2" and "N3" interfaces. As shown, both the 3GPP access network 120 and the interworking entity 135 communicate with an Access and Mobility Management Function ("AMF") 143 using the "N2" interface. The 3GPP access network 120 and the interworking entity 135 also communicate with the UPF 141 using the "N3" interface. Although shown as being outside the mobile core network 140, in other embodiments, the interworking entity 135 may be part of the core network. Although shown as being outside the non-3GPP RAN 130 , in other embodiments, the interworking entity 135 may be part of the non-3GPP RAN 130 .

[0045] In certain embodiments, the non-3GPP access network 130 may be controlled by the operator of the mobile core network 140 and may have direct access to the mobile core network 140. Such a deployment of a non-3GPP AN is referred to as a "trusted non-3GPP access network." A non-3GPP access network 130 is considered "trusted" when it is operated by a 3GPP operator or a trusted partner and supports certain security features such as strong air interface encryption. In contrast, a deployment of a non-3GPP AN that is not controlled by the operator (or trusted partner) of the mobile core network 140, does not have direct access to the mobile core network 140, or does not support certain security features is referred to as an "untrusted" non-3GPP access network. An interworking entity 135 deployed in the trusted non-3GPP access network 130 may be referred to herein as a Trusted Network Gateway Function ("TNGF"). The interworking entity 135 deployed in the untrusted non-3GPP access network 130 may be referred to herein as a non-3GPP interworking function ("N3IWF"). Although shown as part of the non-3GPP access network 130, in some embodiments the N3IWF may be part of the mobile core network 140 or may be located in the data network 150.

[0046] In one embodiment, the mobile core network 140 is a 5G Core ("5GC") or EPC that may be coupled to a data network 150, such as the Internet and a private data network, among other data networks. The remote units 105 may have a subscription or other account with the mobile core network 140. Each mobile core network 140 belongs to a single public land mobile network ("PLMN"). This disclosure is not intended to be limited to any particular wireless communications system architecture or protocol implementation.

[0047] The mobile core network 140 includes several network functions ("NFs"). As shown, the mobile core network 140 includes at least one UPF 141. The mobile core network 140 also includes a number of control plane functions, including but not limited to an AMF 143 that provides services to the 5G-RAN 115, a session management function ("SMF") 145, a policy control function ("PCF") 147, an authentication server function ("AUSF") 148, a unified data management ("UDM"), and a unified data repository function ("UDR").

[0048] The UPF 141 is responsible for packet routing and forwarding, packet inspection, handling of QoS, and external PDU sessions for interconnecting data networks ("DNs") in the 5G architecture. The AMF 143 is responsible for termination of non-access stratum ("NAS") signaling, encryption & integrity protection of NAS, registration management, connection management, mobility management, access authentication and authorization, security context management. The SMF 145 is responsible for session management (e.g., session establishment, modification, release), allocation & management of remote unit (e.g., UE) IP addresses, DL data notification, and traffic steering configuration of the UPF for proper traffic routing.

[0049] The PCF 147 is responsible for providing a unified policy framework, policy rules to the control plane ("CP") function, and access subscription information for policy decisions in the UDR. The AUSF 148 acts as an authentication server.

[0050] The UDM is responsible for Authentication and Key Agreement ("AKA") credential generation, user identification, access authorization, and subscription management. The UDR is a repository of subscriber information and can be used to provide services to many network functions. For example, the UDR may store subscription data, policy related data, subscriber related data that is allowed to be exposed to third party applications, etc. In some embodiments, the UDM is co-located with the UDR and is denoted as a combined entity "UDM / UDR" 149.

[0051] In various embodiments, the mobile core network 140 may also include a Network Exposure Function ("NEF") (responsible for making network data and resources easily accessible by customers and network partners, for example, via one or more application programming interfaces ("APIs"), a Network Repository Function ("NRF") (which provides registration and discovery of NF services, allowing NFs to identify each other's appropriate services and communicate with each other via APIs), or other NFs defined for 5GC. In particular embodiments, the mobile core network 140 may include an Authentication, Authorization, and Accounting ("AAA") server.

[0052] In various embodiments, the mobile core network 140 supports different types of mobile data connections and different types of network slices, each of which utilizes a particular network slice. Here, a "network slice" refers to a portion of the mobile core network 140 optimized for a particular traffic type or communication service. A network instance may be specified by a single Network Slice Selection Assistance Information ("S-NSSAI"), while a set of network slices that the remote unit 105 is authorized to use is specified by the NSSAI. In certain embodiments, the various network slices may include separate instances of network functions, such as the SMF and UPF 141. In some embodiments, the different network slices may share some common network functions, such as the AMF 143. For ease of illustration, the different network slices are not shown in FIG. 1, but their support is assumed.

[0053] In one embodiment, the network 100 includes an application server 142 that hosts applications for use by the mobile network 140, the RAN 115, the remote units 105, etc. As pertinent to the subject matter disclosed herein, the application server 142 may host video codec-aware applications used to determine and indicate the importance of underlying NAL units of a video encoded elementary stream. The importance indicators may be located in the mobile network 140 (e.g., in the UPF 141), in the RAN 115 (e.g., in higher layers), etc.

[0054] 1, those skilled in the art will recognize that any number and type of network functions may be included in the mobile core network 140. Furthermore, if the mobile core network 140 includes an EPC, the depicted network functions may be replaced by appropriate EPC entities, such as a Mobility Management Entity ("MME"), a Serving Gateway ("S-GW"), a P-GW, a Home Subscriber Server ("HSS"), etc.

[0055] 1 shows components of a 5G RAN and a 5G core network, the described embodiments for using pseudonyms for access authentication over non-3GPP access apply to other types of communication networks and RATs, including IEEE 802.11 variants, GSM, GPRS, UMTS, LTE variants, CDMA 2000, Bluetooth, ZigBee, Sigfox, etc. For example, in a 4G / LTE variant including an EPC, the AMF 143 may be mapped to the MME, the SMF may be mapped to the control plane portion of the P-GW and / or the MME, the UPF 141 may be mapped to the S-GW and user plane portion of the P-GW, the UDM / UDR 149 may be mapped to the HSS, etc.

[0056] As shown, a remote unit 105 (e.g., a UE) may connect to a mobile core network (e.g., a 5G mobile communications network) via two types of access: (1) via a 3GPP access network 120 and (2) via a non-3GPP access network 130. The first type of access (e.g., the 3GPP access network 120) uses a 3GPP-defined type of wireless communication (e.g., NG-RAN), and the second type of access (e.g., the non-3GPP access network 130) uses a non-3GPP-defined type of wireless communication (e.g., WLAN). The 5G-RAN 115 refers to any type of 5G access network that can provide access to the mobile core network 140, including the 3GPP access network 120 and the non-3GPP access network 130.

[0057] By way of background, the common framework adopted at the 3GPP level for immersive XR and high performance video content transmission, e.g., 3GPP Technical Report TR 26.928 (v16.0.0 - November 2020), 5G; Extended Reality (XR) in 5G, S4-211210: [FS_XRTraffic]: Permanent document, v0.8.0, reported by Qualcomm Inc., (2021), relies on the concept of split rendering. It uses an application server located at the edge and connected to the core network ("CN"), which is used to encode application video content and forward it to the RAN for mobile communication. In return, the RAN communicates with the connected UE, which may use additional hardware / software processing to render the video content to match the user's pose / input / control state. This architectural approach is shown in Figure 2 for reference.

[0058] FIG. 2 shows a split rendering architecture for a mobile network based on an edge / cloud video application server and an XR UE device 203. The device 203 is connected to a radio access network 208, which is further connected to an application server 202 via a core network 205. The application server 202 may deliver XR media based on local XR processed content or based on remote XR processed content. The processing may take into account and / or further process tracking and sensing information uplinked by the XR UE device 203. The application server 202 streams the XR multimedia content via a content delivery gateway 210 to which the XR UE device 203 is connected by any real-time transport protocol. After decoding the XR content received from the application server 202, the XR device 203 may locally render the XR content on the display using the XR engine 212 of the XR device 203 and additional local hardware / software capabilities and / or XR pre-rendered content, as well as XR-related XR metadata.

[0059] In the illustrated embodiment, therefore, the video application server 202 is used to process, encode, transcode, and / or provide to the XR UE 203 local video content 204 or remote video content 206 related to the AR / XR / CG / teleoperation application session. As a result, the video application server 202 may encode / transcode and control the video viewport content and transmit it in the downlink to the RAN 208 based on UE specific parameters, configurations, and sensing inputs that may affect the rendering viewpoint, rate, quality, panning, etc. This overall architecture is expected to enable scalable AR / XR / CG / teleoperation applications and use cases with low latency, high rate, and efficient energy usage, leveraging the benefits of various compute and network domains (e.g., cloud, edge, smart handsets / headsets). The architecture is therefore universally applicable to both split rendering with asynchronous time warping, e.g., where a video application server encodes a rasterized pre-processed viewport representation to assist the UE, and split rendering with device-side viewport rendering, e.g., where a video viewport may be fully or partially rendered on the device side given the media encoded video content and its available corresponding metadata.

[0060] In one embodiment, in the context of the video coding domain, interactivity including these applications requires guarantees in terms of meeting packet error rate ("PER") and packet delay budget ("PDB") for the QoE of rendering the associated video stream at the UE. Jitter of the video source and the stochastic nature of wireless channels of mobile communication systems make the former difficult to meet, especially for certain high-rate digital video transmissions, e.g., 4K, 3D video, 2x2K eye-buffered video, etc.

[0061] In one embodiment, the current video source information is encoded based on a 2D representation of the video content. The encoded elementary stream video content is generally organized into two abstraction layers, e.g., NAL and video coding layer ("VCL"), intended to separate the storage domain and the video coding domain, respectively, regardless of the source encoder. The NAL syntax encapsulates the VCL information and provides an abstract containerization mechanism for the encoded stream during transmission, e.g., for disk storage / caching / transmission and / or parsing / decoding.

[0062] On the other hand, the VCL encapsulates the video encoding procedure of an encoder and compresses the source encoded video information based on some entropy coding method, e.g., context-adaptive binary arithmetic encoding ("CABAC"), context-adaptive variable-length coding ("CAVLC"), etc. A simplified description of the VCL procedure for generally encoding video content is as follows: pictures 302 in a video sequence are partitioned 304 into coding units of configured size (e.g., macroblocks, coding tree units, or variations thereof).The coding unit then uses some tree partitioning structures (see ITU-T Series H: Audiovisual and Multimedia Systems: Infrastructure of Audiovisual Services - Coding of Moving video, Advanced Video Coding for Generic Audiovisual Services (H.264) (v08 / 2021), ITU-T Series H: Audiovisual and Multimedia Systems: Infrastructure of Audiovisual Services - Coding of Moving video, High Efficiency Video Coding (H.265) (v08 / 2021), ITU-T Series H: Audiovisual and Multimedia Systems: Infrastructure of Audiovisual Services - Coding of Moving video, Versatile Video Coding (H.266) (v08 / 2020)), for example under a bipartite / ternary / quadtree, or according to some predefined geometrically motivated 2D segmentation patterns (see de Rivaz, available at https: / / aomediacodec.github.io / av1-spec / av1-spec.pdf). P. and Haughton, J., (2018), AV1 Bitstream & Decoding Process Specification, The Alliance for Open Media, 182), for example, it may be split under a 10-way split.

[0063] In one embodiment, the encoder uses such inter-encoding unit visual references to encode picture content in a residual-based differential manner. The residual is determined given a prediction mode related to the reconstruction of information. Two modes of prediction are universally available as intra prediction 306 (also called intra for short) or inter prediction 308 (or inter for short). The intra mode is based on deriving and predicting the residual based on the content of other encoding units in the current picture, for example by calculating the residual of the current encoding unit given the coded content of the neighboring encoding units of the current encoding unit. On the other hand, the inter mode is based on deriving and predicting the residual based on the content of encoding units from other pictures, for example by calculating the residual of the current encoding unit given the content of the neighboring coded pictures of the current encoding unit.

[0064] The residual is then further transformed for compression using some multi-dimensional (2D / 3D) spatial multi-modal transform 310, e.g., a frequency-based or wavelet-based linear transform, to extract the most prominent frequency components of the coding unit residual. The insignificant high frequency contributions of the residual are removed and the floating-point transformed representation of the remaining residual is further quantized (312) based on some parametric quantization procedure to a selected number of bits per sample, e.g., 8 / 10 / 12 bits. Finally, the transformed and quantized residuals and their associated motion vectors relative to their prediction references in either intra or inter mode are encoded using an entropy encoding mechanism to compress the information based on the probability distribution of the source bit content. The output of this operation is a bitstream 316 of the VCL coded residual content. A simplified comprehensive diagram of the blocks of a modern hybrid (applying both temporal and spatial compression with intra / inter prediction) video codec is shown in FIG. 3.

[0065] FIG. 3 shows a generic video codec that performs both spatial and temporal (motion) compression of a video source. Encoder blocks are captured in the domain tagged "encoder". Decoder blocks are captured in the light grey domain tagged "decoder". Those skilled in the art may associate the above generic diagram describing a hybrid codec with a number of state-of-the-art video codecs, such as but not limited to MPEG-1, MPEG-2, MPEG-4 (collectively referred to as MPEG-x), H.264, H.265, H.266 (collectively referred to as H.26x), or VP8 / VP9 / AV1. Thus, the concepts utilized herein are considered in a broad sense hereafter, unless specifically clarified and scoped to any codec embodiment.

[0066] The coded residual bitstream is then encapsulated in an elementary stream as NAL units ready for storage or transmission over a network. NAL units are the main syntax elements of a video codec; they may encapsulate encoded video metadata, e.g., video / sequence / picture parameter sets ("VPS" / "SPS" / "PPS"), supplemental enhancement information ("SEI") messages, etc., as well as encoded video headers and residual data, typically as picture slices (pictures or similarly partitions of a video frame). The typical syntax of the encapsulation carries information described by codec-specific semantics that are intended to determine the usage of the metadata and video encoded data and to aid the decoding process.

[0067] The NAL unit encapsulation syntax consists of a header part that determines the beginning of the NAL unit and the type of the NAL unit, and a raw byte payload sequence that contains information related to the NAL unit. The NAL unit payload is then formed from the payload syntax or payload-specific header and the associated payload-specific syntax. A very important subset of NAL units is formed from parameter sets, e.g., VPS, SPS, PPS, SEI message and configuration NAL units (collectively also known as non-VCL NAL units), as well as picture slice NAL units that contain video encoded data as VCL information. An effective decoder is Implement a bitstream parser to extract the necessary metadata information and VCL related metadata from the NAL unit sequence; Decoding the VCL residual coded data sequence into its transformed and quantized values; Apply an inverse linear transformation to recover the significant content of the residuals; Performing intra or inter prediction to reconstruct the luma and color representations of each coding unit; Applying additional filtering and error concealment procedures; · May reproduce raw picture sequence representation as video playback.

[0068] These operations and procedures may occur in the order listed, or out of order, depending on the specific implementation of the decoder.

[0069] Current RAN implementations do not consider any knowledge of the application data payloads they are required to serve. However, this level of intelligence may be necessary when dealing with AR / XR / CG / Teleoperation traffic containing multiple streams that may require different QoS flows with heterogeneous but nearly deterministic rate, latency, and reliability requirements for the provision of a satisfactory QoE. Considerations regarding video codec and application awareness at the (5th generation) 5G RAN level have started (see S4-211210: [FS_XRTraffic]: Permanent document, v0.8.0, Reporter Qualcomm Inc., (2021)), with Group of Pictures (GoP) and prediction type (e.g., intra / inter prediction) currently considered as the main characteristics for determining the relevance of video coded slices (as a partition of a picture / frame) at the RAN level. By definition, a GoP represents a temporal sequence of video frames that starts (or ends) with an intra-coded video frame (I-frame) and is followed by multiple forward predicted frames (P-frames) or bidirectionally predicted frames (B-frames). A group of picture parameter sets is further described by its associated VPS NAL unit (containing a description of video layer attributes and encoder configuration), SPS NAL unit (containing a description of sequence-level attributes and encoder configuration), and PPS NAL unit (containing a description of picture-level attributes and encoder configuration).

[0070] Specifically, the GoP-oriented importance of a slice as a video frame partition is based on whether it contains intra-coded elements (I slice) or inter-coded elements (P slice for slices with some inter-coded coding units that reference previous references, or B slice for slices with inter-coded coding units that reference previous and later references in the video sequence in both directions), as well as the offset to the last intra-coded picture in the current GoP given the cyclic GoP encoding configuration. Thus, if a slice is an I slice and belongs to the current intra-coded picture, it has the highest importance, but when the slice may be a P / B slice and belongs to an inter-coded picture further away from the last intra-coded picture, the importance decreases linearly to 1. This mechanism, as briefly highlighted in Table 1, may oversimplify some aspects of video coding and visual perception by equally weighting coding units and subsequent regions of a picture to their lowest importance, given the specifications of common video codecs.

[0071] As a result, if a slice contains both intra-coded and inter-coded coding units, the slice is de-prioritized in terms of importance even though it may contain a more significant portion of the intra-coded components that are more relevant than the inter-coded components.

[0072] [Table 1]

[0073] In one embodiment, in conjunction with layer-based streaming and dynamic control, video codecs are designed to compress video information sources under the constraint of a rate-distortion tradeoff, such that their encoding is not specifically targeted for lossy mobile wireless communications, where the characteristics of the wireless channel randomly affect the transfer of encoded bits. Given multiple network hops separating the video source from the RAN, or similarly the video decoder at the receiver, dynamic adaptation of the video source encoding based on channel statistics often involves large feedback loop delays. Scalable expansion of video codecs (at the expense of increased encoding / decoding complexity) is achieved by layered encoding using spatial and / or temporal video coding layers.

[0074] As a result, the encoded video content may be progressively encoded / transmitted / decoded, assuming some hierarchical relationship between the video layers (e.g., base video layer, enhancement layer #1, enhancement layer #2, etc.). Such encoding / decoding procedures have either been developed as extensions to existing codec families, e.g., H.264, or have been incorporated into new generation codecs, such as H.265, H.266, etc. This scalable layered approach, together with the bandwidth ("BW") occupied by each individual layer, offers the possibility of dynamic streaming adaptation of the video source rate outside the practically applicable video encoder domain.

[0075] Regarding the 3GPP RAN overview and E2E XR (video) transport architecture, in 5G NR RAN, and in previous releases of 3GPP, the lower layers of the RAN do not have specific knowledge of the traffic of the higher layers that serve as Data Radio Bearers ("DRBs") on the physical wireless channel between the CN / data network and the UE. Therefore, no optimized decisions can be made in properly scheduling the associated traffic and controlling the rates of the various application streams for transmission with the high granularity and low latency adaptability constraints required by the highly mobile environment.

[0076] Interactive multimedia applications such as AR / VR / XR with high rate and low latency constraints for the transmission of video coded data synchronized with any user interaction are such particular use cases with high QoE requirements. To provide such applications reliably and robustly, the RAN may benefit from metadata and available insight into the AR / VR / XR video traffic required to be carried over the air.

[0077] However, certain embodiments do not provide such capabilities and procedures. As shown in Figure 4, protocol data units ("PDUs") associated with XR application sessions of application servers 402 connected to the CN 404 are forwarded over Internet Protocol ("IP") through the UPF of the CN 404 to the RAN 406. Multimedia traffic may be further supported by a real-time multimedia transport protocol, such as the Real-Time Transport Protocol ("RTP"), to handle jitter, packet loss, and out-of-order delivery that may occur within the framework of a typical IP network.

[0078] The QoS associated with IP packets of the XR traffic is handled by the CN 404 via QoS flows 408 created in the UPF within the established PDU session. This procedure is opaque to the RAN 406, which only manages the mapping of QoS flows 408 associated with received IP packets to their corresponding DRBs, given the QoS profile associated with the indicators of each QoS flow 408. For example, in 5GS, the QoS flows 408 are characterized by a 5QI (see 3GPP Technical Specification TS 23.501 (V17.2.0 - September 2021), System architecture for the 5G System (5GS); Stage 2 (Release 17)).

[0079] This latter mapping of QoS flows 408 to DRBs is performed in the RAN 406 by the Service Data Adaptation Protocol ("SDAP") layer 410. The SDAP service data units ("SDUs") are then processed by the Packet Data Convergence Protocol ("PDCP") 412, which performs, among other things, header compression and encryption, and the output is further processed by the Radio Link Control ("RLC") 414, which may perform segmentation of the PDCP 412 SDUs and implements automatic request response ("ARP") repeat retransmissions. The RLC 414 SDUs are then processed over a logical channel interface by the Medium Access Control ("MAC") layer 416, which handles logical channel multiplexing, hybrid automatic repeat request ("HARQ"), scheduling, and scheduled retransmission functions. Finally, the MAC SDUs are combined into transport blocks ("TBs") at the physical ("PHY") layer 418 level via transport channels. The TBs are partitioned into equal-sized coding blocks ("CBs"), which are forward error correction (FEC) encoded by the PHY. The PHY thus handles encoding / decoding, rate matching, modulation / demodulation, multi-antenna mapping, and other typical wireless PHY layer functions.

[0080] As a result, all dynamic adaptations required at the RAN level to meet QoS flow requirements are constrained only to meet hard thresholds in terms of statistical PER, PDB or guaranteed bit rate (GBR), without any prior information of the content / importance of the transmitted data. As already outlined, for multimedia video traffic, not all packets / data segments (e.g., video frames / slices) are equally important, and even not all bits are equal in achieving a satisfactory level of QoE for the end user. For a reliable, robust, user-oriented and immersive video experience, it is therefore of interest for the RAN lower layer transmission strategies to exploit the video codec syntax and semantics to determine the quantitative importance of the video encoded data transmitted at any given time, and to support RAN adaptive protocols for scheduling and transmission configuration.

[0081] In general, the subject matter herein presents a solution for determining the importance of data units of elementary streams of video codecs based on syntactic analysis of the streams, semantic extraction of information, and processing of the extracted information with an abstract graph-based directed information flow model. The model may apply a rule-based quantitative function or a statistical learning optimized function to determine the importance of data units of the video codecs, which may then be used to annotate / tag / decorate application data units ("ADUs") of the video coded streams for enhanced and optimized processing by the network, e.g., RAN awareness and adaptation to video coded traffic.

[0082] In one embodiment, the encoded video content is encapsulated in a storage / transport abstraction layer, e.g., NAL, e.g., according to the H.26x standard. The NAL partitions the elementary video stream into NAL units 502 as the basic unit of storage of the video encoded content, either to be stored on disk or to be transmitted over a packet-based network. The NAL units 502 are further partitioned into a header portion 504 as the NAL prefix and a NAL payload 506 containing raw uncoded (VPS / SPS / PPS / SEI) or coded (slice / slice segment) byte information. The NAL unit header 504 is formed of a delimiter for identifying the NAL unit start position and information about the content of the NAL unit payload, e.g., NAL unit type, NAL byte length, video coding layer / temporal layer identifier, as highlighted in FIG. 5, which shows the video coded elementary stream and its partitioning in multiple NAL units. A NAL unit is formed from a header and a payload, which contains information about the type, size, and video coding attributes and parameters of the information contained in the NAL unit data.

[0083] By parsing the NAL unit payload 506 and interpreting the specific semantics of the payload header 504 and the entropy video non-encoded data, based on the NAL unit type, more information about the content of the NAL unit may be gleaned. This procedure may be performed in real-time with linear complexity and low latency overhead on specialized hardware embodiments or software implementations. Extracting this information may form a feature set of attributes that semantically describe the NAL unit 502 from the perspective of the video codec in order to classify / estimate / predict / predict the importance of the NAL unit 502 for playback of the encoded video elementary stream. This procedure of classification / estimation / prediction may be performed by a policy-based method (e.g., an expert or model-based system) in some embodiments, or by a learning-based method in other embodiments. The results of such a method, e.g., an importance indicator of the NAL unit 502 from the perspective of the video codec itself, may be leveraged at the network and control layers to better schedule and forward high-rate, low-latency video encoded traffic associated with immersive AR / VR / XR applications on the mobile CN and RAN.

[0084] For example, the NR RAN level typically does not exhibit any inherent knowledge of the traffic served, as shown in FIG. 4, which shows an overview of the communication system architecture. The figure includes source application servers connected to a core mobile network (possibly at the edge), which in turn is connected to a radio access network that serves subscribed and connected user equipment. Thus, the configuration, scheduling, and transmission of the RAN depends on the QoS flows and their mapping to the DRB configuration signaled by the UPF of the CN. While the target PER, PDB, or GBR values ​​of the QoS flows and their associated QoS profiles ensure proper configuration of the lower levels of the RAN, this contract between the CN and the RAN does not include any finer granularity indicators of the content of the traffic encapsulated in the QoS flows to help the RAN statistically meet its contractual agreement at the QoS profile level.

[0085] This problem is particularly challenging for high-rate, low-latency video-encoded XR traffic that has noisy periodicity due to inherent source and network jitter, as the RAN may need additional knowledge of the data it is forwarding over the air for better decisions in scheduling and transmission / retransmission to meet the target PER, PDB, and GBR figures. This knowledge represents the RAN's awareness of the video-encoded content and its relevance to the end-user's QoE. Armed with such awareness, the RAN may better determine how to prioritize traffic and dynamically adapt its configuration, rates, and FEC to reach the target PER, PDB, and GBR figures.

[0086] In a first embodiment, in order to classify, estimate, predict, etc. the importance of NAL units for the playback of a coded video elementary stream in the coded data domain, the elementary stream needs to be analyzed and interpreted according to the corresponding codec information derived or readily available through various side metadata, e.g., RTP header information, service level metadata information, etc. To achieve this in an embodiment, the corresponding processing required needs to perform a combination of the following steps: NAL unit detection, NAL unit header parsing, NAL unit type identification and filtering, NAL unit payload metadata processing and semantic extraction based on the video codec information, and extraction and selection of a feature set that proactively describes the importance of NAL units in the video coded elementary stream. The main procedures described may be performed in part or in full, either in the order described, in parallel, or in an alternative order specific to a particular embodiment.

[0087] A block diagram of an embodiment of the described process is shown in Fig. 6. Fig. 6 illustrates syntax parsing and semantic extraction of information from a video encoded elementary stream. The dashed lines represent optional processing paths according to various embodiments, as described in more detail in the detailed description below. A video encoded elementary stream 602 is segmented into NAL units by a NAL unit detector and header parser 604. The information extracted at this stage may be a combination of NAL unit type and at least one of the following: a byte length of the NAL unit, a video coding layer identifier that allocates the contained NAL unit to a particular video coding layer (e.g., base layer (id = 0), enhancement layer (id > 0), a temporal video coding layer that allocates the NAL unit to a particular video temporal coding layer (e.g., base temporal layer @ 30 fps (tid = 0), enhancement temporal layer @ 60 fps (tid = 1), secondary enhancement temporal layer @ 90 fps (tid = 2)).

[0088] Given the identified NAL unit type, further conditional processing may be performed based on the codec's semantics specifications. Thus, a codec-specific semantic filter 606 is applied, for example, in the NAL unit payload parser & semantic interpreter 608, to further parse the NAL unit payload information depending on the corresponding NAL unit type. The embodiment considers at least a subset of the following NAL unit types for further semantic filtering and processing: video / sequence / picture parameters (e.g., NAL VPS / SPS / PPS), video coded payload of video pictures / frames / slice (e.g., NAL slice / slice segment in H.26x), and / or supplemental metadata information (e.g., NAL SEI payload).

[0089] Based on the quality of the semantic filters of the available video codecs and the codec-specific knowledge representation (e.g., specification semantics depth), in some embodiments, the NAL unit payload is further processed, parsed, and interpreted outside the entropy coding domain (e.g., CABAC, CAVLC, etc.). Thus, the encoded information of the metadata bits / bytes in signed, unsigned, or custom tabular formats (e.g., Exponential-Golomb 0th / 1st order coding, etc.) is parsed (610) and its semantic content is extracted into a number of features. Such features may include non-limiting combinations (e.g., the features provided as examples represent a useful subset of, but not only combinations to be considered) of NAL unit type-specific parameters that are layer-mapped (e.g., linked to a specific video layer / temporal layer identifier) ​​or not layer-mapped (e.g., not linked to a specific video layer / temporal layer identifier in cases where layer information is unnecessary or clear from the context--e.g., single layer stream). For example, in one embodiment, the semantics of the following parameters may be interpreted and converted into features: Video parameters: ○ Video coding layer information of the current video coded sequence (if available) Recommended frames per second (if available) Additional metadata applicable to the parsing / interpretation of sequence parameters (as per the video codec specifications) Sequence parameters: * Width in pixels (resolution width) * Height pixel count (resolution height) Video coding block size (determines the number of coded rows and columns in a picture) Maximum number of reference pictures in the buffer for inter-coded modes Additional metadata applicable to the parsing / interpretation of picture parameters (as per the video codec specification) Picture parameters: Tile-based encoding mode configuration flag (determines whether to divide a sequence of pictures into rectangular tiles on which video coding is applied) Row encoding mode configuration flag (determines whether video coding units are indexed by rows within a slice) Picture segmentation information into picture partitions (determining picture partitions within tiles -- for tile-based encoding modes -- and within slices -- for both tile-based and row-based encoding modes) Additional metadata applicable to parsing / interpreting slice parameters (as per video codec specifications) Slice parameters: * Slice type (including subtypes, if any) * Size of the video encoded content of the slice ○ Slice position within a picture / frame / tile Number of slice rows The size of the video coding unit of the slice in bytes for each row of the slice Slice reference lists (one list for P slices and two for B slices containing the indices of slice references on which the motion inter prediction of the current slice is based)

[0090] Once multiple parameters are converted into features, some embodiments may numerically encode, combine, and / or normalize the latter for annotation of the video elementary stream (producing annotated video elementary stream 616) and / or for general further processing purposes.

[0091] In some embodiments, the essential information contained within the header metadata information of the NAL units and / or the NAL unit payloads may be supplemented and augmented with metadata information traces generated by the source video encoder / XR media encoding functionality. The video encoder may provide extrinsic information given its video encoding output on a frame-by-frame, picture-by-picture, slice-by-slice, etc. In one embodiment, such supplemental metadata may consist of at least one of frame-, picture-, and / or slice-level information related to the GoP configuration (which may be semi-statically configured between a minimum and maximum length, allowing the encoder to determine the GoP length on the fly, taking into account scene cuts that the encoder may detect while attempting to perform compression of P / B frames); peak signal-to-noise ratio ("PSNR") statistics of encoded video coding blocks (either aggregated per luma / chroma encoding or globally aggregated per frame / picture / slice); statistics of each encoding mode (e.g., intra / inter encoding, skip encoding, etc.) of used video coding blocks (either aggregated per row of a frame / slice or aggregated per frame / picture / slice); prediction and transform unit size statistics of video coding blocks, etc.

[0092] This additional extrinsic video metadata information may thus, in some embodiments, serve as a supplement to the intrinsic information available in the non-entropy encoded portions of the elementary video stream. A synchronized mixer, in one embodiment, may combine and compile the intrinsic and extrinsic information into a single feature set. Examples of such embodiments are outlined in FIG. 6 by the dashed processing blocks, e.g., "Extrinsic Feature Collection" 612, and "Combine & Summarize Features" 614.

[0093] In the second embodiment, the universally extracted stream features may then be used by other processing blocks and logical filters, either alone or as annotations of the video elementary stream, to classify, infer, predict, estimate, etc. the importance of individual NAL units with respect to the playback quality of the video encoded stream. Given the NAL unit syntax analysis and semantic filtering of the NAL unit headers and various payloads, the extracted features are already synchronized with the NAL unit content. Thus, the incremental processing blocks may jointly exploit such structures among the available feature sets and their associated information, taking advantage of the general hierarchical design and compression procedure of the video codec, as highlighted in FIG. 3.

[0094] For example, some embodiments may take into account the hierarchical and encapsulation relationships between abstract video encoding components of modern hybrid video codecs. Thus, a video encoded elementary stream may be formed from one or more hierarchically encoded (either spatial or temporal) video layers. And, further, each video layer is formed from multiple video encoded sequences, of which only one sequence, e.g., the current GoP, is presented at any given time.

[0095] 7 illustrates a hierarchical P GoP structure, a hierarchical B GoP structure, and various conventional frame-to-slice partitionings. The arrows indicate the dependency of the video / motion information contained in a frame (e.g., a P / B frame) on the video / motion information contained in its corresponding reference frame (e.g., an I / P / B frame). As illustrated in FIG. 7, a GoP is further formed from one or more video images, e.g., frames, the first of which, in some embodiments, is an intra-coded frame without any references, followed by multiple inter-coded frames that reference other frames, such as an I frame 702 and a P frame 704. Typical arrangements of GoPs relevant for network streaming applications are low-latency encoding / decoding hierarchical P GoP (an I-frame 702 followed by multiple P-frames 704, typically up to mod 4 = 3 P-frames 704) and low-rate hierarchical B GoP (a reference I-frame 702 followed by at least one B-pyramid formed of three B-frames 706, which is then followed by an end I-frame 702). These two typical GoP arrangements are shown in Figure 7.

[0096] Finally, each individual frame may be further segmented into one or more partitions, e.g., as slices. Slices may be geometrically organized in a typical row-oriented raster scan or rectangular tile scan that spans either a row set portion of a frame or a rectangular portion of a sub-picture and, correspondingly, the contained video coding units / blocks. Slices indicate their own type (I / P / B, which also determines the global type of their containing frame) based on the video coding units / blocks they contain, as already outlined (see Table 1). Several types of frame-to-slice partitioning are shown in FIG. 7 for illustrative reference.

[0097] In some embodiments, the information contained in the extracted feature set and the hierarchical and encapsulation structure of the modern hybrid video codec may be utilized as a graph-based directed information flow model, which thus represents the flow of information contained in the feature set according to the video encoding specification by integrating the directed hierarchical and encapsulation structures of the main components of the video codec (e.g., video layers (temporal or spatial), video sequences, pictures / frames, slices may be considered within certain embodiments) into the information flow model.

[0098] Thus, the corresponding directed flows in the model outline how the information of a higher level / parent node influences the information content of the subsequent lower level / child nodes. Furthermore, this abstract graph model provides various information-to-importance kernel mappings for the aggregation of importance of different NAL unit types (which can be mapped to levels and nodes in the proposed structure).

[0099]

number

[0100] Allows for the application of

[0101]

number

[0102] represents the subset of extracted features that represent the available information content specific to the lth level, the ith node.

[0103] Two embodiments of the graph-based directed information flow of a video coded elementary stream (GoP 802 of 1-slice IPPP frame and GoP 804 of 2-slice IPPP frame, respectively) are presented in Fig. 8. Fig. 8 shows a graph model of the directed information flow for NAL units of a hybrid modern video coded elementary stream with a hierarchical multi-level structure including a video layer level, a sequence layer level, a video frame level, a video slice level, while utilizing the interrelationship and reference of video coded data at the slice level to determine the individual importance of NAL units without decoding. Those skilled in the art may extend the illustrated connections in the figure from the slice level to the sub-slice level for even finer granularity at the expense of the complexity of the model.

[0104] The graph model of FIG. 8 outlines the information flow given both the data encapsulation relationships and the temporal relationships of video coded pictures, and based on these, presents an abstract model-driven method for classifying, inferring, predicting, estimating, etc. the importance of video coded NAL units (mappable to specific nodes in the graph) without performing video decoding. The latter is an advantage over conventional methods that determine the importance of pictures and related frames based on the video content, essentially requiring analysis and decoding of the video content. Furthermore, the proposed graph model of the directed information flow method allows policy-based functions and / or statistical learning kernels for information-to-importance mapping to be further incorporated for a full quantitative description, as described below.

[0105] In a graph-based directed information flow model structure, as shown for illustrative purposes in Fig. 8, the highest level of information (e.g., the zeroth level 806) is accumulated by features related to the entire video coded layer, which may be collected either directly by interpreting the semantics of the NAL unit headers or by interpreting the semantics of the non-VCL NAL video parameter set units, as in the previous embodiment. Subsequent levels (e.g., the first level 808) are formed by information of features related to sequences of video coded pictures, such as the content of information extracted from the NAL unit headers or extracted from non-VCL NAL sequences and picture parameter set units.

[0106] Another level (e.g., the second level 810) represents information extracted from features of individual frames in a given video encoded sequence (e.g., in a GoP), e.g., parsed from the NAL unit headers of the frames and / or their associated slices, and semantically extracted information content. In some embodiments, the last level (e.g., the third level 812) is formed from slice-specific information that may be obtained from features extracted from the slice-type NAL unit headers and from the headers of the NAL slice payloads, respectively. On the other hand, taking into account the availability of further features and subsequent information content, other embodiments may consider additional levels of information in the graph-based directed information model, such as, for example, the sub-slice / row coded / video coding block level (e.g., fourth level 814), where statistical information about the coded components in the slice payload may be utilized to derive / calculate importance functions for individual sub-slice / row coded / video coding block components of the slice for finer granularity of importance indicators of any associated NAL units (e.g., NAL units that present a directed path to the individual sub-slice / row coded / video coding block components that are additionally considered).

[0107] 8, some embodiments take full advantage of a graph-based, directed information model to hierarchically and recursively derive importance indicators for NAL units. As a result, a node n in the graph-based model that is associated with a NAL unit l,i The importance indicator of is n at any level l, 0 ≦ l < L, in the L-depth model. l,i The importance function of

[0108]

number

[0109] is calculated / inferred / predicted / estimated based on its direct subordinates at the same level and the next level down, as

[0110] Weight

[0111]

number

[0112] (Weight

[0113]

number

[0114] In general, the i-th node of level l, e.g., n l,i to the jth node of level p, say n p,j , which indicates the weight associated with a directed edge of information flow to, may, in one embodiment, represent a binary indicator of directed information flow (e.g., a "0" value for no directed edge and a "1" value for a directed edge).

[0115] Based on the recursive rule in Equation 1, the importance indicators of various NAL units are essentially determined by their features θ (l.i)Information vs. importance function kernel related to the essential value of information of each node obtained solely based on, e.g.

[0116]

number

[0117] A node with no children (i.e., node i at the lowest level L with no directed edge going out to its outer edge) is determined by * ), the value of this function kernel is particularly important since it completely determines the relative importance of the higher level connected nodes by accumulation according to equation (1).

[0118] In some embodiments, the information vs. importance function

[0119]

number

[0120] is the complete set of extracted features, e.g., θ (l,i) In some embodiments, the same function kernel may be used for all layers, e.g.,

[0121]

number

[0122] may be considered for all layers as

[0123] In some embodiments, a function at the video slice level

[0124]

number

[0125] may take into account the size distribution of the row-encoded video content, slice type, slice size, and slice reference / reference information to determine the importance of associated NAL slices at the third level 812 of the graph-based directed information model, as presented in Fig. 8. As an example, in one embodiment, the compression gain is calculated based on the rate representation of the uncompressed picture / frame / slice, e.g., R raw can be calculated based on the available sequence-level picture resolution information, chroma subsampling type, slice type, and slice reference.

[0126] In one embodiment, such an embodiment considers and quantifies video compression coding gain as an indirectly proportional measure of importance following a rate-distortion function that seeks to minimize the source rate under a bounded distortion constraint, e.g., the average mutual information between a raw source of information and its compressed representation. An example of the latter information-to-importance function, in one embodiment, at the picture / frame / slice level is:

[0127]

number

[0128] and R enc (D) is a certain visual quality, or equivalently, D ≦ D * The upper limit of distortion D such that * is the desired encoder compressed rate for video compression, given

[0129]

number

[0130] ) less important (highly redundant) frame components and less compressed temporally and spatially (lower coding gain than less important frames, i.e. higher

[0131]

number

[0132] ) essentially fine-grained quantification of video-encoded visual importance in critical (high visual information content) frame components. This constant visual quality encoding mode exploited thereby is almost always the default encoding procedure in modern hybrid video encoders and the preferred encoding strategy for high-quality advanced video media such as immersive AR / VR / XR applications.

[0133] In another embodiment, the function

[0134]

number

[0135] may simply determine the position and role of a slice within a GoP, taking into account at least one of slice type and slice reference / reference information, and then progressively determine / predict / estimate its importance with low latency. For example, according to the model described in Equation 1, such an embodiment linearly allocates the lowest importance to slices / associated frames that have no reference connection, and consequently allocates the highest importance to key slices / associated frames that root most of the information contained within the GoP (having the most cumulative references across the GoP plane). Such an embodiment results in an importance indicator that is inversely matched to a low-latency decoding order encoding of video-encoded frames / slices, e.g., the frame / slice with the highest importance indicator will be first in the decoding order, while the frame / slice with the lowest importance indicator will be last in the decoding order. As an example, for all I / P slices,

[0136]

number

[0137] In one embodiment, the GoP sequence of single-slice IPPP frames shown in FIG. 8 is labeled with importance levels I(4)-P(3)-P(1)-P(1).

[0138] In some embodiments, the importance of higher level nodes (e.g., the 0th level--the video layer level 806, the 1st level--the sequence level 808) is aggregated into a high priority constant, e.g.,

[0139]

number

[0140] and

[0141]

number

[0142] , l = {0,1},∀j≠i,k. Such a specialization of Equation 1 is intended to simplify processing and acknowledges the fact that the NAL units associated with these levels of information are in fact stream metadata components that describe encoder parameters and configurations for subsequent contained frames in a video coded sequence that belong to a video coded layer. Thus, for bitstream conformance purposes, such metadata and header related information may always be treated with great importance in some embodiments.

[0143] Meanwhile, in other embodiments, the importance of NAL units contained in a video coded layer may be increased by a positive offset to prioritize between various video coded layers with reduced complexity, such that the interactions / dependencies / relationships between layers of the graph model are grouped together at a constant offset. As an example, a base video coded layer 902 with id=0 may be assigned a constant importance offset c0=50. As a result, in an embodiment including a video stream with two video coded temporal layers, the 60 fps base layer may have an importance offset of 50, while a second temporal layer 904, streamed at, for example, 120 fps, may be assigned a lower priority offset c1=10. Thus, all importance indicators of the first temporal layer are all:

[0144]

number

[0145]

number

[0146] It will be updated as follows:

[0147] Respectively, information flows between layers in the graph model are discarded / ignored, e.g., their corresponding weights in Equation 1 for the third level 912 and above (e.g., the second level 910, the first level 908, and the zeroth level 906) are fixed to 0. These arrangements allow parallelization of importance indicator processing for multi-layer video coded streams. An illustration of such an embodiment is presented through a graph directed information model shown in FIG. 9. FIG. 9 shows a representation of a graph directed information flow model for a video coded stream with two temporal layers. A first temporal layer with a lower frames per second frequency, denoted as Temporal Layer 0, is presented together with a second temporal layer with a higher frames per second frequency (i.e., 2x oversampling), denoted as Temporal Layer 1. The figure outlines a reduced complexity embodiment in which importance offsets are used to avoid fully connected processing of the lower layers, which further parallelizes the processing. Those skilled in the art may extend and apply the concepts detailed herein with respect to the arrangement of temporal video coded layers within a video elementary stream to the arrangement of spatial video coded layers as well.

[0148] In some embodiments, the weights associated with the graph-based directed information flow model shown in Equation 1 may be optimized to floating-point representations other than binary 0 and 1 by statistical learning algorithms tuned for specific types of video encoding and video streams (e.g., conversational XR, game XR, etc.). In other embodiments, further information-to-importance kernel mappings may be used.

[0149]

number

[0150] may similarly be selected and / or optimized based on statistical learning, where expert system knowledge is not applied to determine various rule-based mappings like those described above, but instead statistical learning methods are deployed to learn optimal functional representations based on the encoded video data content.

[0151] In a subset of such embodiments, the information-to-importance kernel mapping may be restricted to linear or nonlinear function families (e.g., polynomials, nomographic functions (e.g., functions that allow scalable aggregation (sum of various function kernels) of distributed information along a graphical model)), while other embodiments may jointly optimize the function space and weight space by deep learning models and tools for a more accurate indication of the importance of a NAL unit compared to several qualitative and quantitative video encoded measures, e.g., minimum mean squared error ("MMSE"), PSNR, structural similarity index ("SSIM"), video multi-method assessment fusion ("VMAF"), that assess the importance that the video content in which the NAL unit was encoded may have to the rendering of the video playback if the NAL unit is missing from the stream.

[0152] In the third embodiment, labeling / describing / marking the importance of video NAL units therefore consists of two high-level procedural steps that may be performed with a practical trade-off between low-latency processing and high-granularity processing / output resolution. Specifically, these two steps are:

[0153] i) a video-encoded specific syntax analysis and semantic extraction of the video encoded data stream resulting in a normalized set of features associated with every NAL unit of the video encoded stream; and

[0154] ii) A feature processing information filter and a mapping to importance indicators associated with each NAL unit of the video encoded stream that takes into account either a model-based or non-model-based (e.g., statistically learned) structure among multiple hierarchically connected and contained NAL units that form the video encoded stream.

[0155] The output of such an embodiment is typically a scalar importance indicator, in integer or floating point format, for every NAL unit belonging to the video encoded stream. This knowledge of NAL level importance is particularly important in packet-switched networks such as 3GPP systems, since for traffic on the RTP / User Datagram Protocol ("UDP") / IP layer, ADUs of video-encoded streams such as H.264, H.265, etc. are actually represented by NAL units or aggregated units of multiple NALs (described by RTP hint samples contained in an RTP hint track), see, e.g., 3GPP Technical Specification TS 26.244 (v 16.1.0 - October 2020); Digital cellular telecommunications system (Phase 2+) (GSM); Universal Mobile Telecommunications System (UMTS); LTE; 5G; Transparent end-to-end packet switched streaming service (PSS); 3GPP file format (3GP) (Release 16). Thus, knowledge of NAL level importance directly provides ADU awareness over any packet-switched network, including but not limited to 3GPP CN and RAN understanding.

[0156] As a result, this awareness of the importance of ADUs of video traffic may benefit the distributed processing system in making better decisions and performing subsystem-based / block-based optimization among the components of the distributed processing system. An embodiment in this sense is the application of such video codec NAL importance to the components and subsystems of the RAN. Such an embodiment is thus aware of the importance of the video codec NALs and, therefore, the relevance of the traffic of the provided video encoded data. This awareness may be further leveraged in decision making at the RAN level, where the NAL unit importance indicators may prioritize certain radio-specific control actions and configurations, such as packet scheduling, radio power control, radio frequency and spectrum resource allocation, PHY modulation and coding scheme ("MCS") selection, HARQ pre-emption, multi-antenna / multiple transmission reception point ("TRP") transmission and schemes, etc.

[0157] The derivation of the importance of the video encoded NAL units may in some embodiments be performed at the source, e.g., in the application server 1002 as shown in FIG. 10. The importance indicators 1001 of such embodiments are then transmitted to the RAN 1006 via the CN 1004, synchronized with the corresponding NAL units as annotation metadata describing the importance of their corresponding packets encapsulating the video encoded content of the NALs. This signaling mechanism may in some embodiments be incorporated into existing control protocols and procedures for communication between the CN 1004 and the RAN 1006 over the UPF-SDAP interface. In other embodiments, the importance annotation metadata of the NAL units may be encapsulated in the header supplement information corresponding to a packet-based transport / network protocol, e.g., IP, that carries the information of the NAL units. Thus, an embodiment of the RAN may extract the annotated importance indicators 1001 of the NAL units and thereafter recognize the importance of the video encoded NAL units.

[0158] In other embodiments, the derivation of the importance of the video coded NAL units may be incorporated at the level of the CN 1004 as one of the network functions or as part of the network function, e.g., as an extension of the network function (e.g., an extension of the UPF function for network slices serving XR / immersive applications), respectively, as highlighted in Figure 10. The derived video coded NAL unit importances are similarly synchronized as annotation metadata describing the importance of packets encapsulating the video coded content of their corresponding NALs on the UPF interface towards the RAN.

[0159] On the other hand, to avoid the need for extensive synchronization and the small network overhead induced by annotating metadata, in some embodiments, the derivation of the importance of video coded NAL units may be directly incorporated at the level of the RAN 1006, as shown in Figure 10. This may be realized in one embodiment as an extension of the SDAP layer functionality specific to video coded data transmission, or in another embodiment as an intelligent on-demand filtering functionality provided before / after the SDAP layer functionality but before the PDCP layer.

[0160] The derived importance indicator 1001 and its mapping to some radio priority policy at the RAN level (e.g., VERY HIGH, HIGH, MEDIUM, LOW, VERY LOW priority, etc.) is performed in some embodiments by the Radio Resource Control ("RRC") / MAC Control Element ("MAC-CE") / Downlink Control Information ("DCI"), which may further use this knowledge to optimize the lower layers and subsystems of the RAN. Thus, this optimization will be performed adaptively according to statistical / predicted knowledge of the wireless medium conditions to meet with high guarantees the QoS flow requirements associated with video encoded streams / IP packets of high rate and low latency immersive AR / VR / XR applications with low latency.

[0161] In an embodiment, the derived importance indicator 1001 is indicated via the MAC-CE or the first DCI, and the indication indicates a derived importance for a sequence of frames (e.g., each frame of the sequence of frames may be indicated to have a different importance).

[0162] In one example, the UE determines an index of a frame belonging to a sequence of frames based on a second DCI that schedules the frame or a portion of the frame. In one example, the MAC-CE or the first DCI may indicate a length of the sequence. In another example, the length of the sequence is fixed or configured by RRC. In another example, a pattern of a video sequence, e.g., IPPPPPPP, is configured by RRC and the MAC-CE or the first DCI indicates an importance for a subset of the sequence, e.g., in the IPPPPPPP example, the MAC-CE or the first DCI indicates only two importance indications, i.e., one importance indication corresponding to the "I" frames and another importance indication corresponding to the remaining "P" frames.

[0163] 11 illustrates a user equipment device 1100 that may be used for video codec importance indication and RAN awareness configuration according to an embodiment of the present disclosure. In various embodiments, the user equipment device 1100 is used to implement one or more of the solutions described above. The user equipment device 1100 may be an embodiment of the remote unit 105 and / or UE described above. Additionally, the user equipment device 1100 may include a processor 1105, a memory 1110, an input device 1115, an output device 1120, and a transceiver 1125.

[0164] In some embodiments, the input devices 1115 and the output devices 1120 are combined into a single device, such as a touch screen. In certain embodiments, the user equipment device 1100 may not include any input devices 1115 and / or output devices 1120. In various embodiments, the user equipment device 1100 may include one or more of the processor 1105, the memory 1110, and the transceiver 1125, and may not include the input devices 1115 and / or the output devices 1120.

[0165] As shown, the transceiver 1125 includes at least one transmitter 1130 and at least one receiver 1135. In some embodiments, the transceiver 1125 communicates with one or more cells (or wireless coverage areas) supported by one or more base units 121. In various embodiments, the transceiver 1125 is operable in an unlicensed spectrum. Further, the transceiver 1125 may include multiple UE panels supporting one or more beams. In addition, the transceiver 1125 may support at least one network interface 1140 and / or application interface 1145. The application interface 1145 may support one or more APIs. The network interface 1140 may support 3GPP reference points such as Uu, N1, PC5, etc. As will be appreciated by one skilled in the art, other network interfaces 1140 may be supported.

[0166] The processor 1105, in one embodiment, may include any known controller capable of executing computer-readable instructions and / or performing logical operations. For example, the processor 1105 may be a microcontroller, a microprocessor, a central processing unit ("CPU"), a graphics processing unit ("GPU"), an auxiliary processing unit, a field programmable gate array ("FPGA"), or a similar programmable controller. In some embodiments, the processor 1105 executes instructions stored in the memory 1110 to perform the methods and routines described herein. The processor 1105 is communicatively coupled to the memory 1110, the input devices 1115, the output devices 1120, and the transceiver 1125. In a particular embodiment, the processor 1105 may include an application processor (also known as a "main processor") that manages application domains and operating system ("OS") functions, and a baseband processor (also known as a "baseband radio processor") that manages radio functions.

[0167] In various embodiments, the processor 1105 and the transceiver 1125 control the user equipment device 1100 to implement the UE behaviors described above. In one embodiment, the transceiver 1125 transmits a wireless sensing capability report to a network node of a mobile wireless communications network in response to a capability indication request configuration received from the network node. In one embodiment, the transceiver 1125 receives a configuration from the network node defining at least one sensing reference signal ("RS") pattern, a sensing task, and a sensing RS mode based on the wireless sensing capability report.

[0168] In one embodiment, the processor 1105 encodes and compresses the uncompressed video sequence into a video-encoded stream formed from a plurality of network abstraction layer ("NAL") units using a selected video codec specification. In one embodiment, the processor 1105 extracts semantic information associated with the plurality of NAL units according to syntax elements and a semantic knowledge base defined within the determined video codec specification. In one embodiment, the processor 1105 extracts the semantic information associated with the plurality of NAL units includes determining the semantic information associated with the plurality of NAL units.

[0169] In one embodiment, the processor 1105 combines the extracted semantic information associated with the multiple NAL units to form multiple feature sets that are correspondingly synchronized with the multiple NAL units that contain the extracted semantic information. In one embodiment, the processor 1105 combining the extracted semantic information associated with the multiple NAL units to form the multiple feature sets includes determining the multiple feature sets based on the extracted semantic information associated with the multiple NAL units. In one embodiment, the processor 1105 determines an information vs. importance value for each of the multiple NAL units based on the multiple feature sets without performing video decoding of the video encoded stream.

[0170] In one embodiment, the processor 1105 annotates the NAL units with importance values ​​to form ADUs of a video encoded stream for a packet-switched communication network. In one embodiment, the processor 1105 signals the importance values ​​of the ADUs and the underlying NAL units to a video encoded traffic aware transceiver.

[0171] In one embodiment, the transceiver 1125 is a video-coded traffic aware transceiver that processes video-coded traffic awareness information to determine optimized radio scheduling and control procedures for transmit and receive operations for the video-coded stream ADUs, and transmits the video-coded stream ADUs to the RAN based on the video-coded traffic aware optimization.

[0172] The memory 1110, in one embodiment, is a computer-readable storage medium. In some embodiments, the memory 1110 includes a volatile computer storage medium. For example, the memory 1110 may include RAM, including dynamic RAM ("DRAM"), synchronous dynamic RAM ("SDRAM"), and / or static RAM ("SRAM"). In some embodiments, the memory 1110 includes a non-volatile computer storage medium. For example, the memory 1110 may include a hard disk drive, a flash memory, or any other suitable non-volatile computer storage device. In some embodiments, the memory 1110 includes both volatile and non-volatile computer storage media.

[0173] In some embodiments, memory 1110 stores data related to the configuration of video codec importance and RAN awareness. For example, memory 1110 may store various parameters, panel / beam configurations, resource allocations, policies, etc., as discussed above. In particular embodiments, memory 1110 also stores program code and associated data, such as operating system or other controller algorithms running on user equipment device 1100.

[0174] The input device 1115, in one embodiment, may include any known computer input device including a touch panel, buttons, a keyboard, a stylus, a microphone, etc. In some embodiments, the input device 1115 may be integrated with the output device 1120, for example as a touch screen or similar touch-sensitive display. In some embodiments, the input device 1115 includes a touch screen such that text may be entered using a virtual keyboard displayed on the touch screen and / or by handwriting on the touch screen. In some embodiments, the input device 1115 includes two or more different devices, such as a keyboard and a touch panel.

[0175] The output device 1120, in one embodiment, is designed to output visual, auditory, and / or tactile signals. In some embodiments, the output device 1120 includes an electronically controllable display or display device capable of outputting visual data to a user. For example, the output device 1120 may include, but is not limited to, an LCD display, an LED display, an OLED display, a projector, or similar display device capable of outputting images, text, and the like to a user. As another non-limiting example, the output device 1120 may include a wearable display that is separate from but communicatively coupled to the remainder of the user equipment device 1100, such as a smart watch, smart glasses, a head-up display, and the like. Additionally, the output device 1120 may be a component of a smartphone, a personal digital assistant, a television, a table computer, a notebook (laptop) computer, a personal computer, a vehicle dashboard, and the like.

[0176] In certain embodiments, the output device(s) 1120 include one or more speakers for generating sound. For example, the output device(s) 1120 may generate an audible alert or notification (e.g., a beep or chime). In some embodiments, the output device(s) 1120 include one or more haptic devices for generating vibration, movement, or other haptic feedback. In some embodiments, all or a portion of the output device(s) 1120 may be integrated with the input device(s) 1115. For example, the input device(s) 1115 and the output device(s) 1120 may form a touchscreen or similar touch-sensitive display. In other embodiments, the output device(s) 1120 may be located near the input device(s) 1115.

[0177] The transceiver 1125 communicates with one or more network functions of a mobile communications network via one or more access networks. The transceiver 1125 operates under the control of the processor 1105 to transmit messages, data, and other signals and to receive messages, data, and other signals. For example, the processor 1105 may selectively activate the transceiver 1125 (or a portion thereof) at a particular time to transmit and receive messages.

[0178] The transceiver 1125 includes at least one transmitter 1130 and at least one receiver 1135. The one or more transmitters 1130 may be used to provide UL communication signals to the base unit 121, such as the UL transmissions described herein. Similarly, the one or more receivers 1135 may be used to receive DL communication signals from the base unit 121, as described herein. Although only one transmitter 1130 and one receiver 1135 are shown, the user equipment device 1100 may have any suitable number of transmitters 1130 and receivers 1135. Furthermore, the transmitters 1130 and receivers 1135 may be any suitable type of transmitter and receiver. In one embodiment, the transceiver 1125 includes a first transmitter / receiver pair used to communicate with a mobile communications network over a licensed radio spectrum and a second transmitter / receiver pair used to communicate with a mobile communications network over an unlicensed radio spectrum.

[0179] In certain embodiments, a first transmitter / receiver pair used to communicate with a mobile communications network over a licensed radio spectrum and a second transmitter / receiver pair used to communicate with a mobile communications network over an unlicensed radio spectrum may be combined into a single transceiver unit, e.g., a single chip that performs functions for use in both the licensed and unlicensed radio spectrum. In some embodiments, the first transmitter / receiver pair and the second transmitter / receiver pair may share one or more hardware components. For example, a particular transceiver 1125, transmitter 1130, and receiver 1135 may be implemented as physically separate components that access shared hardware and / or software resources, such as, for example, a network interface 1140.

[0180] In various embodiments, one or more transmitters 1130 and / or one or more receivers 1135 may be implemented and / or integrated in a single hardware component, such as a multi-transceiver chip, a system on a chip, an ASIC, or other type of hardware component. In certain embodiments, one or more transmitters 1130 and / or one or more receivers 1135 may be implemented and / or integrated in a multi-chip module. In some embodiments, other components, such as a network interface 1140 or other hardware components / circuits, may be integrated in a single chip with any number of transmitters 1130 and / or receivers 1135. In such embodiments, the transmitters 1130 and receivers 1135 may be logically configured as a transceiver 1125 using one or more common control signals, or as modular transmitters 1130 and receivers 1135 implemented in the same hardware chip or multi-chip module.

[0181] 12 illustrates a network device 1200 that may be used for configuring video codec importance indication and RAN awareness according to an embodiment of the present disclosure. In one embodiment, the network device 1200 may be an implementation of one of the RAN nodes, such as the base unit 121, the RAN node 210, or a gNB, described above. Additionally, the base network device 1200 may include a processor 1205, a memory 1210, an input device 1215, an output device 1220, and a transceiver 1225.

[0182] In some embodiments, the input device(s) 1215 and the output device(s) 1220 are combined into a single device, such as a touch screen. In certain embodiments, the network device 1200 may not include any input device(s) 1215 and / or output device(s) 1220. In various embodiments, the network device 1200 may include one or more of the processor 1205, the memory 1210, and the transceiver 1225, and may not include the input device(s) 1215 and / or the output device(s) 1220.

[0183] As shown, the transceiver 1225 includes at least one transmitter 1230 and at least one receiver 1235, where the transceiver 1225 communicates with one or more remote units 105. In addition, the transceiver 1225 may support at least one network interface 1240 and / or application interface 1245. The application interface 1245 may support one or more APIs. The network interface 1240 may support 3GPP reference points such as Uu, N1, N2, and N3. As will be appreciated by one skilled in the art, other network interfaces 1240 may be supported.

[0184] The processor 1205, in one embodiment, may include any known controller capable of executing computer-readable instructions and / or performing logical operations. For example, the processor 1205 may be a microcontroller, a microprocessor, a CPU, a GPU, an auxiliary processing unit, an FPGA, or a similar programmable controller. In some embodiments, the processor 1205 executes instructions stored in the memory 1210 to perform the methods and routines described herein. The processor 1205 is communicatively coupled to the memory 1210, the input device 1215, the output device 1220, and the transceiver 1225. In a particular embodiment, the processor 1205 may include an application processor (also known as a "main processor") that manages application domains and operating system ("OS") functions, and a baseband processor (also known as a "baseband radio processor") that manages radio functions.

[0185] Memory 1210, in one embodiment, is a computer-readable storage medium. In some embodiments, memory 1210 includes a volatile computer storage medium. For example, memory 1210 may include RAM, including DRAM, SDRAM, and / or SRAM. In some embodiments, memory 1210 includes a non-volatile computer storage medium. For example, memory 1210 may include a hard disk drive, a flash memory, or any other suitable non-volatile computer storage device. In some embodiments, memory 1210 includes both volatile and non-volatile computer storage media.

[0186] In some embodiments, memory 1210 stores data related to the configuration of the video codec importance indication and RAN awareness. For example, memory 1210 may store the parameters, configurations, resource allocations, policies, etc. described above. In particular embodiments, memory 1210 also stores program code and associated data, such as an operating system or other controller algorithms running on network device 1200.

[0187] The input device 1215, in one embodiment, may include any known computer input device including a touch panel, buttons, a keyboard, a stylus, a microphone, etc. In some embodiments, the input device 1215 may be integrated with the output device 1220, for example as a touch screen or similar touch-sensitive display. In some embodiments, the input device 1215 includes a touch screen such that text may be entered using a virtual keyboard displayed on the touch screen and / or by handwriting on the touch screen. In some embodiments, the input device 1215 includes two or more different devices, such as a keyboard and a touch panel.

[0188] The output device 1220, in one embodiment, is designed to output visual, auditory, and / or tactile signals. In some embodiments, the output device 1220 includes an electronically controllable display or display device capable of outputting visual data to a user. For example, the output device 1220 may include, but is not limited to, an LCD display, an LED display, an OLED display, a projector, or similar display device capable of outputting images, text, and the like to a user. As another non-limiting example, the output device 1220 may include a wearable display separate from but communicatively coupled to the rest of the network device 1200, such as a smart watch, smart glasses, a head-up display, and the like. Additionally, the output device 1220 may be a component of a smartphone, a personal digital assistant, a television, a table computer, a notebook (laptop) computer, a personal computer, a vehicle dashboard, and the like.

[0189] In certain embodiments, the output device(s) 1220 include one or more speakers for generating sound. For example, the output device(s) 1220 may generate an audible alert or notification (e.g., a beep or chime). In some embodiments, the output device(s) 1220 include one or more haptic devices for generating vibration, movement, or other haptic feedback. In some embodiments, all or a portion of the output device(s) 1220 may be integrated with the input device(s) 1215. For example, the input device(s) 1215 and the output device(s) 1220 may form a touchscreen or similar touch-sensitive display. In other embodiments, the output device(s) 1220 may be located near the input device(s) 1215.

[0190] The transceiver 1225 includes at least one transmitter 1230 and at least one receiver 1235. The one or more transmitters 1230 may be used to communicate with a UE, as described herein. Similarly, the one or more receivers 1235 may be used to communicate with a network function of a non-public network ("NPN"), a PLMN, and / or a RAN, as described herein. Although only one transmitter 1230 and one receiver 1235 are shown, the network device 1200 may have any suitable number of transmitters 1230 and receivers 1235. Furthermore, the transmitters 1230 and receivers 1235 may be any suitable type of transmitter and receiver.

[0191] In one embodiment, the processor 1205 detects video encoded NAL units of the video encoded stream according to defined syntax elements of the determined video codec specification, and extracts semantic information associated with the NAL units according to syntax elements and a semantic knowledge base defined within the determined video codec specification.

[0192] In one embodiment, the processor 1205 combines the extracted semantic information associated with the NAL units to form feature sets that are correspondingly synchronized with the NAL units encompassing the extracted semantic information. In one embodiment, the processor 1205 determines an information vs. importance value for each of the NAL units based on the feature sets without performing video decoding of the video encoded stream. In one embodiment, the processor 1205 indicates the determined information vs. importance value for each of the NAL units of the determined video codec specification to a video encoded traffic aware transceiver for scheduling video traffic based on the indicated information vs. importance value.

[0193] In one embodiment, the processor 1205 parses the plurality of detected video encoded NAL units as syntax elements of the determined video codec specification to extract semantic information. In one embodiment, the processor 1205 selects a video codec syntax and a semantic filter from a set of candidate video codec syntax and semantic filters based on the determined video codec specification, and exposes the syntax elements of the video codec specification and a semantic knowledge base.

[0194] In one embodiment, the processor 1205 performs intrinsic semantic information extraction by applying syntax and semantic filters of a selected video codec specification to at least one of the headers of the NAL units and the non-entropy coded payloads of the NAL units to extract semantic information content inherently contained within the video coded stream.In one embodiment, the essential semantic information extraction includes extracting from the video sequence metadata information corresponding to one or more of a video layer identifier and a video layer priority information, video layer frame per second playback metadata information, video layer additional metadata for parsing layer-common video coded sequence information, video sequence resolution corresponding to a combination of a video coded frame width and height of the sequence of video coded frames, video sequence chrominance subsampling information, video sequence coding metadata information corresponding to one or more of a coding block size, a number of coded rows, and a number of coded columns in the sequence of video coded frames, a maximum allowed number of reference frames for inter-frame motion prediction of the video sequence, video sequence additional metadata necessary for parsing sequence-common video coded picture information, a video coded picture encoding mode configuration, a video coded picture coding ... based on processing elements contained in at least one NAL unit selected from a group of video coded slice reference lists, including segmentation information of the video coded picture into picture partitions, additional metadata of the video coded picture for analyzing slice-common video coded slice information, a type of the video coded slice, a payload size of the video coded slice, a position of the video coded slice within one or more of a video coded picture, a tile, and a subpicture, a number of included video coded rows of the video coded slice, a payload size of each included video coded row of the video coded slice, and a video coded slice reference list, where the video coded slice reference list defines at least one reference slice for inter-slice motion prediction and video encoding for a current slice contained in the NAL unit.

[0195] In one embodiment, the processor 1205 enhances the intrinsic extracted semantic values ​​and information using an extrinsic source of semantic information synchronized with the NAL units of the video encoded stream provided by a video encoding function implementing the video codec specification that generated the NAL units. In one embodiment, the extracted intrinsic semantic information and / or extrinsic semantic information of the determined video codec is at least one of combined, summarized, and post-processed into normalized feature sets that are synchronized with the NAL units of the video encoded stream.

[0196] In one embodiment, determining the information versus importance value for each of the plurality of NAL units of the determined video codec specification includes processing a hierarchical structure and encapsulation encoding of the video codec specification across at least one hierarchy selected from the group of a video layer level, a video sequence level, a video encoded frame level, a video encoded slice level, and a video encoded slice segment level.

[0197] In one embodiment, the processor 1205 models a graph-based directed information flow of the hierarchical structure of NAL units and the processing of encapsulation encoding toward determining an information versus importance value for each of the multiple NAL units based on at least an information versus importance function kernel and a recursive accumulation of information flows across directed edges of a graph-based directed information flow model formed with the multiple NAL units as nodes.

[0198] In one embodiment, the information versus importance function kernel

[0199]

number

[0200] is a function of θ for any level l of depth in a directed information flow model processing through a graph. (l.i)is defined as the functional parameter of the subset of the extracted feature set associated with graph node i at level l as

[0201] In one embodiment, the recursive accumulation of information flows across directed edges of a directed information flow model through a graph formed from NAL unit nodes is calculated as follows: for node i at level l, the accumulated importance value is

[0202]

number

[0203] is restricted to the weighted aggregation of the intra-level directed edges and the inter-level directed edges of the level immediately below, obtained as l,i The aggregation weight between and node j ≠ i at level l is the scalar

[0204]

number

[0205] and node n l,i The aggregation weight between and node k at level l + 1 is the scalar

[0206]

number

[0207] It is.

[0208] In one embodiment, the processor 1205 performs an information versus importance function kernel

[0209]

number

[0210] Let us use the model common information vs. importance function kernel d I ( ;θ (l,i)) In one embodiment, the information versus importance function kernel is a scalar constant for at least some of the nodes and levels of the graph-based directed information flow model.

[0211] In one embodiment, the processor 1205 determines importance values ​​of the video coded slice level and / or video coded slice segment level information versus importance function kernel inversely proportional to the video compression gain relative to the uncompressed video picture resolution content under constant quality encoding, such that frames compressed above a predefined threshold have low importance values ​​and frames compressed below a predefined threshold have high importance values.

[0212] In one embodiment, the processor 1205 determines the importance value of the information related to the video coded slice level and / or the video coded slice segment level versus the importance function kernel based on at least one selected from the group including the video coded sequence of the picture, the video frame type, the video slice type, and the video slice reference list.

[0213] In one embodiment, the processor 1205 simplifies the complete connection of multiple video coding layers of at least one of spatial and temporal layer types to ignore the graph-based directed information flow model connections between the lowest level layers, and separates and offsets the multiple video coding layers by a constant scalar to determine their importance levels and the importance levels of their contained NAL units.

[0214] In one embodiment, the processor 1205 optimizes weights of directed information flows through multiple information versus importance function kernels and associated graphs using statistical learning based on a comprehensive set of training data of video encoded sequences to at least one of minimize and maximize an expected metric of visual rendering quality based on multiple feature sets associated with each of multiple NAL units of the video encoded stream.

[0215] In one embodiment, the processor 1205 signals information versus importance values ​​of the plurality of NAL units to at least one of the RAN and the UE for optimization of transmission, retransmission, and reception procedures related to the video encoded stream over at least one of the limited capacity wireless link and the varying capacity wireless link, the signaling being performed using at least one indication selected from the group of an RRC semi-static indication, an DCI indication, and an MAC-CE dynamic header information indication, the optimization of the transmission, retransmission, and reception procedures including at least one of packet scheduling, radio power control, radio frequency and resource allocation, physical layer modulation and coding scheme selection, hybrid automatic repeat request retransmission preemption, and multi-antenna and multiple transmit / receive point transmission, beamforming, and spatial multiplexing procedures.

[0216] 13 is a flow diagram of a method 1300 for video codec importance indication and RAN awareness configuration. Method 1300 may be performed by a network entity, such as a base node, a gNB, and / or a network equipment device 1200. In some embodiments, method 1300 may be performed by a processor executing program code, e.g., a microcontroller, a microprocessor, a CPU, a GPU, an auxiliary processing unit, an FPGA, etc.

[0217] In one embodiment, the method 1300 includes a step 1305 of detecting video encoded NAL units of the video encoded stream according to defined syntax elements of the determined video codec specification. In one embodiment, the method 1300 includes a step 1310 of extracting semantic information associated with the NAL units according to syntax elements and a semantic knowledge base defined within the determined video codec specification. In one embodiment, the method 1300 includes a step 1315 of combining the extracted semantic information associated with the NAL units to form feature sets that are correspondingly synchronized with the NAL units containing the extracted semantic information.

[0218] In one embodiment, the method 1300 includes determining 1320 an information vs. importance value for each of the plurality of NAL units based on the plurality of feature sets without performing video decoding of the video encoded stream. In one embodiment, the method 1300 includes indicating 1325 the determined information vs. importance value for each of the plurality of NAL units of the determined video codec specification to a video encoded traffic aware transceiver for scheduling video traffic based on the indicated information vs. importance value, and the method 1300 ends.

[0219] 14 is a flow diagram of a method 1400 for video codec importance indication and RAN awareness configuration. The method 1400 may be performed by a remote unit 105, such as a UE or user equipment device 1100. In some embodiments, the method 1400 may be performed by a processor executing program code, e.g., a microcontroller, microprocessor, CPU, GPU, auxiliary processing unit, FPGA, etc.

[0220] In one embodiment, the method 1400 includes a step 1405 of encoding and compressing the uncompressed video sequence into a video encoded stream formed from a plurality of NAL units using a selected video codec specification. In one embodiment, the method 1400 includes a step 1410 of extracting semantic information associated with the plurality of NAL units according to syntax elements and a semantic knowledge base defined within the determined video codec specification.

[0221] In one embodiment, method 1400 includes step 1415 of combining the extracted semantic information associated with the NAL units to form feature sets that are correspondingly synchronized with the NAL units containing the extracted semantic information. In one embodiment, method 1400 includes step 1420 of determining an information vs. importance value for each of the NAL units based on the feature sets without performing video decoding of the video encoded stream.

[0222] In one embodiment, the method 1400 includes annotating 1425 the NAL units with importance values ​​to form ADUs of the video coded stream for a packet switched communication network. In one embodiment, the method 1400 includes signaling 1430 the importance values ​​of the ADUs and the underlying NAL units to a video coded traffic aware transceiver. In one embodiment, the method 1400 includes processing 1435 the video coded traffic aware information to determine optimized radio scheduling and control procedures for transmission and reception operations of the ADUs of the video coded stream. In one embodiment, the method 1400 includes transmitting 1440 the ADUs of the video coded stream to the RAN based on the video coded traffic aware optimization, and the method 1400 ends.

[0223] A first apparatus for video codec importance indication and RAN aware configuration is disclosed. The first apparatus may include a network entity such as a base node, a gNB, and / or a network equipment device 1200. In some embodiments, the first apparatus includes a processor that executes program code, such as a microcontroller, a microprocessor, a CPU, a GPU, an auxiliary processing unit, an FPGA, etc.

[0224] In one embodiment, the first device includes a processor that detects a plurality of video coded network abstraction layer ("NAL") units of the video coded stream according to defined syntax elements of the determined video codec specification. In one embodiment, the processor extracts semantic information associated with the plurality of NAL units according to the defined syntax elements and a semantic knowledge base within the determined video codec specification.

[0225] In one embodiment, the processor combines the extracted semantic information associated with the NAL units to form feature sets that are correspondingly synchronized with the NAL units encompassing the extracted semantic information. In one embodiment, the processor determines an information vs. importance value for each of the NAL units based on the feature sets without performing video decoding of the video encoded stream. In one embodiment, the processor indicates the determined information vs. importance value for each of the NAL units of the determined video codec specification to a video encoded traffic aware transceiver for scheduling video traffic based on the indicated information vs. importance value.

[0226] In one embodiment, the processor parses the plurality of detected video encoded NAL units as syntax elements of the determined video codec specification to extract semantic information. In one embodiment, the processor selects a video codec syntax and a semantic filter from a set of candidate video codec syntax and semantic filters based on the determined video codec specification, and exposes the syntax elements of the video codec specification and a semantic knowledge base.

[0227] In one embodiment, the processor performs intrinsic semantic information extraction by applying syntax and semantic filters of a selected video codec specification to at least one of the headers of the NAL units and the non-entropy coded payloads of the NAL units to extract semantic information content inherently contained within the video coded stream.In one embodiment, the essential semantic information extraction includes extracting from the video sequence metadata information corresponding to one or more of a video layer identifier and a video layer priority information, video layer frame per second playback metadata information, video layer additional metadata for parsing layer-common video coded sequence information, video sequence resolution corresponding to a combination of a video coded frame width and height of the sequence of video coded frames, video sequence chrominance subsampling information, video sequence coding metadata information corresponding to one or more of a coding block size, a number of coded rows, and a number of coded columns in the sequence of video coded frames, a maximum allowed number of reference frames for inter-frame motion prediction of the video sequence, video sequence additional metadata necessary for parsing sequence-common video coded picture information, a video coded picture encoding mode configuration, a video coded picture coding ... based on processing elements contained in at least one NAL unit selected from a group of video coded slice reference lists, including segmentation information of the video coded picture into picture partitions, additional metadata of the video coded picture for analyzing slice-common video coded slice information, a type of the video coded slice, a payload size of the video coded slice, a position of the video coded slice within one or more of a video coded picture, a tile, and a subpicture, a number of included video coded rows of the video coded slice, a payload size of each included video coded row of the video coded slice, and a video coded slice reference list, where the video coded slice reference list defines at least one reference slice for inter-slice motion prediction and video encoding for a current slice contained in the NAL unit.

[0228] In one embodiment, the processor extends the intrinsic extracted semantic values ​​and information using an extrinsic source of semantic information synchronized with the NAL units of the video encoded stream provided by a video encoding function implementing the video codec specification that generated the NAL units. In one embodiment, the extracted intrinsic and / or extrinsic semantic information of the determined video codec is at least one of combined, summarized, and post-processed into normalized feature sets that are synchronized with the NAL units of the video encoded stream.

[0229] In one embodiment, determining the information versus importance value for each of the plurality of NAL units of the determined video codec specification includes processing a hierarchical structure and encapsulation encoding of the video codec specification across at least one hierarchy selected from the group of a video layer level, a video sequence level, a video encoded frame level, a video encoded slice level, and a video encoded slice segment level.

[0230] In one embodiment, the processor models a graph-based directed information flow of a hierarchical structure of NAL units and an encapsulation encoding process toward determining an information versus importance value for each of the multiple NAL units based on at least an information versus importance function kernel and a recursive accumulation of information flows across directed edges of a graph-based directed information flow model formed with the multiple NAL units as nodes.

[0231] In one embodiment, the information versus importance function kernel

[0232]

number

[0233] is a function of θ for any level l of depth in a directed information flow model processing through a graph. (l.i)is defined as the functional parameter of the subset of the extracted feature set associated with graph node i at level l as

[0234] In one embodiment, the recursive accumulation of information flows across directed edges of a directed information flow model through a graph formed from NAL unit nodes is calculated as follows: for node i at level l, the accumulated importance value is

[0235]

number

[0236] is restricted to the weighted aggregation of the intra-level directed edges and the inter-level directed edges of the level immediately below, obtained as l,i The aggregation weight between and node j ≠ i at level l is the scalar

[0237]

number

[0238] and node n l,i The aggregation weight between and node k at level l + 1 is the scalar

[0239]

number

[0240] It is.

[0241] In one embodiment, the processor comprises an information versus importance function kernel

[0242]

number

[0243] Let us use the model common information vs. importance function kernel d I ( ;θ (l,i)) In one embodiment, the information versus importance function kernel is a scalar constant for at least some of the nodes and levels of the graph-based directed information flow model.

[0244] In one embodiment, the processor determines importance values ​​of the video coded slice level and / or video coded slice segment level information versus importance function kernel inversely proportional to the video compression gain relative to the uncompressed video picture resolution content under constant quality encoding, such that frames compressed above a predefined threshold have low importance values ​​and frames compressed below a predefined threshold have high importance values.

[0245] In one embodiment, the processor determines an importance value of the information versus importance function kernel related to the video coded slice level and / or the video coded slice segment level based on at least one selected from the group including a video coded sequence of pictures, a video frame type, a video slice type, and a video slice reference list.

[0246] In one embodiment, the processor simplifies the complete connection of multiple video coding layers of at least one of spatial and temporal layer types to ignore graph-based directed information flow model connections between the lowest level layers, and separates and offsets the multiple video coding layers by a constant scalar to determine their importance levels and the importance levels of their contained NAL units.

[0247] In one embodiment, the processor optimizes weights of directed information flows through multiple information versus importance function kernels and associated graphs using statistical learning based on a comprehensive set of training data of video encoded sequences to at least one of minimize an expected metric of visual rendering quality and maximize an expected metric based on multiple feature sets associated with each of multiple NAL units of the video encoded stream.

[0248] In one embodiment, the processor signals information versus importance values ​​of a plurality of NAL units to at least one of a radio access network ("RAN") and a user equipment ("UE") for optimization of transmission, retransmission, and reception procedures related to the video encoded stream over at least one of a limited capacity wireless link and a varying capacity wireless link, the signaling being performed using at least one indication selected from the group of a radio resource control ("RRC") semi-static indication, a downlink control information ("DCI") indication, and a medium access control control element ("MAC-CE") dynamic header information indication, the optimization of the transmission, retransmission, and reception procedures including at least one of packet scheduling, radio power control, radio frequency and resource allocation, physical layer modulation and coding scheme selection, hybrid automatic repeat request retransmission preemption, and multi-antenna and multiple transmit / receive point transmission, beamforming, and spatial multiplexing procedures.

[0249] A first method for video codec importance indication and RAN aware configuration is disclosed. The first method may be performed by a network entity, such as a base node, a gNB, and / or a network equipment device 1200. In some embodiments, the first method may be performed by a processor executing program code, e.g., a microcontroller, a microprocessor, a CPU, a GPU, an auxiliary processing unit, an FPGA, etc.

[0250] In one embodiment, the first method includes detecting a plurality of video coded network abstraction layer ("NAL") units of the video coded stream according to defined syntax elements of the determined video codec specification. In one embodiment, the first method includes extracting semantic information associated with the plurality of NAL units according to defined syntax elements and a semantic knowledge base within the determined video codec specification.

[0251] In one embodiment, the first method includes combining extracted semantic information associated with a plurality of NAL units to form a plurality of feature sets that are correspondingly synchronized with the plurality of NAL units encompassing the extracted semantic information. In one embodiment, the first method includes determining an information vs. importance value for each of the plurality of NAL units based on the plurality of feature sets without performing video decoding of the video encoded stream. In one embodiment, the first method includes indicating the determined information vs. importance value for each of the plurality of NAL units of the determined video codec specification to a video encoded traffic aware transceiver for scheduling video traffic based on the indicated information vs. importance value.

[0252] In one embodiment, the first method includes parsing the plurality of detected video encoded NAL units as syntax elements of a determined video codec specification to extract semantic information. In one embodiment, the first method includes selecting a video codec syntax and a semantic filter from a set of video codec syntax and semantic filter candidates based on the determined video codec specification, and exposing the syntax elements of the video codec specification and a semantic knowledge base.

[0253] In one embodiment, the first method includes performing intrinsic semantic information extraction by applying syntax and semantic filters of a selected video codec specification to at least one of a header of a NAL unit and a non-entropy coded payload of the NAL unit to extract semantic information content inherently contained within the video coded stream.In one embodiment, the essential semantic information extraction includes extracting from the video sequence metadata information corresponding to one or more of a video layer identifier and a video layer priority information, video layer frame per second playback metadata information, video layer additional metadata for parsing layer-common video coded sequence information, video sequence resolution corresponding to a combination of a video coded frame width and height of the sequence of video coded frames, video sequence chrominance subsampling information, video sequence coding metadata information corresponding to one or more of a coding block size, a number of coded rows, and a number of coded columns in the sequence of video coded frames, a maximum allowed number of reference frames for inter-frame motion prediction of the video sequence, video sequence additional metadata necessary for parsing sequence-common video coded picture information, a video coded picture encoding mode configuration, a video coded picture coding ... based on processing elements contained in at least one NAL unit selected from a group of video coded slice reference lists, including segmentation information of the video coded picture into picture partitions, additional metadata of the video coded picture for analyzing slice-common video coded slice information, a type of the video coded slice, a payload size of the video coded slice, a position of the video coded slice within one or more of a video coded picture, a tile, and a subpicture, a number of included video coded rows of the video coded slice, a payload size of each included video coded row of the video coded slice, and a video coded slice reference list, where the video coded slice reference list defines at least one reference slice for inter-slice motion prediction and video encoding for a current slice contained in the NAL unit.

[0254] In one embodiment, the first method includes enhancing the intrinsic extracted semantic values ​​and information using an extrinsic source of semantic information synchronized with the NAL units of the video encoded stream provided by a video encoding function implementing a video codec specification that generated the NAL units. In one embodiment, the extracted intrinsic semantic information and / or extrinsic semantic information of the determined video codec are at least one of combined, summarized, and post-processed into normalized feature sets that are synchronized with the NAL units of the video encoded stream.

[0255] In one embodiment, determining the information versus importance value for each of the plurality of NAL units of the determined video codec specification includes processing a hierarchical structure and encapsulation encoding of the video codec specification across at least one hierarchy selected from the group of a video layer level, a video sequence level, a video encoded frame level, a video encoded slice level, and a video encoded slice segment level.

[0256] In one embodiment, a first method includes modeling a graph-based directed information flow of a hierarchical structure of NAL units and an encapsulation encoding process toward determining an information versus importance value for each of a plurality of NAL units based at least on an information versus importance function kernel and a recursive accumulation of information flows across directed edges of a graph-based directed information flow model formed with a plurality of NAL units as nodes.

[0257] In one embodiment, the information versus importance function kernel

[0258]

number

[0259] is a function of θ for any level l of depth in a directed information flow model processing through a graph.(l.i) is defined as the functional parameter of the subset of the extracted feature set associated with graph node i at level l as

[0260] In one embodiment, the recursive accumulation of information flows across directed edges of a directed information flow model through a graph formed from NAL unit nodes is calculated as follows: for node i at level l, the accumulated importance value is

[0261]

number

[0262] is restricted to the weighted aggregation of the intra-level directed edges and the inter-level directed edges of the level immediately below, obtained as l,i The aggregation weight between and node j ≠ i at level l is the scalar

[0263]

number

[0264] and node n l,i The aggregation weight between and node k at level l + 1 is the scalar

[0265]

number

[0266] It is.

[0267] In one embodiment, the first method comprises using an information versus importance function kernel

[0268]

number

[0269] Let us use the model common information vs. importance function kernel d I ( ; θ (l,i)In one embodiment, the information versus importance function kernel is a scalar constant for at least some of the nodes and levels of the graph-based directed information flow model.

[0270] In one embodiment, the first method includes determining importance values ​​of the video coded slice level and / or video coded slice segment level information versus importance function kernel inversely proportional to the video compression gain relative to the uncompressed video picture resolution content under constant quality encoding, such that frames compressed above a predefined threshold have low importance values ​​and frames compressed below a predefined threshold have high importance values.

[0271] In one embodiment, the first method includes determining an importance value of an importance function kernel versus information related to a video coded slice level and / or a video coded slice segment level based on at least one selected from the group including a video coded sequence of pictures, a video frame type, a video slice type, and a video slice reference list.

[0272] In one embodiment, the first method includes a step of simplifying the complete connection of multiple video coding layers of at least one of spatial and temporal layer types to ignore graph-based directed information flow model connections between the lowest level layers, and a step of decoupling and offsetting the multiple video coding layers by a constant scalar to determine their importance levels and the importance levels of their contained NAL units.

[0273] In one embodiment, a first method includes optimizing weights of directed information flows through a plurality of information versus importance function kernels and associated graphs using statistical learning based on a comprehensive set of training data of video encoded sequences to at least one of minimize an expected metric of visual rendering quality and maximize an expected metric based on a plurality of feature sets associated with each of a plurality of NAL units of the video encoded stream.

[0274] In one embodiment, a first method includes signaling information versus importance values ​​of a plurality of NAL units to at least one of a radio access network ("RAN") and a user equipment ("UE") for optimization of transmission, retransmission, and reception procedures related to a video encoded stream over at least one of limited capacity wireless links and varying capacity wireless links, the signaling being performed using at least one indication selected from the group of a radio resource control ("RRC") semi-static indication, a downlink control information ("DCI") indication, and a medium access control control element ("MAC-CE") dynamic header information indication, the optimization of the transmission, retransmission, and reception procedures including at least one of packet scheduling, radio power control, radio frequency and resource allocation, physical layer modulation and coding scheme selection, hybrid automatic repeat request retransmission preemption, and multi-antenna and multiple transmit / receive point transmission, beamforming, and spatial multiplexing procedures.

[0275] A second apparatus for video codec importance indication and RAN aware configuration is disclosed. The second apparatus may include a remote unit 105, such as a UE or user equipment device 1100. In some embodiments, the second apparatus includes a processor that executes program code, such as a microcontroller, microprocessor, CPU, GPU, auxiliary processing unit, FPGA, etc.

[0276] In one embodiment, the second device includes a processor that encodes and compresses the uncompressed video sequence into a video encoded stream formed from a plurality of network abstraction layer ("NAL") units using a selected video codec specification. In one embodiment, the processor extracts semantic information associated with the plurality of NAL units according to syntax elements and a semantic knowledge base defined within the determined video codec specification.

[0277] In one embodiment, the processor combines the extracted semantic information associated with the NAL units to form feature sets that are correspondingly synchronized with the NAL units containing the extracted semantic information. In one embodiment, the processor determines an information versus importance value for each of the NAL units based on the feature sets without performing video decoding of the video encoded stream.

[0278] In one embodiment, the processor annotates NAL units with importance values ​​to form application data units ("ADUs") of a video encoded stream for a packet-switched communication network. In one embodiment, the processor signals the importance values ​​of the ADUs and the underlying NAL units to a video encoded traffic aware transceiver.

[0279] In one embodiment, the second apparatus includes a video coded traffic aware transceiver that processes the video coded traffic awareness information to determine optimized radio scheduling and control procedures for transmission and reception operations for the ADUs of the video coded stream, and transmits the ADUs of the video coded stream to a radio access network ("RAN") based on the video coded traffic aware optimization.

[0280] A second method for video codec importance indication and RAN aware configuration is disclosed. The second method may be performed by a remote unit 105, such as a UE or user equipment device 1100. In some embodiments, the second method may be performed by a processor executing program code, e.g., a microcontroller, microprocessor, CPU, GPU, auxiliary processing unit, FPGA, etc.

[0281] In one embodiment, the second method includes encoding and compressing the uncompressed video sequence into a video encoded stream formed from a plurality of network abstraction layer ("NAL") units using a selected video codec specification. In one embodiment, the second method includes extracting semantic information associated with the plurality of NAL units according to syntax elements and a semantic knowledge base defined within the determined video codec specification.

[0282] In one embodiment, the second method includes combining the extracted semantic information associated with the plurality of NAL units to form a plurality of feature sets that are correspondingly synchronized with the plurality of NAL units containing the extracted semantic information. In one embodiment, the second method includes determining an information versus importance value for each of the plurality of NAL units based on the plurality of feature sets without performing video decoding of the video encoded stream.

[0283] In one embodiment, the second method includes annotating a plurality of NAL units with importance values ​​to form a plurality of application data units ("ADUs") of a video encoded stream for a packet-switched communication network. In one embodiment, the second method includes signaling importance values ​​of the plurality of ADUs and the plurality of underlying NAL units to a video encoded traffic aware transceiver.

[0284] In one embodiment, the second method includes processing video-encoded traffic awareness information to determine optimized radio scheduling and control procedures for transmission and reception operations for ADUs of the video-encoded stream, and transmitting the ADUs of the video-encoded stream to a radio access network ("RAN") based on the video-encoded traffic aware optimization.

[0285] The embodiments may be embodied in other specific forms. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is therefore indicated by the appended claims rather than by the foregoing description. All changes that come within the meaning and range of equivalency of the claims are to be embraced therein. [Explanation of symbols]

[0286] 100 Wireless communication system 105 Remote Unit 107 Applications 115 5G-RAN 120 3GPP Access Network 121 Cellular Base Unit 123 3GPP communication links 130 Non-3GPP Access Networks, Non-3GPP RAN 131 Access Points 133 Non-3GPP Communication Links 135 Interworking Entities 140 Mobile Core Network 141 UPF 142 Application Server 143 AMF 145 SMF 147 PCF 148 AUSF 149 UDM / UDR 150 Data Network 160 Data Network 202 Application Server 203 XR UE device 204 Local Video Content 205 Core Network 206 Remote Video Content 208 Wireless Access Network 210 Content Delivery Gateway 212 XR engine 302 Pictures 306 Intra Prediction 308 Inter Prediction 310 Multi-dimensional (2D / 3D) spatial multi-modal transformation 316 bitstream 402 Application Server 404CN 406 RAN 408 QoS Flow 410 SDAP Layer 412 PDCP 414 RLC 416 MAC Layer 418 PHY Layer 502 NAL Units 504 Header 506 NAL Payload 602 Video-encoded elementary stream 604 NAL unit detector and header parser 606 Codec-specific Semantic Filters 608 NAL unit payload parser & semantic interpreter 612 Incidental feature collection 614 Combining & Summarizing Features 616 Annotated Video Elementary Stream 702 I-Frame 704 P Frame 706 B Frame 802 GoP of 1-slice IPPP frame 804 2-slice IPPP frame GoP 806 0th Level 808 1st Level 810 Second Level 812 Third Level 814 Fourth Level 902 Base Video Coding Layer 904 Second Time Layer 906 0th Level 908 1st Level 910 Second Level 912 Third Level 1001 Importance Indicator 1002 Application Server 1004CN 1006 RAN 1100 User equipment device 1105 Processor 1110 Memory 1115 Input Devices 1120 Output Device 1125 Transceiver 1130 Transmitter 1135 Receiver 1140 Network Interface 1145 Application Interface 1200 Network Equipment 1205 Processor 1210 Memory 1215 Input Devices 1220 output device 1225 Transceiver 1230 Transmitter 1235 Receiver 1240 Network Interface 1245 Application Interface 1300 methods 1400 methods

Claims

1. Detecting a plurality of video-encoded network abstraction layer ("NAL") units of the video-encoded stream according to defined syntax elements of the determined video codec specification; extracting semantic information associated with the plurality of NAL units according to syntax elements and a semantic knowledge base defined within the determined video codec specification; combining the extracted semantic information associated with the NAL units to form feature sets that are correspondingly synchronized with the NAL units that contain the extracted semantic information; determining an information versus importance value for each of the plurality of NAL units based on the plurality of feature sets without performing video decoding of the video encoded stream; indicating the determined information vs. importance values ​​of each of the plurality of NAL units of the determined video codec specification to a video coded traffic aware transceiver for scheduling video traffic based on the indicated information vs. importance values. A network device apparatus including a processor.

2. 2. The apparatus of claim 1, wherein the processor parses the detected video encoded NAL units as syntax elements of the determined video codec specification to extract the semantic information.

3. 2. The apparatus of claim 1, wherein the processor selects a video codec syntax and a semantic filter from a set of candidate video codec syntax and semantic filters based on the determined video codec specification, and exposes the syntax elements and semantic knowledge base of the video codec specification.

4. 4. The apparatus of claim 3, wherein the processor performs intrinsic semantic information extraction by applying syntax and semantic filters of a selected video codec specification to at least one of NAL unit headers and non-entropy coded payloads of NAL units to extract semantic information content inherently contained within the video coded stream.

5. The essential semantic information extraction includes: video layer metadata information corresponding to one or more of the video layer identifier and the video layer priority information; Frame-per-second playback metadata information for the video layer, Additional metadata of the video layer for parsing layer-common video coded sequence information; a resolution of the video sequence corresponding to at least a combination of a width and a height of a video encoded frame of the sequence of video encoded frames; Chroma subsampling information of the video sequence, encoding metadata information for a video sequence corresponding to one or more of an encoding block size, a number of encoded rows, and a number of encoded columns in said sequence of video encoded frames; the maximum allowed number of reference frames for inter-frame motion estimation of a video sequence; Additional metadata of the video sequence necessary for parsing sequence-common video coded picture information; video coded picture encoding mode configuration; segmentation information of a video coded picture into picture partitions; Additional metadata of the video coded picture for parsing slice-common video coded slice information; the type of video coded slice; payload size of a video encoded slice, a position of the video encoded slice within one or more of the video encoded picture, tile, and subpicture; the number of video coded rows contained in the video coded slice; the payload size of each contained video coded row of the video coded slice; and a video coded slice reference list, the video coded reference list defining at least one reference slice for inter-slice motion prediction and video encoding for a current slice contained within the NAL unit; The apparatus of claim 4, based on processing elements contained in at least one NAL unit selected from the group consisting of:

6. 6. The apparatus of claim 5, wherein the processor extends the intrinsic extracted semantic value and information using an exogenous source of semantic information synchronized with the NAL units of the video encoded stream provided by a video encoding function implementing the video codec specification that generated the NAL units.

7. 7. The apparatus of claim 6, wherein at least one of the extracted intrinsic and extrinsic semantic information of the determined video codec is at least one of combined, aggregated, and post-processed into a plurality of normalized feature sets that are synchronized with the plurality of NAL units of the video encoded stream.

8. determining the information versus importance value for each of the plurality of NAL units of the determined video codec specification, Video layer level, Video sequence level, Video coding frame level, the video encoded slice level, and Video Encoded Slice Segment Level 2. The apparatus of claim 1, further comprising: processing a hierarchical structure of the video codec specification and encapsulation encoding across at least one layer selected from the group consisting of:

9. 9. The apparatus of claim 8, wherein the processor models a graph-based directed information flow of a hierarchical structure of NAL units and a process of encapsulation encoding towards the determination of the information versus importance value of each of the plurality of NAL units based at least on an information versus importance function kernel and a recursive accumulation of information flow across directed edges of a graph-based directed information flow model formed from the plurality of NAL units as nodes.

10. The information vs. importance function kernel [0010] For any level l of depth in the graph-based directed information flow model, θ (l.i) The apparatus of claim 9, wherein the function parameter of the subset of extracted feature sets associated with the graph node i at level l is defined as:

11. The recursive accumulation of information flows across the directed edges of a directed information flow model through the graph formed from NAL unit nodes is such that, for node i at level l, an accumulated importance value is [0025] is restricted to the weighted aggregation of the intra-level directed edges and the inter-level directed edges of the level immediately below, obtained as l,i and node j ≠ i of level l, the aggregation weight is the scalar [0030] and the node n l,i The aggregation weight between and node k at level l+1 is the scalar [0045] 11. The apparatus according to claim 10,

12. 10. The apparatus of claim 9, wherein the processor determines importance values ​​of the information versus importance function kernel for the video coded slice level and / or the video coded slice segment level inversely proportional to a video compression gain relative to uncompressed video picture resolution content under constant quality encoding, such that frames compressed above a predefined threshold have a low importance value and frames compressed below the predefined threshold have a high importance value.

13. the processor signals the information versus importance values ​​of the plurality of NAL units to at least one of a radio access network ("RAN") and a user equipment ("UE") for optimization of transmission, retransmission, and reception procedures associated with the video encoded stream over at least one of limited capacity wireless links and varying capacity wireless links; the signaling is performed using at least one indication selected from the group of a radio resource control ("RRC") semi-static indication, a downlink control information ("DCI") indication, and a medium access control control element ("MAC-CE") dynamic header information indication; 2. The apparatus of claim 1, wherein the optimization of transmission, retransmission, and reception procedures includes at least one of packet scheduling, radio power control, radio frequency and resource allocation, physical layer modulation and coding scheme selection, hybrid automatic repeat request retransmission preemption, and multi-antenna and multiple transmit / receive point transmission, beamforming, and spatial multiplexing procedures.

14. detecting a plurality of video encoded network abstraction layer ("NAL") units of the video encoded stream according to defined syntax elements of the determined video codec specification; extracting semantic information associated with the plurality of NAL units according to syntax elements and a semantic knowledge base defined within the determined video codec specification; combining the extracted semantic information associated with the NAL units to form feature sets that are correspondingly synchronized with the NAL units that contain the extracted semantic information; determining an information vs. importance value for each of the plurality of NAL units based on the plurality of feature sets without performing video decoding of the video encoded stream; indicating the determined information vs. importance value of each of the plurality of NAL units of the determined video codec specification to a video coded traffic aware transceiver for scheduling video traffic based on the indicated information vs. importance values. A method for a network device comprising:

15. 1. A processor comprising: encoding and compressing the uncompressed video sequence into a video coded stream formed from a plurality of Network Abstraction Layer ("NAL") units using a selected video codec specification; extracting semantic information associated with the plurality of NAL units according to syntax elements and a semantic knowledge base defined within the selected video codec specification; combining the extracted semantic information associated with the NAL units to form feature sets that are correspondingly synchronized with the NAL units that contain the extracted semantic information; determining an information versus importance value for each of the plurality of NAL units based on the plurality of feature sets without performing video decoding of the video encoded stream; annotating the NAL units with the information versus importance values ​​to form a plurality of application data units ("ADUs") of the video encoded stream for a packet-switched communication network; a processor for signaling the information versus importance values ​​of the plurality of ADUs and the plurality of underlying NAL units to a video coded traffic aware transceiver; 1. A video-encoded traffic-aware transceiver, comprising: processing the video encoded traffic awareness information to determine optimized radio scheduling and control procedures for transmission and reception operations of ADUs of the video encoded stream; a video coded traffic aware transceiver that transmits ADUs of the video coded stream to a radio access network ("RAN") based on a video coded traffic aware optimization; A remote network device comprising:

Images