Avatar media representation for transmission

Abstract

Some embodiments of a method can include obtaining an input file, wherein the input file includes avatar content data; determining a data type corresponding to the avatar content data; processing the avatar data based on the data type and a codec type to extract information from the input file; inferring template-based information about the avatar content data; formatting the avatar content data based on a configuration file type and the template-based information; and encoding the formatted avatar data.

Description

Cross-reference to related applications

[0001] This application claims the benefit of European patent application No. EP24305093, filed on January 15, 2024, entitled “Avatar Media Representation for Transmission,” which is hereby incorporated in its entirety by reference.

[0002] Incorporation This application is incorporated herein by reference in its entirety by the following applications: European Patent Application Serial No. EP23306745, entitled “Avatar Media Representation for Transmission”, filed on 10 October 2023 (“745 Application”); European Patent Application Serial No. EP23306756, entitled “Geometric Avatar Media Codec for Transmission”, filed on 28 October 2023 (“756 Application”); European Patent Application Serial No. EP24305092, entitled “Geometric Avatar Media Codec for Transmission”, filed on 15 January 2024; and European Patent Application Serial No. EP24305094, entitled “Avatar JSON Exchange File Format”, filed on 15 January 2024. Background Technology

[0003] Extended Reality (XR) is a technology that enables interactive experiences where real-world environments and / or video content are enhanced with virtual content that can be defined across multiple sensory modalities, including visual, auditory, and tactile modalities. During application runtime, virtual content (e.g., 3D content or audio / video files) is rendered in real-time in a manner consistent with the user's context, such as environment, perspective, and device.

[0004] Media can include encodings (codecs) of n-dimensional data, whether computer-generated or captured from the physical world, and this data can include 3D graphical objects and environments. In MPEG and other alliances, this concept of media representation is well-defined and separated from delivery or presentation mechanisms such as systems or internet protocols. Therefore, multiple groups exist to define different topical issues and advance their progress. Summary of the Invention

[0005] A first exemplary method according to some embodiments may include: acquiring an input file, wherein the input file includes avatar content data; determining a data type corresponding to the avatar content data; processing the avatar data based on the data type and codec type to extract information from the input file; inferring template-based information about the avatar content data; formatting the avatar content data based on a configuration file type and template-based information, wherein the configuration file type is a computer-generated (CG) configuration file type; and encoding the formatted avatar data.

[0006] In some embodiments of the first exemplary method, the data type is one of the following data sets: video data, image data, grid data, and game data.

[0007] For some embodiments of the first exemplary method, the codec type is one of the following codec sets: VPCC, GPCC, V-DMC, glTF, and USD.

[0008] In some embodiments of the first exemplary method, the formatted avatar data includes avatar animation data.

[0009] In some embodiments of the first exemplary method, the formatted avatar data includes avatar geometry data.

[0010] In some embodiments of the first exemplary method, the formatted avatar data includes avatar appearance data.

[0011] A first exemplary apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0012] A second exemplary method according to some embodiments may include: acquiring encoded avatar data, wherein the encoded avatar data is associated with a computer-generated (CG) profile type; decoding the encoded avatar data to generate decoded avatar data; determining a data type corresponding to the decoded avatar data; processing the decoded avatar data based on the data type and the codec type; and rendering the processed and decoded avatar data.

[0013] In some embodiments of the second exemplary method, the data type is one of the following data sets: video data, image data, grid data, and game data.

[0014] For some embodiments of the second exemplary method, the codec type is one of the following codec sets: VPCC, GPCC, V-DMC, glTF, and USD.

[0015] In some embodiments of the second exemplary method, the decoded avatar data includes avatar animation data.

[0016] In some embodiments of the second exemplary method, the decoded avatar data includes avatar geometry data.

[0017] In some embodiments of the second exemplary method, the decoded avatar data includes avatar appearance data.

[0018] A second exemplary apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0019] A third exemplary method according to some embodiments may include: acquiring an input file, wherein the input file includes avatar content data; determining a data type corresponding to the avatar content data; processing the avatar data based on the data type and codec type to extract information from the input file; inferring template-based information about the avatar content data; formatting the avatar content data based on a configuration file type and template-based information, wherein the configuration file type is a parameterized configuration file type; and encoding the formatted avatar data.

[0020] For some embodiments of the third exemplary method, the data type is one of the following data sets: video data, image data, grid data, and game data.

[0021] In some embodiments of the third exemplary method, the formatted avatar data includes the weights of the parameters.

[0022] In some embodiments of the third exemplary method, the parameters are one of a set of attributes including: animation attributes, shape attributes, texture attributes, and pose attributes.

[0023] In some embodiments of the third exemplary method, the formatted avatar data includes a parameter-based avatar model.

[0024] A third exemplary apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0025] A fourth exemplary method according to some embodiments may include: acquiring encoded avatar data, wherein the encoded avatar data is associated with a parameterized profile type; decoding the encoded avatar data to generate decoded avatar data; determining a data type corresponding to the decoded avatar data; processing the decoded avatar data based on the data type and the codec type; and rendering the processed and decoded avatar data.

[0026] For some embodiments of the fourth exemplary method, the data type is one of the following data sets: video data, image data, grid data, and game data.

[0027] In some embodiments of the fourth exemplary method, the decoded avatar data includes the weights of the parameters.

[0028] In some embodiments of the fourth exemplary method, the parameters are one of a set of attributes including: animation attributes, shape attributes, texture attributes, and pose attributes.

[0029] In some embodiments of the fourth exemplary method, the decoded avatar data includes a parameter-based avatar model.

[0030] A fourth exemplary apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0031] A fifth exemplary method according to some embodiments may include: acquiring an input file, wherein the input file includes avatar content data; determining a data type corresponding to the avatar content data; processing the avatar data based on the data type and codec type to extract information from the input file; inferring template-based information about the avatar content data; formatting the avatar content data based on a configuration file type and template-based information, wherein the configuration file type is an artificial intelligence (AI) configuration file type; and encoding the formatted avatar data.

[0032] For some embodiments of the fifth exemplary method, the data type is one of the following data sets: video data, image data, grid data, and game data.

[0033] In some embodiments of the fifth exemplary method, the formatted avatar data includes information corresponding to a neural network.

[0034] For some embodiments of the fifth exemplary method, encoding the formatted avatar data includes using a neural network.

[0035] A fifth exemplary apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0036] A sixth exemplary method according to some embodiments may include: acquiring encoded avatar data, wherein the encoded avatar data is associated with an artificial intelligence (AI) profile type; decoding the encoded avatar data to generate decoded avatar data; determining a data type corresponding to the decoded avatar data; processing the decoded avatar data based on the data type and the codec type; and rendering the processed and decoded avatar data.

[0037] For some embodiments of the sixth exemplary method, the data type is one of the following data sets: video data, image data, grid data, and game data.

[0038] In some embodiments of the sixth exemplary method, the decoded avatar data includes information corresponding to a neural network.

[0039] For some embodiments of the sixth exemplary method, decoding the encoded avatar data includes using a neural network.

[0040] A sixth exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the methods listed above.

[0041] A seventh exemplary device according to some embodiments may include at least one processor configured to perform any of the methods listed above.

[0042] An eighth exemplary device according to some embodiments may include a computer-readable storage medium storing instructions for causing one or more processors to perform any of the methods listed above.

[0043] A ninth exemplary device according to some embodiments may include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing at least one processor to perform any of the methods listed above.

[0044] Exemplary signals according to some embodiments may include encoded and formatted avatar data generated according to any of the methods listed above. Attached Figure Description

[0045] Figure 1A This is a system diagram illustrating an exemplary communication system according to some embodiments.

[0046] Figure 1B This illustrates that, according to some embodiments, it is possible to Figure 1A A system diagram of an exemplary wireless transmit / receive unit (WTRU) used within the communication system shown.

[0047] Figure 1C This is a system diagram illustrating a set of exemplary interfaces for a system according to some embodiments.

[0048] Figure 2A This is a functional block diagram of a block-based video encoder (such as an encoder for general video coding (VVC)) according to some embodiments.

[0049] Figure 2B This is a functional block diagram of a block-based video decoder (such as a decoder for VVC) according to some embodiments.

[0050] Figure 3A This is a schematic side view illustrating an exemplary waveguide display that can be used with extended reality (XR) applications according to some embodiments.

[0051] Figure 3B This is a schematic side view illustrating an exemplary alternative display type that can be used with extended reality applications according to some embodiments.

[0052] Figure 3C This is a schematic side view illustrating an exemplary alternative display type that can be used with extended reality applications according to some embodiments.

[0053] Figure 4 This is a schematic diagram illustrating an exemplary high-level avatar media codec according to some embodiments.

[0054] Figure 5 This is a process diagram illustrating an exemplary top-level avatar media framework according to some embodiments.

[0055] Figure 6 This is a process diagram illustrating an exemplary embodiment encoding and decoding process according to some embodiments.

[0056] Figure 7 This is a process diagram illustrating an exemplary embodiment of a media cataloging process according to some embodiments.

[0057] Figure 8 This is a system diagram illustrating an exemplary top-level avatar media codec stream component interface according to some embodiments.

[0058] Figure 9 This is a process diagram illustrating an exemplary computer-generated avatar media streaming process according to some embodiments.

[0059] Figure 10 This is a process diagram illustrating an exemplary parameterized avatar media stream process according to some embodiments.

[0060] Figure 11 This is a process diagram illustrating an exemplary artificial intelligence avatar media streaming process according to some embodiments.

[0061] Figure 12 This is a process diagram illustrating an exemplary encoder architecture according to some embodiments.

[0062] Figure 13 This is a process diagram illustrating an exemplary decoder architecture according to some embodiments.

[0063] Figure 14 This is a flowchart illustrating an exemplary process for encoding avatar data according to some embodiments.

[0064] Figure 15 This is a flowchart illustrating an exemplary process for decoding avatar data according to some embodiments.

[0065] The entities, connections, arrangements, etc., depicted and described in conjunction with the various figures are presented by way of example rather than limitation. Therefore, any and all statements or other indications regarding what a particular figure “depicts,” what a particular element or entity in a particular figure “is” or “has,” and any and all similar statements that might be interpreted in isolation and out of context as absolute and therefore limiting, should only be correctly interpreted as constructively beginning with a clause such as “In at least one embodiment, …”. For the sake of simplicity and clarity, this implicit leading clause will not be repeated in the detailed description. Annoying . Detailed Implementation

[0066] Figure 1A This diagram illustrates an exemplary communication system 100 that may implement one or more of the disclosed embodiments. The communication system 100 may be a multiple access system providing content such as voice, data, video, messaging, and broadcasting to multiple wireless users. The communication system 100 enables multiple wireless users to access such content through shared system resources including wireless broadband. For example, the communication system 100 may employ one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero-Tail Unique Word DFT Extended OFDM (ZT UW DTS-s OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtered OFDM, Filter Bank Multicarrier (FBMC), etc.

[0067] like Figure 1AAs shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106, public switched telephone network (PSTN) 108, Internet 110, and other networks 112. However, it will be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, and 102d may be any type of device configured to operate and / or communicate in a wireless environment. For example, WTRUs 102a, 102b, 102c, and 102d (any of which may be referred to as a “station” and / or “STA”) may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of industrial and / or automated processing chains), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, etc. Any of WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0068] The communication system 100 may also include base station 114a and / or base station 114b. Each of base stations 114a and 114b may be any type of device configured to wirelessly interface with at least one of WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks such as CN 106, the Internet 110, and / or other networks 112. For example, base stations 114a and 114b may be base transceiver stations (BTS), Node-B, eNode B, home Node-B, home eNodeB, gNB, NR NodeB, site controllers, access points (APs), wireless routers, etc. Although base stations 114a and 114b are each depicted as a single element, it will be understood that base stations 114a and 114b may include any number of interconnected base stations and / or network elements.

[0069] Base station 114a may be part of RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as base station controllers (BSCs), radio network controllers (RNCs), relay nodes, etc. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage for a specific geographic area that may be relatively fixed or may change over time. A cell may also be divided into cell sectors. For example, the cell associated with base station 114a may be divided into three sectors. Therefore, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver per sector of the cell. In embodiments, base station 114a may employ multiple-input multiple-output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0070] Base stations 114a and 114b can communicate with one or more of WTRUs 102a, 102b, 102c, and 102d via air interface 116, which can be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). Any suitable radio access technology (RAT) can be used to establish air interface 116.

[0071] More specifically, as described above, the communication system 100 can be a multiple access system and can employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, base station 114a in RAN 104 / 113 and WTRUs 102a, 102b, 102c can implement wireless technologies, such as using Wideband CDMA (WCDMA) to establish Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA) for air interface 116. WCDMA can include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA can include High-Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0072] In the embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement wireless technologies, such as using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro) to establish Evolved UMTS Terrestrial Radio Access (E-UTRA) for air interface 116.

[0073] In the embodiments, base station 114a and WTRUs 102a, 102b, 102c can implement wireless technologies, such as using New Radio (NR) to establish NR wireless access for air interface 116.

[0074] In the embodiments, base station 114a and WTRUs 102a, 102b, and 102c can implement various radio access technologies. For example, base station 114a and WTRUs 102a, 102b, and 102c can, for instance, use the dual connectivity (DC) principle to jointly implement LTE radio access and NR radio access. Therefore, the air interface used by WTRUs 102a, 102b, and 102c can be characterized by various types of radio access technologies and / or by transmissions sent to / from various types of base stations (e.g., eNBs and gNBs).

[0075] In other embodiments, base station 114a and WTRUs 102a, 102b, 102c may implement wireless technologies such as IEEE 802.11 (i.e., Wi-Fi), IEEE 802.16 (i.e., WiMAX), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rate GSM Evolution (EDGE), GSM EDGE (GERAN), etc.

[0076] Figure 1ABase station 114b can be, for example, a wireless router, a home Node-B, a home eNode-B, or an access point, and can utilize any suitable RAT to facilitate wireless connectivity in localized areas such as commercial locations, homes, vehicles, campuses, industrial facilities, air corridors (e.g., for use by drones), roads, etc. In one embodiment, base station 114b and WTRUs 102c, 102d can implement wireless technologies such as IEEE 802.11 to establish a wireless local area network (WLAN). In another embodiment, base station 114b and WTRUs 102c, 102d can implement wireless technologies such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, base station 114b and WTRUs 102c, 102d can utilize cellular-based RATs (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish picocells or femtocells. Figure 1A As shown, base station 114b can be directly connected to Internet 110. Therefore, base station 114b does not need to access Internet 110 via CN 106.

[0077] RAN 104 / 113 can communicate with CN 106, which can be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRUs 102a, 102b, 102c, and 102d. Data can have different Quality of Service (QoS) requirements, such as different throughput requirements, latency requirements, fault tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. CN 106 can provide call control, billing services, location-based services, prepaid calling, internet connectivity, video distribution, etc., and / or perform advanced security functions such as user authentication. Although... Figure 1A As not shown, but will be understood, RAN 104 / 113 and / or CN 106 can communicate directly or indirectly with other RANs using the same RAT as or a different RAT than RAN 104 / 113. For example, in addition to connecting to RAN 104 / 113, which may be utilizing NR radio technology, CN 106 can also communicate with another RAN (not shown) using GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0078] CN 106 can also serve as a gateway for WTRUs 102a, 102b, 102c, and 102d to access PSTN 108, the Internet 110, and / or other networks 112. PSTN 108 may include a circuit-switched telephone network providing Common Old-Style Telephone Service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices using common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) from the TCP / IP Internet Protocol suite. Network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs, which may use the same RAT as RAN 104 / 113 or a different RAT.

[0079] Some or all of the WTRUs 102a, 102b, 102c, and 102d in communication system 100 may include multi-mode capabilities (e.g., WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). For example, Figure 1A The WTRU 102c shown can be configured to communicate with base station 114a, which can employ cellular wireless technology, and with base station 114b, which can employ IEEE 802 wireless technology.

[0080] Figure 1B This is a system diagram illustrating an exemplary WTRU 102. (See diagram below.) Figure 1B As shown, WTRU 102 may include a processor 118, a transceiver 120, a transmitting / receiving element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a Global Positioning System (GPS) chipset 136, and / or other peripheral devices 138, etc. It will be understood that, while remaining consistent with the embodiments, WTRU 102 may include any sub-combination of the foregoing elements.

[0081] Processor 118 can be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. Processor 118 can perform signal encoding, data processing, power control, input / output processing, and / or any other functions that enable WTRU 102 to operate in a wireless environment. Processor 118 can be coupled to transceiver 120, which can be coupled to transmitting / receiving element 122. Although Figure 1B The processor 118 and transceiver 120 are depicted as separate components, but it will be understood that the processor 118 and transceiver 120 can be integrated together in an electronic package or chip.

[0082] Transmitting / receiving element 122 can be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via air interface 116. For example, in one embodiment, transmitting / receiving element 122 can be an antenna configured to transmit and / or receive RF signals. In another embodiment, transmitting / receiving element 122 can be a transmitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, transmitting / receiving element 122 can be configured to transmit and / or receive both RF signals and optical signals. It will be understood that transmitting / receiving element 122 can be configured to transmit and / or receive any combination of wireless signals.

[0083] Although the transmitting / receiving element 122 is in Figure 1B While depicted as a single element, WTRU 102 may include any number of transmit / receive elements 122. More specifically, WTRU 102 may employ MIMO technology. Thus, in one embodiment, WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals via air interface 116.

[0084] Transceiver 120 can be configured to modulate signals transmitted by transmitting / receiving element 122 and demodulate signals received by transmitting / receiving element 122. As described above, WTRU 102 can have multi-mode capability. Therefore, transceiver 120 can include multiple transceivers for enabling WTRU 102 to communicate via various RATs (e.g., such as NR and IEEE 802.11).

[0085] The processor 118 of WTRU 102 can be coupled to and receive user input data from: a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) unit or an organic light-emitting diode (OLED) display unit). The processor 118 can also output user data to the speaker / microphone 124, keypad 126, and / or display / touchpad 128. Additionally, the processor 118 can access information and store data from any suitable type of memory, such as non-removable memory 130 and / or removable memory 132. Non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. Removable memory 132 may include a subscriber identity module (SIM) card, memory stick, secure digital storage (SD) card, etc. In other embodiments, the processor 118 can access information and store data from memory not actually located on WTRU 102, such as on a server or home computer (not shown).

[0086] The processor 118 may receive power from the power supply 134 and may be configured to distribute power to other components in the WTRU 102 and / or control power to those other components. The power supply 134 may be any suitable device for powering the WTRU 102. For example, the power supply 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0087] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) about the current location of the WTRU 102. In addition to or instead of information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) via air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be understood that, while remaining consistent with the embodiments, the WTRU 102 may acquire location information using any suitable location determination method.

[0088] The processor 118 can also be connected to other peripheral devices 138, which may include one or more software and / or hardware modules that provide additional features, functions, and / or wired or wireless connectivity. For example, peripheral devices 138 may include accelerometers, electronic compasses, satellite transceivers, digital cameras (for photos and / or videos), Universal Serial Bus (USB) ports, vibration devices, television transceivers, hands-free headsets, Bluetooth® modules, FM radio units, digital music players, media players, video game player modules, internet browsers, virtual reality and / or augmented reality (VR / AR) devices, activity trackers, etc. Peripheral devices 138 may include one or more sensors, which may be one or more of the following: gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors; geolocation sensors; altimeters, light sensors, touch sensors, magnetometers, barometers, gesture sensors, biometric sensors, and / or humidity sensors.

[0089] WTRU 102 may include a full-duplex radio, wherein the transmission and reception of some or all of the signals (e.g., associated with a specific subframe of both UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit to reduce and / or substantially eliminate self-interference via hardware (e.g., a choke) or signal processing via a processor (e.g., a separate processor (not shown) or via processor 118). In embodiments, WTRU 102 may include a half-duplex radio, wherein the transmission and reception of some or all of the signals (e.g., associated with a specific subframe of both UL (e.g., for transmission) or downlink (e.g., for reception)) may be concurrent and / or simultaneous.

[0090] Despite WTRU in Figures 1A to 1B While described as a wireless terminal, it is envisioned that, in some representative embodiments, such a terminal may (e.g., temporarily or permanently) use a wired communication interface with a communication network.

[0091] In a representative embodiment, the other network 112 may be a WLAN.

[0092] Given Figures 1A to 1B The functions described herein, along with their corresponding descriptions, can be performed by one or more emulation devices (not shown). An emulation device can be one or more devices configured to emulate one or more or all of the functions described herein. For example, an emulation device can be used to test other devices and / or simulate network and / or WTRU functions.

[0093] Simulation devices can be designed to perform one or more tests on other devices in laboratory and / or carrier network environments. For example, one or more simulation devices may perform one or more functions when fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices within the communication network. One or more simulation devices may perform one or more functions when temporarily implemented / deployed as part of a wired and / or wireless communication network. Simulation devices may be directly coupled to another device for testing purposes and / or may use over-the-air wireless communication to perform tests.

[0094] One or more simulation devices may perform one or more (including all) functions when implemented / deployed without being part of a wired and / or wireless communication network. For example, simulation devices may be used to test scenarios in laboratory and / or undeployed (e.g., tested) wired and / or wireless communication networks to perform tests on one or more components. One or more simulation devices may be test rigs. Simulation devices may transmit and / or receive data using direct RF connections and / or wireless communication via RF circuitry systems (e.g., which may include one or more antennas).

[0095] Figure 1C This is a system diagram illustrating a set of exemplary interfaces for a system according to some embodiments. In some embodiments, an extended reality display device and its control electronics may be implemented. System 150 may be embodied as an apparatus including the various components described below and configured to perform one or more aspects described in this document. Examples of such apparatus include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 150 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 150 are distributed across multiple ICs and / or discrete components. In various embodiments, system 150 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 150 is configured to implement one or more aspects described in this document.

[0096] System 150 includes at least one processor 152 configured to execute instructions loaded therein to implement various aspects, such as those described in this document. Processor 152 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 150 includes at least one memory 154 (e.g., a volatile memory device and / or a non-volatile memory device). System 150 may include a storage device 158, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 158 may include internal storage, attached storage (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0097] System 150 includes an encoder / decoder module 156 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 156 may include its own processor and memory. The encoder / decoder module 156 represents a module that can be included in a device to perform encoding and / or decoding functions. It is well known that a device can include one or both encoding and decoding modules. Alternatively, the encoder / decoder module 156 may be implemented as a separate element of system 150, or it may be incorporated within processor 152 as a combination of hardware and software known to those skilled in the art.

[0098] Program code to be loaded onto processor 152 or encoder / decoder 156 to execute the various aspects described herein may be stored in storage device 158 and subsequently loaded onto memory 154 for execution by processor 152. According to various embodiments, one or more of processor 152, memory 154, storage device 158, and encoder / decoder module 156 may store one or more of various items during the execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from equations, formulas, operations, and operational logic processing.

[0099] In some embodiments, the memory within processor 152 and / or encoder / decoder module 156 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., processor 152 or encoder / decoder module 152) is used for one or more of these functions. External memory may be memory 154 and / or storage device 158, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Various Video Coding, i.e., a new standard developed by JVET (Joint Video Experts Group)).

[0100] Inputs to the components of system 150 can be provided through various input devices, as indicated in box 172. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1C Other examples not shown include composite video.

[0101] In various embodiments, the input device of block 172 has corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting the signal band to a band), (ii) down-converting the selected signal, (iii) further band-limiting to a narrower band to select, for example, a signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired data packet stream. The RF section in various embodiments includes one or more elements performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners performing various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and filtering again to the desired frequency band. Various embodiments rearrange the order of the above (and other) components, remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as insert amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0102] Additionally, USB and / or HDMI terminals may include corresponding interface processors for connecting system 150 to other electronic devices via USB and / or HDMI connections. It should be understood that aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, within a separate input processing IC or within processor 152, as needed. Similarly, aspects of USB or HDMI interface processing may be implemented, as needed, within a separate interface IC or within processor 152. The demodulated, error-corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 152 and an encoder / decoder 156 operating in conjunction with memory and storage elements, to process the data stream as needed for presentation on an output device.

[0103] Various components of system 150 can be housed within an integrated housing. Within the integrated housing, the various components can be interconnected and transmit data between them using suitable connection means 174 (e.g., internal buses known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0104] System 150 includes a communication interface 160 that enables communication with other devices via a communication channel 162. The communication interface 160 may include, but is not limited to, a transceiver configured to send and receive data via the communication channel 162. The communication interface 160 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 162 may be implemented, for example, within a wired and / or wireless medium.

[0105] In various embodiments, data is streamed to or otherwise provided to system 150 using a wireless network, such as a Wi-Fi network, for example, IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). In these embodiments, the Wi-Fi signal is received via a communication channel 162 and a communication interface 160 adapted for Wi-Fi communication. The communication channel 162 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to system 150, delivering data via an HDMI connection to input box 172. Still other embodiments use an RF connection to input box 172 to provide streaming data to system 150. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0106] System 150 can provide output signals to various output devices, including display 176, speaker 178, and other peripheral devices 180. Display 176 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 176 can be used in televisions, tablets, laptops, mobile phones, or other devices. Display 176 can also be integrated with other components (e.g., in a smartphone) or standalone (e.g., an external monitor for a laptop computer). In various examples of embodiments, other peripheral devices 180 include one or more of a standalone digital video disc (or digital versatile disc) (DVR, for both terms), an optical disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 180 based on the output of system 150 to provide functionality. For example, an optical disc player performs the function of playing the output of system 150.

[0107] In various embodiments, signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that implement device-to-device control with or without user intervention are used to transmit control signals between system 150 and display 176, speaker 178, or other peripheral devices 180. Output devices may be communicatively coupled to system 150 via dedicated connections through corresponding interfaces 164, 166, and 168. Alternatively, output devices may be connected to system 150 via communication interface 160 using communication channel 162. Display 176 and speaker 178 may be integrated into a single unit along with other components of system 150 in an electronic device, such as a television. In various embodiments, display interface 164 includes a display driver, such as a timing controller (TCon) chip.

[0108] For example, if the RF input section 172 is part of a separate set-top box, the display 176 and speaker 178 can alternatively be separate from one or more of the other components. In various embodiments where the display 176 and speaker 178 are external components, the output signal can be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0109] System 150 may include one or more sensor devices 168. Examples of sensor devices that may be used include one or more GPS sensors, gyroscope sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors can be used to determine information such as the user's position and orientation. Where system 150 is used as a control module (such as control modules 124, 132) for an extended reality display, the user's position and orientation can be used to determine how to render image data so that the user perceives the correct portion of a virtual object or scene from the correct perspective. In the case of a head-mounted display device, the position and orientation of the device itself can be used to determine the user's position and orientation for rendering virtual content. In the case of other display devices (such as telephones, tablets, computer monitors, or televisions), other inputs can be used to determine the user's position and orientation for rendering content. For example, a user can select and / or adjust the desired viewpoint and / or viewing direction by using a touchscreen, keypad or keyboard, trackball, joystick, or other inputs. Where the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation can be selected and / or adjusted based on the movement of the display device for rendering content.

[0110] The embodiments can be executed by processor 152 or by computer software implemented by hardware or a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, memory 154 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 152 can be of any type suitable for the technical environment and can encompass one or more microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0111] Block-based video coding Like HEVC, VVC is built within a block-based hybrid video coding framework. Figure 2A A block diagram of a block-based hybrid video coding system 200 is given. Variations of this encoder 200 are envisioned, but for clarity, the encoder 200 is described below without describing all anticipated variations.

[0112] Before being encoded, the video sequence may undergo pre-coding processing 204, such as applying color transformations to the input color image (e.g., a conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing remapping of the input image components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0113] The input video signal 202, including the image to be encoded, is partitioned 206 and processed block by block, for example, in units of CUs. Different CUs can have different sizes. In VTM-1.0, CUs can be up to 128x128 pixels. However, unlike HEVC, which partitions blocks solely based on quadtrees, in VTM-1.0, coding tree units (CTUs) are divided into CUs to adapt to different local features based on quadtrees / binary / tritrees. Furthermore, the concept of multiple partitioning unit types in HEVC is removed, so in VVC-1.0, there is no longer a separation between CUs, prediction units (PUs), and transform units (TUs); instead, each CU is always used as a basic unit for both prediction and transform, without further partitioning. In the multi-type tree structure, CTUs are first partitioned using a quadtree structure. Then, each quadtree leaf node can be further partitioned using binary and ternary tree structures. Different partitioning types can be used, such as quadtree partitioning, vertical binary partitioning, horizontal binary partitioning, vertical ternary partitioning, and horizontal ternary partitioning.

[0114] exist Figure 2AIn the encoder, spatial prediction 208 and / or temporal prediction 210 can be performed. Spatial prediction (or "intra-frame prediction") uses pixels from samples of already encoded neighboring blocks (called reference blocks) (located in the same video picture / slice) to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also known as "inter-frame prediction" or "motion-compensated prediction") uses reconstructed pixels from already encoded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. A temporal prediction signal for a given CU can be signaled by one or more motion vectors (MVs), which indicate the amount and direction of motion between the current CU and its temporal reference. Furthermore, if multiple reference pictures are supported, a reference picture index can be sent additionally, which is used to identify which reference picture in the reference picture repository 212 the temporal prediction signal originates from.

[0115] The mode decision block 214 in the encoder selects the optimal prediction mode, for example, based on a rate-distortion optimization method. This selection can be made after spatial and / or temporal prediction. Intra-frame / inter-frame decisions can be indicated, for example, by a prediction mode flag. The prediction block is subtracted from the current video block 216 to generate the prediction residual. The prediction residual is decorrelated using transform 218 and quantization 220. (For some blocks, the encoder can bypass both transform and quantization, in which case the residual can be directly encoded without applying the transform or quantization process.) The quantized residual coefficients are inversely quantized 222 and inversely transformed 224 to form the reconstructed residual, which is then added back to the prediction block 226 to form the reconstructed signal of the CU. Further in the in-loop filtering, such as deblocking / SAO (sample adaptive offset) filtering, 228 can be applied to the reconstructed CU to reduce coding artifacts, which is then placed in the reference picture repository 212 and used to encode future video blocks. In order to form the output video bitstream 230, the coding mode (intra-frame or inter-frame), prediction mode information, motion information, and quantized residual coefficients are sent to the entropy coding unit (108) for further compression and packing to form the bitstream.

[0116] Figure 2B A block diagram of a block-based video decoder 250 is given. In decoder 250, the bitstream is decoded by decoder elements, as described below. Video decoder 250 typically performs operations similar to... Figure 2A The encoding process described herein is the opposite of the decoding process. Encoder 200 typically also performs video decoding as part of the encoded video data.

[0117] Specifically, the decoder's input includes a video bitstream 252, which can be generated by the video encoder 200. The video bitstream 252 is first unpacked and entropy decoded at the entropy decoding unit 254 to obtain transform coefficients, motion vectors, and other encoded information. Picture partitioning information indicates how the pictures are partitioned. Therefore, the decoder can partition 256 pictures based on the decoded picture partitioning information. The encoding mode and prediction information are sent to the spatial prediction unit 258 (if intra-frame coding) or the temporal prediction unit 260 (if inter-frame coding) to form prediction blocks. Residual transform coefficients are sent to the inverse quantization unit 262 and the inverse transform unit 264 to reconstruct the residual blocks. The prediction blocks and residual blocks are then added together at 266 to generate reconstructed blocks. The reconstructed blocks can be further filtered through the in-loop filter 268 before being stored in the reference picture repository 270 for predicting future video blocks.

[0118] The decoded image 272 can further undergo post-decoding processing 274, such as inverse color transformation (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which performs the inverse of the remapping process performed in pre-encoding processing 204. Post-decoding processing can utilize metadata derived in pre-encoding processing and signaled in the bitstream. The decoded video can be sent to display device 276. Display device 276 can be a device independent of decoder 250, or decoder 250 and display device 276 can be components of the same device.

[0119] The various methods and other aspects described in this disclosure can be used to modify modules of the video encoder 200 or decoder 250. Furthermore, the systems and methods disclosed herein are not limited to VVC or HEVC and can be applied to, for example, other standards and recommendations (whether pre-existing or future developments), and any extensions to such standards and recommendations (including VVC and HEVC). Unless otherwise stated or technically excluded, the aspects described in this disclosure can be used alone or in combination.

[0120] Figure 3A This is a schematic side view illustrating an exemplary waveguide display that can be used with extended reality (XR) applications according to some embodiments. An image is projected by an image generator 302. The image generator 302 can use one or more of a variety of techniques for projecting images. For example, the image generator 302 can be a laser beam scanning (LBS) projector, a liquid crystal display (LCD), a light-emitting diode (LED) display (including organic LED (OLED) or micro LED (µLED) displays), a digital light processor (DLP), a liquid crystal on silicon (LCoS) display, or other types of image generators or light engines.

[0121] The light representing the image 312 generated by the image generator 302 is coupled into the waveguide 304 by a diffraction input coupler 306. The input coupler 306 diffracts the light representing the image 312 into one or more diffraction orders. For example, ray 308, which is part of the bottom of the image, is diffracted by the input coupler 306, and one of the diffraction orders 310 (e.g., second order) is at an angle that allows it to propagate through the waveguide 304 via total internal reflection. The image generator 302 displays the image as instructed by a control module 324, which operates to render image data, video data, point cloud data, or other displayable data.

[0122] At least a portion of the light 310, already coupled into waveguide 304 by diffraction input coupler 306, is coupled out of the waveguide by diffraction output coupler 314. At least some of the light coupled out of waveguide 304 replicates the incident angle of the light coupled into the waveguide. For example, in the illustration, output coupled rays 316a, 316b, and 316c replicate the angle of input coupled ray 308. Because the light leaving the output coupler replicates the direction of the light entering the input coupler, the waveguide essentially replicates the original image 312. The user's eye 318 can focus on the replicated image.

[0123] exist Figure 3A In the example, output coupler 314 outputs only a portion of the coupled light on each reflection, allowing a single input beam (such as beam 308) to generate multiple parallel output beams (such as beams 316a, 316b, and 316c). In this way, even if the eye is not perfectly aligned with the center of the output coupler, at least some of the light from each part of the image may reach the user's eye. For example, if eye 318 moves downwards, beam 316c may enter the eye even if beams 316a and 316b do not, so the user can still perceive the bottom of image 312 despite the positional shift. Therefore, output coupler 314 partially functions as an outgoing pupil dilator in the vertical direction. The waveguide may also include one or more additional outgoing pupil dilators (…). Figure 3A (Not shown in the image) to expand the exit pupil in the horizontal direction.

[0124] In some embodiments, waveguide 304 is at least partially transparent relative to light originating from outside the waveguide display. For example, at least some of the light 320 from a real-world object (such as object 322) passes through waveguide 304, allowing the user to see the real-world object when using the waveguide display. Multiple diffraction orders and thus multiple images will exist as the light 320 from the real-world object also passes through diffraction grating 314. To minimize the visibility of multiple images, it is desirable that the zeroth-order diffraction (not deflected by 314) has high diffraction efficiency for both light 320 and the zeroth order, while higher diffraction orders have lower energy. Therefore, in addition to expanding and output coupling the virtual image, output coupler 314 is preferably configured to allow the zeroth order of the real image to pass through. In such embodiments, the image displayed by the waveguide display may appear to be superimposed on the real world.

[0125] Figure 3B This is a schematic side view illustrating exemplary alternative display types that can be used with extended reality applications according to some embodiments. In the XR head-mounted display device 330, a control module 332 controls a display 334 (which may be an LCD) to display images. The head-mounted display includes a partially reflective surface 336 that reflects (and in some embodiments, both reflects and focuses) the image displayed on the LCD to make the image visible to the user. The partially reflective surface 336 also allows at least some external light to pass through, thereby allowing the user to see their surroundings.

[0126] Figure 3C This is a schematic side view illustrating exemplary alternative display types that can be used with extended reality applications according to some embodiments. In the XR head-mounted display device 340, a control module 342 controls a display 344 (which may be an LCD) to display an image. The image is focused by one or more lenses of a display optics 346 to make the image visible to the user. Figure 3C In this example, external light does not reach the user's eyes directly. However, in some such embodiments, an external camera 348 may be used to capture images of the external environment and display such images on a display 344 along with any virtual content that may also be displayed.

[0127] The embodiments described herein are not limited to any particular type or structure of XR display device.

[0128] This paper describes the compression and representation of avatar / user information that is interoperable between systems and applications used for sending and delivering digital content. A new avatar / user format is described that can be compressed and sent between (but not limited to) systems, devices, and applications.

[0129] Digital humans can be represented in various forms (mesh, volume, voxel, point cloud, image, video, and sound), which is crucial for the creation and normalization of digital media content. Normalized templates can be created to capture different representations of humans and allow for interoperable representations. For example, the template could be a general human model with skeletal structure, an object-specific model, a statistical shape model, or metadata representing an individual's social attributes. Any of these approaches can accurately provide a human image as an initial stage capable of statistically representing different human representations.

[0130] Synthetic 3D models can be used for digital human representation. Synthetic representations are generally easier to manipulate and customize to represent specific human anatomy, visual effects, and social parameters, and facilitate the generation of animations in immersive reality environments, given that all parameters are known in advance. Synthetic models also contribute to the generalization and stylization of appearances, which professionals can use for animation and streaming.

[0131] 3D synthetic models appear to be good representations, but they lack the standard definitions and structures that other digital human representations can use and match, such as semantic structures for user-specific details, such as virtual identity, social status, input device control, geometric properties, shape and skeletal anatomy, but not excluding the semantics of animation parameters.

[0132] Standardization of digital human content is understood as addressing social and human privacy issues, disregarding human social behavior and interactions with environmental information, and / or as public attributes in the real world that can be adopted within social technologies. Furthermore, in real-time distribution or communication use cases, the encoding and data structure of digital human content need to be defined. In proprietary systems, these data formats are known. For open systems, the format of such data needs to be specified and standardized to implement interoperability between different systems. This paper describes new media content formats for avatars / user representations.

[0133] Transformed into the media This document presents examples of avatar representations according to some embodiments. According to some embodiments, the proposed avatar representations are intended to be compatible with any technology, such as virtual reality (VR), augmented reality (AR), extended reality (XR), streaming, gaming, interactive, collaborative, or communication systems that consider the delivery of digital human content, such as 2D / 3D videos or images containing body, face, or voice; 3D technologies used for rendering, animate, or manipulating assets of human nature; or any other system that includes social and contextual information about real users or avatars, such as those in the social and communication media industry.

[0134] The following sections detail the format of the media, its associated meanings, encoding schemes, and how they are used. In this description, the format follows the JSON format and is compatible with current MPEG standardization efforts, providing a means for compressing and transmitting media content. However, the meanings and uses are general, and other formats such as OpenXR, XML, and USD can be used for encoding.

[0135] Transform into a media ecosystem Figure 4 This is a schematic diagram illustrating an exemplary high-level avatar media codec according to some embodiments. This section combines... Figure 4 Tables 1 through 10 provide an overview of the Avatar Media Framework Architecture / Top-Level Avatar Media Components.

[0136] Figure 4 This provides a high-level overview of the different components present in the Avatar Media Codec representation. The core component set 400 consists of "Metadata" 402, "Geometry" 404, "Style" 406, "Animation" 408, "Context" 410, "Physics" 412, "Speech" 414, and "Properties" 416. Each component represents a higher layer of the Avatar Media Codec. The following sections will describe and introduce them in detail. Figure 4 All components shown.

[0137] Figure 5 This is a process diagram illustrating an exemplary top-level incarnation media framework according to some embodiments. High-level components may be part of a scene description or encoding format. Figure 5 Presented from Figure 4 The middle-level component is derived from the high-level component shown.

[0138] Figure 5 The structures of different components are shown, and they are categorized according to the nature of their function. For example, components of exemplary structure 500 can be categorized as belonging to media 502, system 504, or a combined collaboration of both 506. The following sections provide... Figure 4 and Figure 5 The document provides detailed information on all components present in the document and discusses the different parts of the work, which are divided into media blocks, system blocks, and union blocks.

[0139] Evolve Media and Systems Overview This chapter introduces the concepts of media and systems in the context of media representation. It is necessary to clarify the concepts of data flow, systems, and the overlap between them. The rationale for this separation is to enable the work within media representation to be clearly understood and effectively represented.

[0140] Media can include encodings (codecs) of n-dimensional data, whether computer-generated or captured from the physical world, and this data can include 3D graphical objects and environments. In MPEG and other alliances, media representation is separate from delivery or presentation mechanisms (such as systems or internet protocols). Therefore, multiple groups exist to define and address different subject issues.

[0141] On the other hand, the system is an encoded representation of encapsulation formats, delivery protocols, multimedia presentation information, and schemes used to declare and describe multimedia content. Figure 7 The differences between these two concepts are shown, and it is demonstrated that both are necessary for the same topic. Although these two concepts have different responsibilities, there are still sometimes unclear and potentially overlapping boundaries between them, such as... Figure 5 As shown.

[0142] Figure 6 This is a flowchart illustrating an exemplary avatar encoding and decoding process according to some embodiments. For some embodiments of exemplary architecture 600, raw avatar and scene description data reside on server 602. The raw scene description data can be retrieved, formatted, and processed by the server (which may be different from the server storing the raw data). Similarly, raw avatar data can be retrieved, formatted, and processed by the server (which may be different from the server storing the raw data) to obtain avatar model data 604, avatar animation data 606, and avatar metadata 608. These three sets of avatar data can be encoded using avatar encoder 610. Furthermore, these three sets of avatar data can be sent to scene description processing block 612 for further processing. The output of the avatar encoder can undergo formatting and binary encoding 614 to generate an encoded avatar binary file. This encoded avatar binary file can be sent to a client device. Client device 616 can decode 618 the file to obtain avatar data. The client device can also obtain scene description data. Client device 616 can use these two sets of data to render an augmented reality scene environment 620 combined with the avatar.

[0143] Figure 7 This is a process diagram illustrating an exemplary embodiment of a media cataloging process according to some embodiments. Figure 7The left side of system 700 shows a codec block 702 that is agnostic to the meaning of the data. This codec block 702 cannot interpret the applicability or differences between different data channels. The codec interprets different input formats 704 equally and stores them in similar data structures. The structure of channel 708 includes elements that allow system models 706 (such as...) Figure 7 The system model shown on the right has attributes that can correctly interpret the meaning of channel 708 or packet data. Figure 7 The example shows the mapping between the "geometry" group 710 (left) marked "Flag_ID:0" and the system object "Geometry" (right) 712. In this way, the system object interprets the streamed content as geometric information and processes the content accordingly.

[0144] Figure 7 This illustrates the differences and overlaps between media and systems. For media, the goal is to compress and efficiently transmit data. For systems, the goal is to process and manage media based on the nature of its attributes.

[0145] Transformed into media description Figure 8 This is a system diagram illustrating an exemplary top-level avatar media codec stream component interface according to some embodiments. Figure 8 This demonstrates data types that can be streamed over a period of time or on a single instance. Figure 8 The block shown represents the core block that embodies the media codec format. Figure 8 A structure 800 is shown that focuses on encoding and transmitting static and time-based animation, geometry, appearance, and / or artificial intelligence data.

[0146] The encoding can be "time-based" or "keyframe-based". The encoding encapsulates at least one of the configuration files of computer graphics (CG) 802, parametric model 804, or artificial intelligence (AI) model 806.

[0147] Time-based data transmission assumes a continuous stream of timing media and does not require intra-frame interpolation. Each transmitted frame is assumed to be temporally continuous and therefore temporally aligned.

[0148] Keyframe data transmission assumes a discontinuous stream of timing media, potentially involving intra-frame or inter-frame interpolation between keyframes or reference frames. A keyframe is assumed to be data received about a single time instance, which can be future, present, or past. A reference frame is assumed to be data transmitted once or multiple times as a keyframe marked as a reference frame, and is either stored in memory or used for inter-frame interpolation.

[0149] Encoding is agnostic to the data type being compressed. Therefore, neither time-based nor keyframe-based data affects the encoded data used for transmission. Encoding can follow three main profiles, but is not limited to these. The main profiles are CG, parametric model, and AI model.

[0150] Computer Graphics (CG) Profile CG profiles follow "traditional" computer graphics concepts and include animation, geometry, and appearance information.

[0151] Animation data transmission processing handles dynamic information that needs to be updated or continuously streamed. Animation data may include (but is not limited to): joint animation 808, blend shape weights 810, controller weights 812, vertex displacements 820, vertices 822, landmarks 832, motion dictionary 814, textures / materials 816, UV displacements 818, shape weights, and other relevant information.

[0152] Geometry data transfer processes static information that is only needed for handshaking and model update transfers. Geometry data may include (but is not limited to): vertices 822, faces 830, bones 824, landmarks 832, UV mapping 826, skinning 834, textures / materials 816, blend shapes 828, vertex displacements 820, and other relevant static data.

[0153] Appearance data transmission processing requires continuous or keyframe-based streaming of texture and appearance information. Appearance data may include (but is not limited to): video, images, UV displacement 818, texture 816, and other relevant appearance data.

[0154] Parameterized model configuration file Parametric model profiles are not AI-based mechanisms, but rather data-driven probabilistic models parameterized with known features. For example, a model might have animation weights 836, which directly modify and drive the model's animation system, such as making the avatar walk or acquire objects in the scene. Shape weights 838 can alter the avatar's skeleton, body, or clothing into different visual forms while preserving the model's essence (topology). Texture weights 840 can change or animate the visual appearance without interfering with the geometry. Pose weights 842 can change the avatar's pose using predefined rig parameters on the shape and skeletal structure.

[0155] Artificial Intelligence (AI) Model Configuration File AI model profiles follow a more recent and sophisticated approach, relying on deep neural networks to encode, decode, and represent avatar information.

[0156] The deep network 844 can be used to encode and reference its architecture 848, trained architecture weights (which, in some embodiments, may include network weights 846), latent features 856, and their semantic definitions. Latent features may include the following categories (previously stated in the CG profile): animation 850, geometry 852, and texture 854, or may have independent definitions defined by the model itself. Since neural networks can process multiple categories simultaneously, a single category definition may not correctly translate into the model's functionality. Therefore, the model itself can define its own definitions and provide a mapping mechanism so that the system can correctly transform and apply the data.

[0157] Transform into a media flow architecture The architecture of the media stream identifies the various elements and interfaces between different blocks. The following sections describe the architecture of each configuration file.

[0158] CG as a media streaming architecture Figure 9 This is a process diagram illustrating an exemplary computer-generated avatar media streaming process according to some embodiments. Figure 9 The architecture 900, which embodies media streaming, is shown, with a focus on computer-generated (CG) content. Usability and performance evaluations can be performed on existing standards and platforms used for encoding and transmitting geometric information, such as VPCC, GPCC, VDMC, glTF, and USD.

[0159] Figure 9 It includes components for encoding and sending avatar information, which are interoperable across systems and platforms. Figure 9 Input block 902 illustrates examples of different inputs that can be used simultaneously or as separate content inputs. Processing block 904 uses algorithms and / or methods to extract information from the input source for a specific codec format (e.g., VPCC, GPCC, V-DMC, glTF, and USD). Prior block 906 includes a model for inferring template-based information about the content of the input source. Data format block 908 defines the elements and data of the avatar media codec. Encoding block 910 and decoding block 912 each use compression / decompression and packing / unpacking processes to allow efficient and lightweight bitstream transmission of the avatar media codec. Output block 914 uses algorithms and / or methods to convert the data into a format used by the application. Application block 916 uses the avatar data, such as through 2D / 3D rendering or other means, to display digital content and / or interact with digital content.

[0160] Data format Data formats include Figure 4 and Figure 5All core elements are shown. In the context of a CG profile, data may include (but is not limited to) the following geometry, animation, appearance, and style attributes.

[0161] Vertex 918 can be represented as a 2D / 3D spatial descriptor, which represents the 2D or 3D geometric composition of the avatar.

[0162] Face 920 can be represented as a triangle or quadrilateral descriptor, which represents the vertex connectivity (topology) of the avatar.

[0163] The skeleton 922 can be represented as a 2D / 3D spatial descriptor, which represents the 2D or 3D skeletal joints of the avatar. Skeletal connectivity, in the form of hierarchical tuples composed of joint indices, can be used to create skeletal representations of skeletal joints.

[0164] Skinning 928 can be represented as a weighted matrix of relationships between joint and vertex indices. For each vertex, a joint / bone is associated with a scalar weight between 0 and 1, where 0 indicates no effect on the vertex and 1 indicates maximum effect. For each vertex, the sum of all joint / bone scalar weights should equal one.

[0165] Landmark 924 can be represented as a 2D / 3D spatial descriptor, which represents a 2D or 3D landmark on the surface of the avatar. Landmarks can include weighted matrix representations to link geometric surfaces (mesh) with animation techniques.

[0166] UV mapping 926 can be represented as a 2D spatial descriptor that maps a 3D geometric representation onto a 2D domain.

[0167] Blend shapes 930 can be represented as 2D / 3D spatial descriptors, representing the 2D or 3D displacement vectors of the original geometry. A collection of blend shapes can represent the animation of avatar geometry. For example, facial expressions are constructed using a collection of blend shapes and can be used by applications to infer personalized animations. This inference can be accomplished by linearly or non-linearly combining multiple blend shapes. Blend shapes can include high-level controllers, which can be predefined to act together with one or more blend shapes in a linear or non-linear manner. Therefore, applications can use blend shapes to achieve accurate and realistic animations.

[0168] Joint animation 932 can be represented as a hierarchical matrix representation of a joint transformation matrix, a quaternion, or Euler angles, which includes translation, rotation, and scaling.

[0169] The blend shape weight 936 can be represented as a scalar value, ranging between a minimum and a maximum value defined linearly by the blend shape deformation. This weight can represent the state transition from a "neutral" pose to a "target" pose. The "neutral" pose is the bound pose for calculating the blend shape, and the "target" pose is the blend shape deformation.

[0170] The controller weights 938 can be represented as 1D, 2D, or 3D scalar values, which linearly or non-linearly combine multiple deformation vectors or blended shapes to animate specific geometric locations.

[0171] The motion dictionary 934 can be represented as a semantic dictionary or tree that maps human-readable descriptions (such as walking, jumping, or idling) to animation parameters (such as joint animation, blend shape weights, or controller weights).

[0172] Textures and materials 940 can be represented in 2D image file formats (such as png, jpeg, etc.), which represent the visual appearance of the avatar.

[0173] UV displacement 942 can be represented as a 2D spatial descriptor that maps a 3D geometric representation to a 2D domain. This mapping provides mechanisms for deforming, animatening, and / or stylizing the geometric and / or appearance representation of an avatar.

[0174] Vertex displacement 944 can be represented as a 3D space descriptor, which provides a mechanism for deforming, animate, or stylize the geometric representation of the avatar.

[0175] Parameterized media streaming architecture Figure 10 This is a process diagram illustrating an exemplary parameterized avatar media stream process according to some embodiments. Figure 10 The architecture 1000, which transforms into a media stream, is shown, with a focus on parametric model content. Usability and performance evaluations can be performed on existing standards and platforms used for encoding and transmitting geometric information, such as VPCC, GPCC, VDMC, glTF, and USD.

[0176] Figure 10 It includes components for encoding and sending avatar information, which are interoperable across systems and platforms. Figure 10Input block 1002 illustrates examples of different inputs that can be used simultaneously or as separate content inputs. Processing block 1004 uses algorithms and / or methods to extract information from the input source for a specific codec format (e.g., VPCC, GPCC, V-DMC, glTF, and USD). Prior block 1006 includes a model for inferring template-based information about the content of the input source. Data format block 1008 defines the elements and data of the avatar media codec. Encoding block 1010 and decoding block 1012 use compression / decompression and packing / unpacking processes to allow efficient and lightweight bitstream transmission of the avatar media codec. Output block 1014 uses algorithms and / or methods to convert the data into a format used by the application. Application block 1016 uses the avatar data, such as through 2D / 3D rendering or other means, to display digital content and / or interact with digital content.

[0177] Data format Data formats include Figure 4 and Figure 5 All core elements are shown. In the context of a parameterized configuration file, the data may include (but is not limited to) the following animation properties, shape properties, texture properties, and pose properties.

[0178] Animation weights 1018 can be represented as 1D, 2D, or nD (for any integer greater than 0) descriptors, representing the parameters that animate the geometry and / or skeletal representation of the avatar. Such descriptors are model-based and not transferable across different models. These descriptors can take various forms, such as those with associated semantic meanings or more numerical forms.

[0179] The shape weight 1020 can be represented as a 1D, 2D, or nD (for any integer greater than 0) descriptor, which represents the parameters that deform the geometric representation of the avatar. Such descriptors are model-based and not transferable across different models. These descriptors can have various forms, such as those with associated semantic meanings or more numerical forms.

[0180] Texture weights 1022 can be represented as 1D, 2D, or nD (for any integer greater than 0) descriptors, which represent parameters that deform the texture and / or appearance representation of the avatar. Such descriptors are model-based and not transferable across different models. These descriptors can have various forms, such as those with associated semantic meanings or more numerical forms.

[0181] The pose weights 1024 can be represented as 1D, 2D, or nD (for any integer greater than 0) descriptors, which represent the parameters that deform the geometry and / or skeletal pose representation of the avatar. Such descriptors are model-based and not transferable across different models. These descriptors can take various forms, such as those with associated semantic meanings or more numerical forms.

[0182] Artificial intelligence (AI) becomes the architecture of media streams Figure 11 This is a process diagram illustrating an exemplary artificial intelligence avatar media streaming process according to some embodiments. Figure 11 The architecture 1100, which transforms into a media stream, is shown, with a focus on artificial intelligence (AI) content. Usability and performance evaluations can be performed on existing standards and platforms used for encoding and transmitting geometric information, such as VPCC, GPCC, VDMC, glTF, and USD.

[0183] Figure 11 It includes components for encoding and sending avatar information, which are interoperable across systems and platforms. Figure 11 Input block 1102 illustrates examples of different inputs that can be used simultaneously or as separate content inputs. Processing block 1104 uses algorithms and / or methods to extract information from the input source for a specific codec format (e.g., VPCC, GPCC, V-DMC, glTF, and USD). Prior block 1106 includes a model for inferring template-based information about the content of the input source. Data format block 1108 defines the elements and data of the avatar media codec. Encoding block 1110 and decoding block 1112 use compression / decompression and packing / unpacking processes to allow efficient and lightweight bitstream transmission of the avatar media codec. Output block 1114 uses algorithms and / or methods to convert the data into a format used by the application. Application block 1116 uses the avatar data, such as through 2D / 3D rendering or other means, to display digital content and / or interact with digital content.

[0184] Data format Data formats include Figure 4 and Figure 5 All core elements are shown. In the context of an AI profile, data may include (but is not limited to) the following animation properties, shape properties, texture properties, and pose properties.

[0185] Deep networks can be represented as pre-trained neural networks that use input features (video, images, or shapes, etc.) to represent a given task for streaming, representing, and / or manipulating. Deep networks can include mechanisms for encoding and decoding using a combination of linear or convolutional layers. At the receiving end, a deep network can use the latent feature representation of the encoded data along with a decoder. For a given latent feature, the decoder can generate the output provided to the network in training mode.

[0186] Architecture 1118 can be represented in binary or human-readable (JSON) format, which includes the format and representation of all layers in the network. The architecture may include encoders, decoders, and / or generators. The architecture includes details of all properties associated with such layers, such as input and output parameters, linear and convolutional layers, activation mechanism type (if any), normalization mechanism type, skip connection details, arithmetic operations between layers (multiplication, addition, subtraction, and / or division of layers, weights, parameters, and / or outputs), and initialization parameters.

[0187] The network weights 1120 can be represented in binary or human-readable (JSON) format, and include all the weights and bias parameters of the network architecture. These weights initialize the network architecture to a state that can be instantiated (e.g., the network state when training is satisfied / completed).

[0188] Animation, geometry, and texture parameters can take the form presented in the CG profile above and can be available at the frame's receiving end or the application's end. The information sent can be related to one or more attributes of the data presented in the CG profile above. To help the decoder or application read the stream, tags can accompany latent features to indicate the data type the decoder or application expects to process (e.g., "Animation," "JointAnimation," or other latent features). Such tags can announce to the application or decoder that the latent feature is joint animation. This joint animation can be joint rotation, translation, and / or scaling.

[0189] Latent feature 1122 can be represented as a 1D, 2D, or nD (for any integer greater than 0) descriptor, which represents parameters that deform one or more parts of the avatar's animation, geometry, and / or texture / appearance representation. Such descriptors are AI-based and cannot be transferred across different networks without retraining. These descriptors can have various forms, such as those with associated semantic meanings or more numerical forms.

[0190] Exchange file format The exchange format can be JSON format or other textual implementations of the data model. This format is human-readable and can be manipulated by the user. In some embodiments, this format is the Avatar JSON Exchange Format (AJIF). AJIF information about the digital human (avatar) can be encoded and represented in a stream between systems. Table 1 shows higher-level information that can be used for streaming and representing avatar media.

[0191]

[0192] Table 1. Transform into an encoder API The following section illustrates an exemplary application programming interface (API) for encoding avatar media content.

[0193] Geometric shapes The geometry of an avatar represents an attribute that defines the avatar asset and its semantic meaning. This can take the form of connected or disconnected vertices, skeletal structures, 3D markers, images, or contextual information. The function "processGeometry()" handles the acquisition of (related) information.

[0194] In some embodiments, semantics may include markers for vertices, faces, regions (e.g., geometry), landmarks and skeleton markers, and texture markers for purposes such as classification, avatar-object interaction, avatar-avatar interaction, avatar redirection (e.g., animation, style, geometry, and so on). Figure 4 Applications such as (and other elements shown in Tables 2 to 10).

[0195] The purpose of this signal is to inform the decoder that geometric information is available for reading. This signal can take many forms, such as traditional coding standards for geometric information, including MPEG-I SD, MPEG-AI, MPEG V-DMC, L-PCC, MPEG 4, H-anim, JVET, etc.

[0196]

[0197] Table 2: Geometric Shape Semantics style The shape representation of the avatar media defines the attributes of the avatar style, such as clothing, hairstyle, accessories, visual appearance, personality, and behavior. The function "processStyle()" handles the retrieval of (related) information.

[0198] In some embodiments, style can include the movement style that the avatar can perform, for example, the avatar has a "wandering" style or a "dunking" style. In terms of animation style, style can be the clothing or accessories the avatar can possess, the hairstyle the avatar has, or the shape style of the avatar. Even an individual's personality can be considered style in society.

[0199] The purpose of this signal is to inform the decoder that style information is available for reading. This signal can take many forms, such as traditional coding standards used for geometric and supplementary information, including MPEG-I SD, MPEG-AI, MPEG V-DMC, L-PCC, MPEG 4, H-anim, JVET, etc.

[0200]

[0201] Table 3: Style Semantics Animation Animation in avatar media defines the properties of the avatar's motion, such as whether it is presented as 2D video, immersive video, multi-view video, or in the form of a skeleton or geometric shape. The function "processAnimation()" handles the acquisition of (related) information.

[0202] In some embodiments, the animated avatar media element may be associated with animation parameters; for example, a style may be associated with motion and have specific motion vectors that can be used for animations of certain styles. In some embodiments, the animation may be associated with skeletal animation or geometric displacement (whether rigid or non-rigid). In some embodiments, the animation may be associated with clothing movement of the avatar if such movement exists.

[0203] The purpose of this signal is to inform the decoder that animation information is available for reading. This signal can take many forms, such as traditional coding standards used for geometric and supplementary information, such as MPEG-I SD, MPEG-AI, MPEG V-DMC, L-PCC, MPEG 4, H-anim, JVET, etc.

[0204]

[0205] Table 4: Animation Semantics scene The context of the avatar defines the attributes of the scene in which the avatar exists, such as a video stream within a VR or AR system or a description of the world environment in a scene description format. The function "processContext()" handles the retrieval of (related) information.

[0206] In some embodiments, the avatar's scene includes the avatar's current surrounding environment. For example, the scene can be the background of a 2D video stream that is to be retained or removed based on existing signals during streaming along with the avatar data.

[0207] The purpose of this signal is to inform the decoder that contextual information is available for reading. This signal can take many forms, such as traditional coding standards used for geometric and supplementary information, such as MPEG-I SD, MPEG-AI, MPEG V-DMC, L-PCC, MPEG 4, H-anim, JVET, etc.

[0208]

[0209] Table 5: Contextual Semantics physical properties The physical properties of the avatar medium represent the attributes of avatar movement and scene interaction defined by a physical model based on the avatar and its environment. These attributes can describe gravity, wind or other natural forces, as well as lighting or material properties. The function "processPhysics()" handles the acquisition of (related) information.

[0210] The purpose of this signal is to inform the decoder that physical characteristic information is available for reading. This signal can take many forms, such as traditional coding standards used for geometric and supplementary information, such as MPEG-I SD, MPEG-AI, MPEG V-DMC, L-PCC, MPEG 4, H-anim, JVET, etc.

[0211]

[0212] Table 6: Semantics of Physical Characteristics Touch The haptic representation of an avatar's media defines the attributes that provide haptic feedback features available to the avatar. These features can take the form of, for example, object material properties, haptic sensors on the avatar, potential feedback responses from other objects and systems in the environment, haptic signals attached to specific body parts, or sensitivity maps. The function "processHaptics()" handles the acquisition of (related) information.

[0213] In some embodiments, tactile sensation may include signals that embody the tactile sensations that might be present if hardware is available. Examples include wearing gloves to obtain tactile sensations or wearing a tactile vest to simulate impact.

[0214] The purpose of this signal is to inform the decoder that tactile information is available for reading. This signal can take many forms, such as traditional coding standards used for geometric information and supplementary tactile information, such as MPEG-I SD, MPEG-AI, MPEG V-DMC, L-PCC, MPEG 4, H-anim, JVET, HMPG, HJIF, etc.

[0215]

[0216] Table 7: Tactile Semantics audio The audio representation of the avatar media defines the properties of that avatar audio, such as speech and noise generated through contact or spatial audio properties. The function "processAudio()" handles the acquisition of (related) information.

[0217] In some embodiments, audio attributes may include audio associated with spatial audio or transmittable speech output.

[0218] The purpose of this signal is to inform the decoder that audio information is available for reading. This signal can take many forms, such as traditional audio coding standards like MPEG, AVC, HEVC, and VVC.

[0219]

[0220] Table 8: Audio Semantics Metadata Metadata representing media allows for the decoding of signals and data from other entities (such as other standards (MPEG-I SD, MPEG-AI, MPEG V-DMC, L-PCC, MPEG 4, H-anim, JVET)). The function "processMetada()" handles the acquisition of (related) information.

[0221]

[0222] Table 9: Metadata Semantics property The properties of the avatar media represent additional information that can further define the transmitted avatar, such as social or personal information, or additional information related to the application receiving data content that may not be included in the specification (such as proprietary physical properties, proprietary avatar geometry semantics, etc.). The function "processProperties()" handles the retrieval of (related) information.

[0223] In some embodiments, the attributes of a media element may include, for example, social and personal user information, as well as additional information that may be needed for, for example, encoding, transmitting and / or decoding bitstreams or processing any information present in them.

[0224] The purpose of this signal is to inform the decoder that attribute information is available for reading. This signal can take many forms, such as traditional coding standards used for geometric and supplementary information, such as MPEG-I SD, MPEG-AI, MPEG V-DMC, L-PCC, MPEG 4, H-anim, JVET, etc.

[0225]

[0226] Table 10: Attribute Semantics Transform into a decoder API The following section illustrates an exemplary application programming interface (API) for decoding avatar media content and processing information from the encoder.

[0227] Geometric shapes The geometry of an avatar represents the attributes that define the avatar asset and its semantic meaning. This can take the form of connected or disconnected vertices, skeletal structures, 3D markers, images, or contextual information. The function "decodeGeometry()" handles the acquisition of (related) information sent by the encoder.

[0228] In some embodiments, semantics may include decoding of vertex, face, region (e.g., geometry) markers, landmarks and skeletal markers, and texture markers for use in purposes such as classification, avatar-object interaction, avatar-avatar interaction, and avatar redirection (e.g., animation, style, geometry, etc.). Figure 4 Applications such as (and other elements shown in Tables 2 to 10).

[0229]

[0230] Table 11: Geometric Shape Semantics style The shape representation of the avatar media defines the attributes of the avatar style, such as clothing, hairstyle, accessories, visual appearance, personality, and behavior. The function "decodeStyle()" handles the retrieval of (related) information sent by the encoder.

[0231] In some embodiments, semantics may include decoding of vertex, face, region (e.g., geometry) markers, landmarks and skeletal markers, and texture markers for use in purposes such as classification, avatar-object interaction, avatar-avatar interaction, and avatar redirection (e.g., animation, style, geometry, etc.). Figure 4 Applications such as (and other elements shown in Tables 2 to 10).

[0232]

[0233] Table 12: Style Semantics Animation Animation in avatar media defines the properties of avatar motion, such as being presented as 2D video, immersive video, multi-view video, or in the form of a skeleton or geometric shape. The function "decodeAnimation()" handles the acquisition of (related) information sent by the encoder.

[0234] In some embodiments, semantics may include decoding of vertex, face, region (e.g., geometry) markers, landmarks and skeletal markers, and texture markers for use in purposes such as classification, avatar-object interaction, avatar-avatar interaction, and avatar redirection (e.g., animation, style, geometry, etc.). Figure 4 Applications such as (and other elements shown in Tables 2 to 10).

[0235]

[0236] Table 13: Animation Semantics scene The context of the avatar media defines the attributes of the scene in which the avatar exists, such as a video stream within a VR or AR system or a description of the world environment in a scene description format. The function "decodeContext()" handles the acquisition of (related) information sent by the encoder.

[0237] In some embodiments, semantics may include decoding of vertex, face, region (e.g., geometry) markers, landmarks and skeletal markers, and texture markers for use in purposes such as classification, avatar-object interaction, avatar-avatar interaction, and avatar redirection (e.g., animation, style, geometry, etc.). Figure 4 Applications such as (and other elements shown in Tables 2 to 10).

[0238]

[0239] Table 14: Contextual Semantics physical properties The physical properties of the avatar medium represent the attributes of avatar movement and scene interaction defined by a physical model based on the avatar and its environment. These attributes can describe gravity, wind or other natural forces, as well as lighting or material properties. The function "decodePhysics()" handles the acquisition of (related) information sent by the encoder.

[0240] In some embodiments, semantics may include decoding of vertex, face, region (e.g., geometry) markers, landmarks and skeletal markers, and texture markers for use in purposes such as classification, avatar-object interaction, avatar-avatar interaction, and avatar redirection (e.g., animation, style, geometry, etc.). Figure 4 Applications such as (and other elements shown in Tables 2 to 10).

[0241]

[0242] Table 15: Semantics of Physical Properties Touch The haptic representation of the avatar media defines the attributes that provide haptic feedback features available to the avatar. These features can take the form of, for example, object material properties, haptic sensors on the avatar, potential feedback responses from other objects and systems in the environment, haptic signals attached to specific body parts, or sensitivity maps. The function "decodeHaptics()" handles the acquisition of (related) information sent by the encoder.

[0243] In some embodiments, semantics may include decoding of vertex, face, region (e.g., geometry) markers, landmarks and skeletal markers, and texture markers for use in purposes such as classification, avatar-object interaction, avatar-avatar interaction, and avatar redirection (e.g., animation, style, geometry, etc.). Figure 4 Applications such as (and other elements shown in Tables 2 to 10).

[0244]

[0245] Table 16: Tactile Semantics audio The audio representation of the avatar media defines the properties of that avatar audio, such as speech and noise generated through contact or spatial audio properties. The function "decodeAudio()" handles the acquisition of (related) information sent by the encoder.

[0246] In some embodiments, semantics may include decoding of audio attributes, which may include audio related to spatial audio or speech.

[0247]

[0248] Table 17: Audio Semantics Metadata Metadata representing the media allows for the decoding of signals and data from other entities (such as other standards (MPEG-I SD, MPEG-AI, MPEG V-DMC, L-PCC, MPEG 4, H-anim, JVET)). The function "decodeMetadata()" handles the acquisition of (related) information sent by the encoder.

[0249] In some embodiments, semantics may include decoding of vertex, face, region (e.g., geometry) markers, landmarks and skeletal markers, and texture markers for use in purposes such as classification, avatar-object interaction, avatar-avatar interaction, and avatar redirection (e.g., animation, style, geometry, etc.). Figure 4 Applications such as (and other elements shown in Tables 2 to 10).

[0250]

[0251] Table 18: Metadata Semantics property The properties of the avatar media represent additional information that can further define the transmitted avatar, such as social or personal information, or additional information related to the application receiving data content that may not be included in the specification (such as proprietary physical properties, proprietary avatar geometry semantics, etc.). The function "decodeProperties()" handles the acquisition of (related) information sent by the encoder.

[0252] In some embodiments, semantics may include decoding of vertex, face, region (e.g., geometry) markers, landmarks and skeletal markers, and texture markers for use in purposes such as classification, avatar-object interaction, avatar-avatar interaction, and avatar redirection (e.g., animation, style, geometry, etc.). Figure 4 Applications such as (and other elements shown in Tables 2 to 10).

[0253]

[0254] Table 19: Attribute Semantics encoder Figure 12 This is a process diagram illustrating an exemplary encoder architecture according to some embodiments. Figure 12 The encoder architecture 1200 for the avatar media codec and the aforementioned properties are shown in more detail. This encoder is capable of processing various types of input files 1202, descriptive avatar media files (such as Graphics Library Transfer Format (.gltf) 1204, Extensible 3D Format (.x3d) 1206, JavaScript Object Notation (.json) 1208, Extensible Markup Language Format (.xml) 1210, and text formats (.txt, etc., to name some formats), which contain the information presented above. Output data from the encoder can have various forms. For some embodiments, such as... Figure 12As shown, two encoder output formats are used: a human-readable format (based on JSON in this exemplary embodiment) and a binary format. The JSON-based format (which, in some embodiments, may be stored in a JSON-based file 1212) transforms each element of the data structure (such as geometry, style, animation, and...) Figure 4 Metadata information is provided (as well as other elements shown in Tables 2 to 10) and references appropriate files. The binary format compresses (all) data into a single binary file 1214. In this format, data can be "grouped" to allow independent access to each element of the avatar data structure. According to some embodiments, in this context, "grouping" 1222 should be understood as grouping the elements of the avatar data structure into separate blocks containing, for example, metadata information, such as that described in Tables 2 to 10, and which may encapsulate the associated data after binary compression 1220. In some embodiments, grouping 1222 may include, for example, organizing the binary compressed data into dedicated data packets for each element of the data structure (e.g., geometry, style, etc.). For example, this feature can be used to request access to only a portion of the data during transmission. In some embodiments, data can be compressed into two or more binary files. Other encoding schemes and associated data formats representing avatar data ( Figure 12 (Not shown in the image) can also be used in some embodiments.

[0255] For example, in some embodiments, formatting the avatar data structure 1218 may include formatting the avatar data structure into blocks, wherein each block contains, for example... Figure 4 The avatar description element shown, and includes, for example, a header, thereby allowing the decoder to explicitly identify the content of the block.

[0256] In some embodiments, the input file can be a single format or a collection of files in different formats. Figure 12 In the scenario shown, which has a collection of multiple file formats, the goal of block format analysis 1216 and formatting 1218 is to acquire (relevant) information and arrange that information in such a way that the original file format can be recovered in binary or human-readable form. In some embodiments, a header portion may be included in the avatar data structure element "group" as defined above to inform the decoder how to decompress and recover the original format. In some embodiments, the arrangement of information acquired from the input file may be stored (or populated into) the avatar data structure.

[0257] The JSON output file can include one or more avatar data attributes. Figure 12A set of exemplary avatar data attributes is shown. Some JSON output files may include other avatar attributes not shown in this example. Furthermore, some JSON output files may use attribute formats different from those used in this example. Figure 12 In the example, the Geometry attribute can be encoded in ".x3d" format. The Style attribute can be encoded in ".xml" format. The Animation attribute can be encoded in ".gltf" format. The Haptics attribute can be encoded in ".json" format. Therefore, header sections or groups can be referenced in the same way to decode back to the original format in a lossless manner.

[0258] Figure 12 The format analysis 1216 and formatting 1218 of the blocks shown can use a parsing mechanism to extract (related) information from different file formats so that the information can be packaged and compressed into a binary compressed format or a human-readable format.

[0259] Formatting block 1218 allows the generation of a human-readable format, rather than a binary format. The human-readable format can be a JSON file. In some embodiments, the human-readable format can be a format other than JSON.

[0260] In some embodiments, binary compressed block 1220 applies lossless compression using the standard SPIHT algorithm and arithmetic coding (AC) and converts the data into a bit stream.

[0261] For some implementations, for “.json” files, binary compression 1220 follows the RFC 8949 concise binary object representation standard.

[0262] For some implementations, for “.x3d” files, binary compression 1220 conforms to the ISO / IEC 19776-3.2:2011 scalable 3D encoding standard.

[0263] In some implementations, for “.gltf” files, binary compression 1220 follows the open glTF 2.0 specification from the Khronos group to generate binary “.glb” files.

[0264] For some embodiments, for “.xml” files, binary compression 1220 follows the ISO / IEC 23001-1:2006 binary MPEG format for XML.

[0265] In some embodiments, grouping 1222 may include organizing binary compressed data into a data structure (such as geometry, style, or...). Figure 4Grouping of each element (and any other elements shown in Tables 2 to 10). In some embodiments, group 1222 may be used during transmission to request access to only a portion of the data.

[0266] In one scenario according to some embodiments, both human-readable files (such as JSON files) and binary files can be output from the encoder. Figure 12 Exemplary configurations according to some embodiments are presented. In another scenario according to some embodiments, only human-readable files may be output from the encoder. In yet another scenario according to some embodiments, only binary files may be output from the encoder.

[0267] In some embodiments, human-readable files may provide additional information to supplement and / or further interpret the binary compressed file. In some embodiments, information (e.g., information of a large size) may be compressed and transmitted using binary files to reduce the overall size of the transmission.

[0268] decoder Figure 13 This is a process diagram illustrating an exemplary decoder architecture according to some embodiments. Figure 13 The encoder architecture 1300 used to transform into a media codec and its aforementioned properties are shown in more detail. The decoder takes a binary file 1302 and / or a set of text files as input and outputs an interoperable raw file format. The input text files can be in the format shown below. Figure 13 The JSON format file shown is 1204, or it can be represented in any other text file format.

[0269] For a specific file format, the input binary format is degrouped 1306 and (binary) decompressed 1308 according to the same specifications used and presented in the encoder section described above. Data is extracted from the file and mapped to data structures associated with Tables 2 through 10. After this step, the application has information related to the avatar media codec. In some embodiments, a human-readable format can be extracted and mapped back to the original file format based on additional header information about attributes and the file format from which said attributes are extracted.

[0270] In some embodiments, binary decompressed data and / or JSON format data (or, in some embodiments, human-readable format data) can be processed via an exchange file format 1310, and one or more output files can be output 1312. In some embodiments, the output files may be in Graphics Library Transfer Format (.gltf) 1314, Extensible 3D Format (.x3d) 1316, JavaScript Object Notation (.json) 1318, Extensible Markup Language Format (.xml) 1320, and / or Text Format (.txt) format.

[0271] In one scenario according to some embodiments, the decoder can receive both human-readable files and binary files. Figure 13 Exemplary configurations according to some embodiments are presented. In another scenario according to some embodiments, only human-readable files may be received at the decoder, or the decoder may be configured only to receive human-readable files. In another scenario according to some embodiments, only binary files may be received at the decoder, and / or the decoder may be configured only to receive binary files.

[0272] According to some embodiments, the decoder may have binary output in addition to or in lieu of human-readable output.

[0273] Figure 14 This is a flowchart illustrating an exemplary process for encoding avatar data according to some embodiments. In some embodiments, exemplary process 1400 may include acquiring 1402 an input file, wherein the input file includes avatar content data. In some embodiments, exemplary process 1400 may also include determining 1404 a data type corresponding to the avatar content data. In some embodiments, exemplary process 1400 may also include processing 1406 the avatar data based on data type and codec type to extract information from the input file. In some embodiments, exemplary process 1400 may also include inferring 1408 template-based information about the avatar content data. In some embodiments, exemplary process 1400 may also include formatting 1410 the avatar content data based on configuration file type and template-based information. In some embodiments, exemplary process 1400 may also include encoding 1412 the formatted avatar data.

[0274] Figure 15This is a flowchart illustrating an exemplary process for decoding avatar data according to some embodiments. In some embodiments, exemplary process 1500 may include acquiring 1502 encoded avatar data. In some embodiments, exemplary process 1500 may also include decoding the encoded avatar data 1504 to generate decoded avatar data. In some embodiments, exemplary process 1500 may also include determining 1506 the data type corresponding to the decoded avatar data. In some embodiments, exemplary process 1500 may also include processing 1508 the decoded avatar data based on the data type and codec type. In some embodiments, exemplary process 1500 may also include rendering 1510 the processed and decoded avatar data.

[0275] While methods and systems according to some embodiments are generally discussed within the context of extended reality (XR), some embodiments can be applied to any XR scenario, such as virtual reality (VR) / mixed reality (MR) / augmented reality (AR) scenarios. Additionally, although the term "head-mounted display (HMD)" is used herein according to some embodiments, for some embodiments, some embodiments can be applied to wearable devices capable of supporting, for example, XR, VR, AR, and / or MR (which may or may not be attached to the head).

[0276] A first exemplary method according to some embodiments may include: acquiring an input file, wherein the input file includes avatar content data; determining a data type corresponding to the avatar content data; processing the avatar data based on the data type and codec type to extract information from the input file; inferring template-based information about the avatar content data; formatting the avatar content data based on a configuration file type and template-based information, wherein the configuration file type is a computer-generated (CG) configuration file type; and encoding the formatted avatar data.

[0277] In some embodiments of the first exemplary method, the data type is one of a dataset including: video data, image data, grid data, and game data.

[0278] In some embodiments of the first exemplary method, the codec type is one of the following codec sets: VPCC, GPCC, V-DMC, glTF, and USD.

[0279] In some embodiments of the first exemplary method, the formatted avatar data includes avatar animation data.

[0280] In some embodiments of the first exemplary method, the formatted avatar data includes avatar geometry data.

[0281] In some embodiments of the first exemplary method, the formatted avatar data includes avatar appearance data.

[0282] A first exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the methods listed above.

[0283] A second exemplary method according to some embodiments may include: acquiring encoded avatar data; decoding the encoded avatar data to generate decoded avatar data; determining a data type corresponding to the decoded avatar data; processing the decoded avatar data based on the data type and codec type, wherein the profile type is a computer-generated (CG) profile type; and rendering the processed and decoded avatar data.

[0284] In some embodiments of the second exemplary method, the data type is one of the following datasets: video data, image data, grid data, and game data.

[0285] In some embodiments of the second exemplary method, the codec type is one of the following codec sets: VPCC, GPCC, V-DMC, glTF, and USD.

[0286] In some embodiments of the second exemplary method, the decoded avatar data includes avatar animation data.

[0287] In some embodiments of the second exemplary method, the decoded avatar data includes avatar geometry data.

[0288] In some embodiments of the second exemplary method, the decoded avatar data includes avatar appearance data.

[0289] A second exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the methods listed above.

[0290] A third exemplary method according to some embodiments may include: acquiring an input file, wherein the input file includes avatar content data; determining a data type corresponding to the avatar content data; processing the avatar data based on the data type and codec type to extract information from the input file; inferring template-based information about the avatar content data; formatting the avatar content data based on a configuration file type and template-based information, wherein the configuration file type is a parameterized configuration file type; and encoding the formatted avatar data.

[0291] In some embodiments of the third exemplary method, the data type is one of the following datasets: video data, image data, grid data, and game data.

[0292] In some embodiments of the third exemplary method, the formatted avatar data includes weights of parameters.

[0293] In some embodiments of the third exemplary method, the parameters are one of a set of attributes including: animation attributes, shape attributes, texture attributes, and pose attributes.

[0294] In some embodiments of the third exemplary method, the formatted avatar data includes a parameter-based avatar model.

[0295] A third exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the methods listed above.

[0296] A fourth exemplary method according to some embodiments may include: acquiring encoded avatar data; decoding the encoded avatar data to generate decoded avatar data; determining a data type corresponding to the decoded avatar data; processing the decoded avatar data based on the data type and the codec type, wherein the configuration file type is a parameterized configuration file type; and rendering the processed and decoded avatar data.

[0297] In some embodiments of the fourth exemplary method, the data type is one of the following datasets: video data, image data, grid data, and game data.

[0298] In some embodiments of the fourth exemplary method, the decoded avatar data includes the weights of the parameters.

[0299] In some embodiments of the fourth exemplary method, the parameter is one of a set of attributes including: animation attributes, shape attributes, texture attributes, and pose attributes.

[0300] In some embodiments of the fourth exemplary method, the decoded avatar data includes a parameter-based avatar model.

[0301] A fourth exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the methods listed above.

[0302] A fifth exemplary method according to some embodiments may include: acquiring an input file, wherein the input file includes avatar content data; determining a data type corresponding to the avatar content data; processing the avatar data based on the data type and codec type to extract information from the input file; inferring template-based information about the avatar content data; formatting the avatar content data based on a configuration file type and template-based information, wherein the configuration file type is an artificial intelligence (AI) configuration file type; and encoding the formatted avatar data.

[0303] In some embodiments of the fifth exemplary method, the data type is one of the following datasets: video data, image data, grid data, and game data.

[0304] In some embodiments of the fifth exemplary method, the formatted avatar data includes information corresponding to a neural network.

[0305] In some embodiments of the fifth exemplary method, encoding the formatted avatar data includes using a neural network.

[0306] A fifth exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the methods listed above.

[0307] A sixth exemplary method according to some embodiments may include: acquiring encoded avatar data; decoding the encoded avatar data to generate decoded avatar data; determining a data type corresponding to the decoded avatar data; processing the decoded avatar data based on the data type and codec type, wherein the profile type is an artificial intelligence (AI) profile type; and rendering the processed and decoded avatar data.

[0308] In some embodiments of the sixth exemplary method, the data type is one of the following datasets: video data, image data, grid data, and game data.

[0309] In some embodiments of the sixth exemplary method, the decoded avatar data includes information corresponding to a neural network.

[0310] In some embodiments of the sixth exemplary method, decoding the encoded avatar data includes using a neural network.

[0311] A sixth exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the methods listed above.

[0312] A seventh exemplary device according to some embodiments may include at least one processor configured to perform any of the methods listed above.

[0313] An eighth exemplary device according to some embodiments may include a computer-readable storage medium storing instructions for causing one or more processors to perform any of the methods listed above.

[0314] A ninth exemplary device according to some embodiments may include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing at least one processor to perform any of the methods listed above.

[0315] Exemplary signals according to some embodiments may include encoded and formatted avatar data generated according to any of the methods listed above.

[0316] This disclosure describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described in a specific manner and are generally described in a way that may sound limiting, at least for the purpose of illustrating the various features. However, this is for the purpose of clarity of description and does not limit the disclosure or scope of these aspects. In fact, all the different aspects can be combined and interchanged to provide other aspects. Furthermore, these aspects can also be combined and interchanged with aspects described in previous documents.

[0317] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are specifically shown, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of implementation. At least one aspect relates generally to video encoding and decoding, and at least one other aspect relates generally to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions thereon stored thereon for encoding or decoding video data according to any of the methods, and / or computer-readable storage media having bitstreams generated according to any of the methods stored thereon.

[0318] In this disclosure, the terms "reconstruction" and "decoding" are used interchangeably, as are the terms "pixel" and "sample," and the terms "image," "picture," and "frame." Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0319] The terms HDR (High Dynamic Range) and SDR (Standard Dynamic Range) generally convey specific values ​​of dynamic range to those skilled in the art. However, it is also intended to include additional embodiments in which references to HDR should be understood as meaning "higher dynamic range" and references to SDR should be understood as meaning "lower dynamic range." Such additional embodiments are not bound by any specific values ​​of dynamic range that may generally be associated with the terms "high dynamic range" and "standard dynamic range."

[0320] This document describes various methods, each of which includes one or more steps or actions for implementing the described method. The order and / or use of specific steps and / or actions may be modified or combined unless the correct operation of the method requires a specific order. Additionally, in various embodiments, terms such as "first," "second," etc., may be used to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless specifically required, the use of such terms does not imply a sequence of operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding, but may occur, for example, before, during, or in a time period overlapping with the second decoding.

[0321] For example, various numerical values ​​may be used in this disclosure. Specific values ​​are for illustrative purposes, and the aspects described are not limited to these specific values.

[0322] The embodiments described herein can be implemented by computer software, either by a processor or other hardware or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, the processor can be of any type suitable for the technical environment and can encompass one or more microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0323] Various implementations involve decoding. As used in this disclosure, “decoding” can encompass all or part of a process performed, for example, on a received encoded sequence, to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also includes, or alternatively includes, processes performed by a decoder of the various embodiments described in this disclosure, such as extracting an image from a tiled (packed) image, determining an upsampling filter to use and then upsampling the image, and flipping the image back to its intended orientation.

[0324] As another example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Based on the specific context of the description, it will become clear whether the phrase "decoding process" is intended to specifically refer to a subset of operations or to refer to the broader decoding process.

[0325] Various implementations involve encoding. Similar to the discussion above regarding “decoding,” “encoding” as used in this disclosure can include, for example, all or part of the processing performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential coding, transform, quantization, and entropy coding. In various embodiments, such processes also include, or alternatively include, processes performed by a decoder of the various embodiments described in this disclosure.

[0326] As another example, in one embodiment, "encoding" refers only to entropy encoding; in another embodiment, "encoding" refers only to differential encoding; and in yet another embodiment, "encoding" refers to a combination of entropy encoding and differential encoding. Based on the specific context of the description, it will become clear whether the phrase "encoding process" is intended to specifically refer to a subset of operations or to refer to a broader encoding process.

[0327] When a diagram is presented as a flowchart, it should be understood that a block diagram of the corresponding device is also provided. Similarly, when a diagram is presented as a block diagram, it should be understood that a flowchart of the corresponding method / process is also provided.

[0328] Various embodiments involve rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is typically considered, often due to constraints on computational complexity. Rate-distortion optimization is generally formulated as minimizing a rate-distortion function, which is a weighted sum of code rate and distortion. Different approaches exist to address the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all encoding options, including all considered modes or encoding parameter values, where a comprehensive evaluation of the encoding cost and associated distortion of the reconstructed signal is performed after encoding and decoding. Faster methods can also be used to save encoding complexity, particularly by calculating approximate distortion based on prediction or prediction of the residual signal rather than the reconstructed signal. A hybrid of these two approaches can also be used, such as by using approximate distortion for only some possible encoding options and full distortion for others. Other methods evaluate only a subset of possible encoding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization is not necessarily a comprehensive evaluation of encoding cost and associated distortion.

[0329] The embodiments and aspects described herein can be implemented, for example, in methods or processes, apparatuses, software programs, data streams, or signals. Even if discussed only in the context of a single embodiment (e.g., discussed only as a method), embodiments of the discussed features can also be implemented in other forms (e.g., apparatuses or programs). Apparatuses can be implemented, for example, with suitable hardware, software, and firmware. Methods can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0330] The references to "an embodiment" or "an embodiment" or "an implementation" or "implementation", as well as other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the phrases "in an embodiment" or "in a manifestation" or "in an implementation" or "in an implementation", as well as any other variations, appearing throughout this disclosure, do not necessarily refer to the same embodiment.

[0331] Additionally, this disclosure may relate to “determining” various types of information. Determining information may include one or more of the following: for example, estimated information, calculated information, predicted information, or information retrieved from memory.

[0332] Furthermore, this disclosure may relate to “accessing” various types of information. Accessing information may include one or more of the following: for example, receiving information, retrieving information (e.g., retrieving from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0333] Additionally, this disclosure may relate to "receiving" various types of information. Like "access," the intention to receive is a broad term. Receiving information may include one or more of the following: for example, accessing information or retrieving information (e.g., retrieving from memory). Furthermore, "receiving" is generally referred to in one or more ways during operation, such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0334] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As yet another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” this wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). This can be extended to as many as the items listed.

[0335] Furthermore, as used herein, the term “signal” specifically refers to instructing the corresponding decoder to do something. For example, in some embodiments, the encoder signals a specific parameter among several parameters for region-based filter parameter selection for artifact removal filtering. In this way, in embodiments, the same parameter is used at both the encoder and decoder sides. Thus, for example, the encoder can send (explicitly signal) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling can be used without sending (implicitly signal) to simply allow the decoder to know and select the specific parameter. Bit savings are implemented in various embodiments by avoiding the transmission of any actual functionality. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to send information to the corresponding decoder. Although the verb form of the word “signal” has been referred to above, the word “signal” can also be used as a noun herein.

[0336] Implementation methods can generate various signals that are formatted to carry information, such as information that can be stored or transmitted. The information may include, for example, instructions for performing a method, or data generated by one of the described implementation methods. For example, the signal may be formatted to carry a bit stream of the described embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding the data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0337] Several embodiments are described. Features of these embodiments may be provided individually or in any combination across various claim classes and types. Furthermore, embodiments may include one or more of the following features, means, or aspects individually or in any combination across various claim classes and types: Adjust the residual at the encoder according to any of the embodiments discussed.

[0338] A bitstream or signal that includes one or more of the described syntactic elements or their variants.

[0339] A bitstream or signal, which includes syntax for conveying information generated according to any embodiment of the described embodiments.

[0340] Syntactic elements are inserted into the signaling, which enable the decoder to adjust the residuals in a manner corresponding to that used by the encoder.

[0341] Create and / or send and / or receive and / or decode bitstreams or signals, which include one or more of the described syntactic elements or their variants.

[0342] Create and / or send and / or receive and / or decode according to any of the described embodiments.

[0343] The method, process, apparatus, medium for storing instructions, medium for storing data, or signal according to any of the described embodiments.

[0344] A TV, set-top box, mobile phone, tablet computer or other electronic device that performs adjustments to the filter parameters according to any of the described embodiments.

[0345] A TV, set-top box, mobile phone, tablet computer or other electronic device that performs adjustment of filter parameters according to any of the described embodiments and displays (e.g., using a monitor, screen or other type of display) the resulting image.

[0346] A TV, set-top box, mobile phone, tablet computer or other electronic device that selects channels (e.g., using a tuner) to receive signals including encoded images and performs adjustments to filter parameters according to any of the described embodiments.

[0347] A TV, set-top box, mobile phone, tablet computer or other electronic device that receives signals including encoded images over the air (e.g., using an antenna) and performs adjustments to filter parameters according to any of the described embodiments.

[0348] It should be noted that the various hardware elements, one or more of which are described in the embodiments, are referred to as “modules”, which implement (i.e., perform, execute, etc.) the various functions described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices) that a person skilled in the art would consider suitable for a given implementation. Each described module may also include executable instructions for performing one or more functions described as being performed by the respective module, and it should be noted that these instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., and may be stored in any suitable non-transitory computer-readable medium such as RAM, ROM, etc.

[0349] Although the features and elements have been described above in specific combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with other features and elements. Furthermore, the methods described herein can be implemented in a computer program, software, or firmware incorporated into a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM discs and digital multifunction disks (DVDs). The processor associated with the software can be used to implement a wireless frequency transceiver for a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method comprising: Get the input file. The input file mentioned above includes avatar content data; Determine the data type corresponding to the avatar content data; The avatar data is processed based on the data type and codec type to extract information from the input file; Infer template-based information about the avatar content data; The avatar content data is formatted based on the configuration file type and the template-based information. The configuration file type mentioned above is a computer-generated (CG) configuration file type; and Encode the formatted avatar data.

2. The method of claim 1, wherein the data type is one of the following data sets: video data, image data, grid data, and game data.

3. The method of any one of claims 1 to 2, wherein the codec type is one of a set of codecs including VPCC, GPCC, V-DMC, glTF, and USD.

4. The method of any one of claims 1 to 3, wherein the formatted avatar data includes avatar animation data.

5. The method of any one of claims 1 to 3, wherein the formatted avatar data includes avatar geometry data.

6. The method of any one of claims 1 to 3, wherein the formatted avatar data includes avatar appearance data.

7. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform the method as described in any one of claims 1 to 6.

8. A method comprising: Obtain the encoded avatar data. The encoded avatar data is associated with a computer-generated (CG) profile type; The encoded avatar data is decoded to generate decoded avatar data; Determine the data type corresponding to the decoded avatar data; The decoded avatar data is processed based on the data type and codec type; as well as Render the processed and decoded avatar data.

9. The method of claim 8, wherein the data type is one of the following: video data, image data, grid data, and game data.

10. The method of any one of claims 8 to 9, wherein the codec type is one of a set of codecs including VPCC, GPCC, V-DMC, glTF, and USD.

11. The method of any one of claims 8 to 10, wherein the decoded avatar data includes avatar animation data.

12. The method of any one of claims 8 to 10, wherein the decoded avatar data includes avatar geometry data.

13. The method of any one of claims 8 to 10, wherein the decoded avatar data includes avatar appearance data.

14. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform the method as described in any one of claims 8 to 13.

15. A method comprising: Get the input file. The input file mentioned above includes avatar content data; Determine the data type corresponding to the avatar content data; The avatar data is processed based on the data type and codec type to extract information from the input file; Infer template-based information about the avatar content data; The avatar content data is formatted based on the configuration file type and the template-based information. The configuration file type mentioned above is a parameterized configuration file type; and Encode the formatted avatar data.

16. The method of claim 15, wherein the data type is one of a dataset including: video data, image data, grid data, and game data.

17. The method of any one of claims 15 to 16, wherein the formatted avatar data includes weights of parameters.

18. The method of claim 17, wherein the parameter is one of a set of attributes including: animation attributes, shape attributes, texture attributes, and pose attributes.

19. The method of any one of claims 15 to 18, wherein the formatted avatar data includes a parameter-based avatar model.

20. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform the method as described in any one of claims 15 to 19.

21. A method comprising: Obtain the encoded avatar data. The encoded avatar data is associated with a parameterized profile type; The encoded avatar data is decoded to generate decoded avatar data; Determine the data type corresponding to the decoded avatar data; The decoded avatar data is processed based on the data type and codec type; as well as Render the processed and decoded avatar data.

22. The method of claim 21, wherein the data type is one of a dataset including: video data, image data, grid data, and game data.

23. The method of any one of claims 21 to 22, wherein the decoded avatar data includes weights of parameters.

24. The method of claim 23, wherein the parameter is one of a set of attributes including: animation attributes, shape attributes, texture attributes, and pose attributes.

25. The method of any one of claims 21 to 24, wherein the decoded avatar data includes a parameter-based avatar model.

26. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform the method as described in any one of claims 21 to 25.

27. A method comprising: Get the input file. The input file mentioned above includes avatar content data; Determine the data type corresponding to the avatar content data; The avatar data is processed based on the data type and codec type to extract information from the input file; Infer template-based information about the avatar content data; The avatar content data is formatted based on the configuration file type and the template-based information. The configuration file type mentioned above is an artificial intelligence (AI) configuration file type; and Encode the formatted avatar data.

28. The method of claim 27, wherein the data type is one of a dataset including: video data, image data, grid data, and game data.

29. The method of any one of claims 27 to 28, wherein the formatted avatar data includes information corresponding to a neural network.

30. The method of any one of claims 27 to 29, wherein encoding the formatted avatar data comprises: Use neural networks.

31. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform the method as described in any one of claims 27 to 30.

32. A method comprising: Obtain the encoded avatar data. The coded avatar data is associated with an artificial intelligence (AI) profile type; The encoded avatar data is decoded to generate decoded avatar data; Determine the data type corresponding to the decoded avatar data; The decoded avatar data is processed based on the data type and codec type; as well as Render the processed and decoded avatar data.

33. The method of claim 32, wherein the data type is one of the following data sets: video data, image data, grid data, and game data.

34. The method of any one of claims 32 to 33, wherein the decoded avatar data includes information corresponding to a neural network.

35. The method of any one of claims 32 to 34, wherein decoding the encoded avatar data comprises: Use neural networks.

36. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform the method as described in any one of claims 32 to 35.

37. An apparatus comprising at least one processor configured to perform the method as described in any one of claims 1 to 6, 8 to 13, 15 to 19, 21 to 25, 27 to 30 and 32 to 35.

38. An apparatus comprising a computer-readable medium storing instructions for causing one or more processors to perform the method as described in any one of claims 1 to 6, 8 to 13, 15 to 19, 21 to 25, 27 to 30, and 32 to 35.

39. An apparatus comprising at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform the method as described in any one of claims 1 to 6, 8 to 13, 15 to 19, 21 to 25, 27 to 30 and 32 to 35.

40. A signal comprising encoded and formatted avatar data generated according to any one of claims 1 to 6, 15 to 19 and 27 to 30.

Images