Avatar JSON interchange file format

Abstract

Some embodiments of a method can include obtaining avatar JSON interchange file (AJIF) data, decoding the AJIF data to generate avatar data, obtaining animation parameters from the avatar data, obtaining user action data corresponding to an action of a user, generating updated avatar data based on the user action data, and sending the updated avatar data to a client device. For some embodiments, a method can include generating avatar JSON interchange file (AJIF) data corresponding to an avatar, sending the AJIF data to an application server, obtaining user action data corresponding to an action of a user, sending the user action data corresponding to the action of the user to the application server, receiving scene update data from the application server, wherein the scene update data includes updated avatar data corresponding to the action of the user, and rendering a scene update based on the scene update data.

Description

Cross-references to related applications

[0001] This application claims the benefit of European Patent Application No. EP24305094, filed on January 15, 2024, entitled “Transformation into JSON Exchange File Format”, which is hereby incorporated by reference in its entirety.

[0002] Incorporation This application is incorporated herein by reference in its entirety by the following application: European Patent Application Serial No. EP23306745, entitled "Avatar Media Representation for Transmission" and filed on 10 October 2023 (" 745 application ”); European Patent Application Serial No. EP23306756, entitled “Geometric Incarnation Media Codec for Transmission”, filed on October 28, 2023 ( Application 756 (); European Patent Application Serial No. EP24305093, entitled “Avatar Media Representation for Transmission”, filed on January 15, 2024; and European Patent Application Serial No. EP24305092, entitled “Geometric Avatar Media Codec for Transmission”, filed on January 15, 2024. Background Technology

[0003] Extended Reality (XR) is a technology that enables interactive experiences where real-world environments and / or video content are enhanced through virtual content that can be defined across multiple sensory modalities, including visual, auditory, tactile, and other sensory modalities. During application runtime, virtual content (e.g., 3D content or audio / video files) is rendered in real-time in a manner consistent with the user's context (environment, perspective, device, etc.). Scene graphs (e.g., scene graphs proposed by Khronos / glTF and their extensions defined in the MPEG scene description format or Apple / USDZ) are possible ways to represent the content to be rendered. They combine, on the one hand, an explanatory description of the scene structure linking real-world objects and virtual objects, and on the other hand, a binary representation of the virtual content. Summary of the Invention

[0004] A first exemplary method according to some embodiments may include: acquiring avatar JSON exchange file (AJIF) data; decoding the AJIF data to generate avatar data; acquiring animation parameters from the avatar data; acquiring user action data corresponding to the user's actions; generating updated avatar data based on the user action data; and sending the updated avatar data to a client device.

[0005] Some embodiments of the first exemplary method may further include: generating avatar animation rendering data using animation parameters.

[0006] In some embodiments of the first exemplary method, the user action data is facial landmark data.

[0007] In some embodiments of the first exemplary method, generating updated avatar data includes: obtaining a hybrid shape weight update corresponding to facial landmark data; and applying the hybrid shape weight update to the avatar data to generate updated avatar data.

[0008] In some embodiments of the first exemplary method, generating updated avatar data includes: obtaining vertex position data associated with the avatar from the avatar data; obtaining linear blend skin weights from the avatar data; obtaining a transformation matrix from the avatar data; and using the vertex position data, linear blend skin weights, and transformation matrix to generate updated avatar data.

[0009] In some embodiments of the first exemplary method, the animation parameters include a channel object corresponding to a portion of the avatar.

[0010] Some embodiments of the first exemplary method may further include: analyzing avatar data to determine the type of avatar represented in the avatar data.

[0011] Some embodiments of the first exemplary method may further include: analyzing avatar data to determine an avatar portion mapping of the avatar data.

[0012] Some embodiments of the first exemplary method may further include: analyzing avatar data to determine at least one node in the avatar data associated with the avatar.

[0013] In some embodiments of the first exemplary method, AJIF data includes data of at least one of the following data types associated with the avatar: original avatar data, geometry data, texture data, skeleton data, controller data, and level of detail (LOD) data.

[0014] In some embodiments of the first exemplary method, AJIF data includes at least one of the following data types: file node data, metadata, and file asset data.

[0015] In some embodiments of the first exemplary method, AJIF data corresponds to two or more files.

[0016] In some embodiments of the first exemplary method, the AJIF data includes image data associated with the avatar.

[0017] A first exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the claims listed above.

[0018] A second exemplary method according to some embodiments may include: generating avatar JSON exchange file (AJIF) data corresponding to an avatar; sending the AJIF data to an application server; obtaining user action data corresponding to a user's action; sending the user action data corresponding to a user's action to the application server; receiving scene update data from the application server, wherein the scene update data includes updated avatar data corresponding to the user's action; and rendering an update to the scene based on the received scene update data.

[0019] In some embodiments of the second exemplary method, the user action data is facial landmark data.

[0020] A second exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the claims listed above.

[0021] A third exemplary method according to some embodiments may include: acquiring avatar JSON exchange file (AJIF) data; decoding the AJIF data to generate avatar data; acquiring animation parameters from the avatar data; acquiring user action data corresponding to the user's actions; generating updated avatar data based on the user action data; and rendering the updated avatar data to a display device.

[0022] Some embodiments of the third exemplary method may further include: generating avatar animation rendering data using animation parameters.

[0023] In some embodiments of the third exemplary method, the user action data is facial landmark data.

[0024] In some embodiments of the third exemplary method, generating updated avatar data includes: obtaining a hybrid shape weight update corresponding to facial landmark data; and applying the hybrid shape weight update to the avatar data to generate updated avatar data.

[0025] In some embodiments of the third exemplary method, generating updated avatar data includes: obtaining vertex position data associated with the avatar from the avatar data; obtaining linear blend skin weights from the avatar data; obtaining a transformation matrix from the avatar data; and using the vertex position data, linear blend skin weights, and transformation matrix to generate updated avatar data.

[0026] In some embodiments of the third exemplary method, the animation parameters include a channel object corresponding to a portion of the avatar.

[0027] Some embodiments of the third exemplary method may further include: analyzing avatar data to determine the type of avatar represented in the avatar data.

[0028] Some embodiments of the third exemplary method may further include: analyzing avatar data to determine the avatar portion mapping of the avatar data.

[0029] Some embodiments of the third exemplary method may further include: analyzing avatar data to determine at least one node in the avatar data associated with the avatar.

[0030] In some embodiments of the third exemplary method, the AJIF data includes data of at least one of the following data types associated with the avatar: original avatar data, geometry data, texture data, skeleton data, controller data, and level of detail (LOD) data.

[0031] In some embodiments of the third exemplary method, AJIF data includes data of at least one of the following data types: file node data, metadata, and file asset data.

[0032] In some embodiments of the third exemplary method, AJIF data corresponds to two or more files.

[0033] In some embodiments of the third exemplary method, the AJIF data includes image data associated with the avatar.

[0034] A third exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the claims listed above.

[0035] A fourth exemplary method according to some embodiments may include: acquiring avatar JSON exchange file (AJIF) data; decoding the AJIF data to generate avatar data; acquiring animation parameters from the avatar data; receiving facial landmark data corresponding to a user's facial movements; generating updated avatar data based on the facial landmark data; and sending the updated avatar data to a client device.

[0036] A fourth exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the claims listed above.

[0037] A fifth exemplary method according to some embodiments may include: acquiring identity data about an avatar; generating advanced information data using the identity data; encoding the advanced information data; acquiring skeletal data about the avatar; generating mesh, mapping, and skeletal information data using the skeletal data; encoding the mesh, mapping, and skeletal information data; acquiring animation data about the avatar; generating controller information data using the animation data; encoding the controller information data; acquiring appearance data about the avatar; generating texture information data using the appearance data; encoding the texture information data; receiving a request to save avatar data; compressing the encoded advanced information data, the encoded mesh, mapping, and skeletal information data, the encoded controller information data, and the encoded texture information data to generate compressed avatar data; and sending the compressed avatar data.

[0038] A fifth exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the claims listed above.

[0039] A sixth exemplary device according to some embodiments may include: at least one processor configured to perform any of the claims listed above.

[0040] A seventh exemplary device according to some embodiments may include a computer-readable storage medium storing instructions for causing one or more processors to perform any of the claims listed above.

[0041] An eighth exemplary device according to some embodiments may include: at least one processor and at least one non-transitory computer-readable medium storing instructions for causing at least one processor to perform any of the claims listed above.

[0042] Exemplary signals according to some embodiments may include updated avatar data generated according to any of the claims listed above. Attached Figure Description

[0043] Figure 1A This is a system diagram illustrating an exemplary communication system according to some embodiments.

[0044] Figure 1B This illustrates that, according to some embodiments, it is possible to Figure 1A A system diagram of an exemplary wireless transmit / receive unit (WTRU) used within the communication system shown.

[0045] Figure 1CThis is a system diagram illustrating a set of exemplary interfaces for a system according to some embodiments.

[0046] Figure 2A This is a functional block diagram of a block-based video encoder (such as an encoder for general video coding (VVC)) according to some embodiments.

[0047] Figure 2B This is a functional block diagram of a block-based video decoder (such as a decoder for VVC) according to some embodiments.

[0048] Figure 3A This is a schematic side view illustrating an exemplary waveguide display that can be used with extended reality (XR) applications according to some embodiments.

[0049] Figure 3B This is a schematic side view illustrating an exemplary alternative display type that can be used with extended reality applications according to some embodiments.

[0050] Figure 3C This is a schematic side view illustrating an exemplary alternative display type that can be used with extended reality applications according to some embodiments.

[0051] Figure 4 This is a process diagram illustrating an exemplary top-level encoding and decoding process according to some embodiments.

[0052] Figure 5 This is a file data mapping that illustrates an exemplary .ajif file mapping according to some embodiments.

[0053] Figure 6 This is a schematic diagram illustrating an exemplary avatar mesh model according to some embodiments.

[0054] Figure 7 This is a schematic diagram illustrating an exemplary avatar skeletal model according to some embodiments.

[0055] Figure 8 This is a process diagram illustrating an exemplary encoder architecture according to some embodiments.

[0056] Figure 9 This is a process diagram illustrating an exemplary decoder architecture according to some embodiments.

[0057] Figure 10 This is a process diagram illustrating an exemplary process for streaming AJIF files according to some embodiments.

[0058] Figure 11 This is a process diagram illustrating an exemplary video conferencing architecture with AJIF format data according to some embodiments.

[0059] Figure 12 This is a message sequence diagram illustrating an exemplary process for a video conferencing use case according to some embodiments.

[0060] Figure 13A and Figure 13B A system diagram is formed illustrating a set of exemplary interfaces for creating an immersive experience with one or more users / avatars, according to some embodiments.

[0061] Figure 14A and Figure 14B A system diagram is formed illustrating a set of exemplary interfaces for a virtual world having multiple online users / avatars, according to some embodiments.

[0062] Figure 15 This is a message sequence diagram illustrating an exemplary process of asset creation workflow between a client and server application according to some embodiments.

[0063] Figure 16 This is a diagram illustrating exemplary results of an asset creation process related to avatar representation according to some embodiments.

[0064] Figure 17 This is a flowchart illustrating an exemplary process for decoding Avatar JSON Exchange File (AJIF) data according to some embodiments.

[0065] Figure 18 This is a flowchart illustrating an exemplary process for encoding Avatar JSON Exchange File (AJIF) data according to some embodiments.

[0066] The entities, connections, arrangements, etc., depicted and described in conjunction with the various figures are presented by way of example rather than limitation. Therefore, any and all statements or other indications regarding what a particular figure “depicts,” what a particular element or entity in a particular figure “is” or “has,” and any and all similar statements that might be interpreted in isolation and out of context as absolute and therefore limiting, should only be correctly interpreted as constructively beginning with a clause such as “In at least one embodiment, …”. For the sake of simplicity and clarity, this implicit leading clause will not be repeated in the detailed description. Annoying . Detailed Implementation

[0067] Figure 1AThis diagram illustrates an exemplary communication system 100 that may implement one or more of the disclosed embodiments. The communication system 100 may be a multiple access system providing content such as voice, data, video, messaging, and broadcasting to multiple wireless users. The communication system 100 enables multiple wireless users to access such content through shared system resources including wireless broadband. For example, the communication system 100 may employ one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero-Tail Unique Word DFT Extended OFDM (ZT UW DTS-s OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtered OFDM, Filter Bank Multicarrier (FBMC), etc.

[0068] like Figure 1A As shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106, public switched telephone network (PSTN) 108, Internet 110, and other networks 112. However, it will be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, and 102d may be any type of device configured to operate and / or communicate in a wireless environment. For example, WTRUs 102a, 102b, 102c, and 102d (any of which may be referred to as a “station” and / or “STA”) may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of industrial and / or automated processing chains), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, etc. Any of WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0069] The communication system 100 may also include base station 114a and / or base station 114b. Each of base stations 114a and 114b may be any type of device configured to wirelessly interface with at least one of WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks such as CN 106, the Internet 110, and / or other networks 112. For example, base stations 114a and 114b may be base transceiver stations (BTS), Node-B, eNode B, home Node-B, home eNodeB, gNB, NR NodeB, site controllers, access points (APs), wireless routers, etc. Although base stations 114a and 114b are each depicted as a single element, it will be understood that base stations 114a and 114b may include any number of interconnected base stations and / or network elements.

[0070] Base station 114a may be part of RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as base station controllers (BSCs), radio network controllers (RNCs), relay nodes, etc. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage for a specific geographic area that may be relatively fixed or may change over time. A cell may also be divided into cell sectors. For example, the cell associated with base station 114a may be divided into three sectors. Therefore, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver per sector of the cell. In embodiments, base station 114a may employ multiple-input multiple-output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0071] Base stations 114a and 114b can communicate with one or more of WTRUs 102a, 102b, 102c, and 102d via air interface 116, which can be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). Any suitable radio access technology (RAT) can be used to establish air interface 116.

[0072] More specifically, as described above, the communication system 100 can be a multiple access system and can employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, base station 114a in RAN 104 / 113 and WTRUs 102a, 102b, 102c can implement wireless technologies, such as using Wideband CDMA (WCDMA) to establish Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA) for air interface 116. WCDMA can include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA can include High-Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0073] In the embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement wireless technologies, such as using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro) to establish Evolved UMTS Terrestrial Radio Access (E-UTRA) for air interface 116.

[0074] In the embodiments, base station 114a and WTRUs 102a, 102b, 102c can implement wireless technologies, such as using New Radio (NR) to establish NR wireless access for air interface 116.

[0075] In the embodiments, base station 114a and WTRUs 102a, 102b, and 102c can implement various radio access technologies. For example, base station 114a and WTRUs 102a, 102b, and 102c can, for instance, use the dual connectivity (DC) principle to jointly implement LTE radio access and NR radio access. Therefore, the air interface used by WTRUs 102a, 102b, and 102c can be characterized by various types of radio access technologies and / or by transmissions sent to / from various types of base stations (e.g., eNBs and gNBs).

[0076] In other embodiments, base station 114a and WTRUs 102a, 102b, 102c may implement wireless technologies such as IEEE 802.11 (i.e., Wi-Fi), IEEE 802.16 (i.e., WiMAX), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rate GSM Evolution (EDGE), GSM EDGE (GERAN), etc.

[0077] Figure 1A Base station 114b can be, for example, a wireless router, a home Node-B, a home eNode-B, or an access point, and can utilize any suitable RAT to facilitate wireless connectivity in localized areas such as commercial locations, homes, vehicles, campuses, industrial facilities, air corridors (e.g., for use by drones), roads, etc. In one embodiment, base station 114b and WTRUs 102c, 102d can implement wireless technologies such as IEEE 802.11 to establish a wireless local area network (WLAN). In another embodiment, base station 114b and WTRUs 102c, 102d can implement wireless technologies such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, base station 114b and WTRUs 102c, 102d can utilize cellular-based RATs (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish picocells or femtocells. Figure 1A As shown, base station 114b can be directly connected to Internet 110. Therefore, base station 114b does not need to access Internet 110 via CN 106.

[0078] RAN 104 / 113 can communicate with CN 106, which can be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRUs 102a, 102b, 102c, and 102d. Data can have different Quality of Service (QoS) requirements, such as different throughput requirements, latency requirements, fault tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. CN 106 can provide call control, billing services, location-based services, prepaid calling, internet connectivity, video distribution, etc., and / or perform advanced security functions such as user authentication. Although... Figure 1AAs not shown, but will be understood, RAN 104 / 113 and / or CN 106 can communicate directly or indirectly with other RANs using the same RAT as or a different RAT than RAN 104 / 113. For example, in addition to connecting to RAN 104 / 113, which may be utilizing NR radio technology, CN 106 can also communicate with another RAN (not shown) using GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0079] CN 106 can also serve as a gateway for WTRUs 102a, 102b, 102c, and 102d to access PSTN 108, the Internet 110, and / or other networks 112. PSTN 108 may include a circuit-switched telephone network providing Common Old-Style Telephone Service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices using common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) from the TCP / IP Internet Protocol suite. Network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs, which may use the same RAT as RAN 104 / 113 or a different RAT.

[0080] Some or all of the WTRUs 102a, 102b, 102c, and 102d in communication system 100 may include multi-mode capabilities (e.g., WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). For example, Figure 1A The WTRU 102c shown can be configured to communicate with base station 114a, which can employ cellular wireless technology, and with base station 114b, which can employ IEEE 802 wireless technology.

[0081] Figure 1B This is a system diagram illustrating an exemplary WTRU 102. (See diagram below.) Figure 1B As shown, WTRU 102 may include a processor 118, a transceiver 120, a transmitting / receiving element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a Global Positioning System (GPS) chipset 136, and / or other peripheral devices 138, etc. It will be understood that, while remaining consistent with the embodiments, WTRU 102 may include any sub-combination of the foregoing elements.

[0082] Processor 118 can be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. Processor 118 can perform signal encoding, data processing, power control, input / output processing, and / or any other functions that enable WTRU 102 to operate in a wireless environment. Processor 118 can be coupled to transceiver 120, which can be coupled to transmitting / receiving element 122. Although Figure 1B The processor 118 and transceiver 120 are depicted as separate components, but it will be understood that the processor 118 and transceiver 120 can be integrated together in an electronic package or chip.

[0083] Transmitting / receiving element 122 can be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via air interface 116. For example, in one embodiment, transmitting / receiving element 122 can be an antenna configured to transmit and / or receive RF signals. In one embodiment, transmitting / receiving element 122 can be a transmitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, transmitting / receiving element 122 can be configured to transmit and / or receive both RF signals and optical signals. It will be understood that transmitting / receiving element 122 can be configured to transmit and / or receive any combination of wireless signals.

[0084] Although the transmitting / receiving element 122 is in Figure 1B While depicted as a single element, WTRU 102 may include any number of transmit / receive elements 122. More specifically, WTRU 102 may employ MIMO technology. Thus, in one embodiment, WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals via air interface 116.

[0085] Transceiver 120 can be configured to modulate signals transmitted by transmitting / receiving element 122 and demodulate signals received by transmitting / receiving element 122. As described above, WTRU 102 can have multi-mode capability. Therefore, transceiver 120 can include multiple transceivers for enabling WTRU 102 to communicate via various RATs (e.g., such as NR and IEEE 802.11).

[0086] The processor 118 of WTRU 102 can be coupled to and receive user input data from: a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) unit or an organic light-emitting diode (OLED) display unit). The processor 118 can also output user data to the speaker / microphone 124, keypad 126, and / or display / touchpad 128. Additionally, the processor 118 can access information and store data from any suitable type of memory, such as non-removable memory 130 and / or removable memory 132. Non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. Removable memory 132 may include a subscriber identity module (SIM) card, memory stick, secure digital storage (SD) card, etc. In other embodiments, the processor 118 can access information and store data from memory not actually located on WTRU 102, such as on a server or home computer (not shown).

[0087] The processor 118 may receive power from the power supply 134 and may be configured to distribute power to other components in the WTRU 102 and / or control power to those other components. The power supply 134 may be any suitable device for powering the WTRU 102. For example, the power supply 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0088] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) about the current location of the WTRU 102. In addition to or instead of information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) via air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be understood that, while remaining consistent with the embodiments, the WTRU 102 may acquire location information using any suitable location determination method.

[0089] The processor 118 can also be connected to other peripheral devices 138, which may include one or more software and / or hardware modules that provide additional features, functions, and / or wired or wireless connectivity. For example, peripheral devices 138 may include accelerometers, electronic compasses, satellite transceivers, digital cameras (for photos and / or video), Universal Serial Bus (USB) ports, vibration devices, television transceivers, hands-free headsets, Bluetooth® modules, FM radio units, digital music players, media players, video game player modules, internet browsers, virtual reality and / or augmented reality (VR / AR) devices, activity trackers, etc. Peripheral devices 138 may include one or more sensors, which may be one or more of the following: gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors; geolocation sensors; altimeters, light sensors, touch sensors, magnetometers, barometers, gesture sensors, biometric sensors, and / or humidity sensors.

[0090] WTRU 102 may include a full-duplex radio, wherein the transmission and reception of some or all of the signals (e.g., associated with a specific subframe of both UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit to reduce and / or substantially eliminate self-interference via hardware (e.g., a choke) or signal processing via a processor (e.g., a separate processor (not shown) or via processor 118). In embodiments, WTRU 102 may include a half-duplex radio, wherein the transmission and reception of some or all of the signals (e.g., associated with a specific subframe of both UL (e.g., for transmission) or downlink (e.g., for reception)) may be concurrent and / or simultaneous.

[0091] Despite WTRU in Figures 1A to 1B While described as a wireless terminal, it is envisioned that, in some representative embodiments, such a terminal may (e.g., temporarily or permanently) use a wired communication interface with a communication network.

[0092] In a representative embodiment, the other network 112 may be a WLAN.

[0093] Given Figures 1A to 1B The functions described herein, along with their corresponding descriptions, can be performed by one or more emulation devices (not shown). An emulation device can be one or more devices configured to emulate one or more or all of the functions described herein. For example, an emulation device can be used to test other devices and / or simulate network and / or WTRU functions.

[0094] Simulation devices can be designed to perform one or more tests on other devices in laboratory and / or carrier network environments. For example, one or more simulation devices may perform one or more functions when fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices within the communication network. One or more simulation devices may perform one or more functions when temporarily implemented / deployed as part of a wired and / or wireless communication network. Simulation devices may be directly coupled to another device for testing purposes and / or may use over-the-air wireless communication to perform tests.

[0095] One or more simulation devices may perform one or more (including all) functions when implemented / deployed without being part of a wired and / or wireless communication network. For example, simulation devices may be used to test scenarios in laboratory and / or undeployed (e.g., tested) wired and / or wireless communication networks to perform tests on one or more components. One or more simulation devices may be test rigs. Simulation devices may transmit and / or receive data using direct RF connections and / or wireless communication via RF circuitry systems (e.g., which may include one or more antennas).

[0096] Figure 1C This is a system diagram illustrating a set of exemplary interfaces for a system according to some embodiments. In some embodiments, an extended reality display device and its control electronics may be implemented. System 150 may be embodied as an apparatus including the various components described below and configured to perform one or more aspects described in this document. Examples of such apparatus include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 150 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 150 are distributed across multiple ICs and / or discrete components. In various embodiments, system 150 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 150 is configured to implement one or more aspects described in this document.

[0097] System 150 includes at least one processor 152 configured to execute instructions loaded therein to implement various aspects, such as those described in this document. Processor 152 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 150 includes at least one memory 154 (e.g., a volatile memory device and / or a non-volatile memory device). System 150 may include a storage device 158, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 158 may include internal storage, attached storage (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0098] System 150 includes an encoder / decoder module 156 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 156 may include its own processor and memory. The encoder / decoder module 156 represents a module that can be included in a device to perform encoding and / or decoding functions. It is well known that a device can include one or both encoding and decoding modules. Alternatively, the encoder / decoder module 156 may be implemented as a separate element of system 150, or it may be incorporated within processor 152 as a combination of hardware and software known to those skilled in the art.

[0099] Program code to be loaded onto processor 152 or encoder / decoder 156 to execute the various aspects described herein may be stored in storage device 158 and subsequently loaded onto memory 154 for execution by processor 152. According to various embodiments, one or more of processor 152, memory 154, storage device 158, and encoder / decoder module 156 may store one or more of various items during execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from equations, formulas, operations, and operational logic processing.

[0100] In some embodiments, the memory within processor 152 and / or encoder / decoder module 156 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., processor 152 or encoder / decoder module 152) is used for one or more of these functions. External memory may be memory 154 and / or storage device 158, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Various Video Coding, i.e., a new standard developed by JVET (Joint Video Experts Group)).

[0101] Inputs to the components of system 150 can be provided through various input devices, as indicated in box 172. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1C Other examples not shown include composite video.

[0102] In various embodiments, the input device of block 172 has corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting the signal band to a band), (ii) down-converting the selected signal, (iii) further band-limiting to a narrower band to select, for example, a signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired data packet stream. The RF section in various embodiments includes one or more elements performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners performing various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and filtering again to the desired frequency band. Various embodiments rearrange the order of the above (and other) components, remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as insert amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0103] Additionally, USB and / or HDMI terminals may include corresponding interface processors for connecting system 150 to other electronic devices via USB and / or HDMI connections. It should be understood that aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, within a separate input processing IC or within processor 152, as needed. Similarly, aspects of USB or HDMI interface processing may be implemented, as needed, within a separate interface IC or within processor 152. The demodulated, error-corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 152 and an encoder / decoder 156 operating in conjunction with memory and storage elements, to process the data stream as needed for presentation on an output device.

[0104] Various components of system 150 can be housed within an integrated housing. Within the integrated housing, the various components can be interconnected and transmit data between them using suitable connection means 174 (e.g., internal buses known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0105] System 150 includes a communication interface 160 that enables communication with other devices via a communication channel 162. The communication interface 160 may include, but is not limited to, a transceiver configured to send and receive data via the communication channel 162. The communication interface 160 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 162 may be implemented, for example, within a wired and / or wireless medium.

[0106] In various embodiments, data is streamed to or otherwise provided to system 150 using a wireless network, such as a Wi-Fi network, for example, IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). In these embodiments, the Wi-Fi signal is received via a communication channel 162 and a communication interface 160 adapted for Wi-Fi communication. The communication channel 162 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to system 150, delivering data via an HDMI connection to input box 172. Still other embodiments use an RF connection to input box 172 to provide streaming data to system 150. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0107] System 150 can provide output signals to various output devices, including display 176, speaker 178, and other peripheral devices 180. Display 176 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 176 can be used in televisions, tablets, laptops, mobile phones, or other devices. Display 176 can also be integrated with other components (e.g., in a smartphone) or standalone (e.g., an external monitor for a laptop computer). In various examples of embodiments, other peripheral devices 180 include one or more of a standalone digital video disc (or digital versatile disc) (DVR, for both terms), an optical disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 180 based on the output of system 150 to provide functionality. For example, an optical disc player performs the function of playing the output of system 150.

[0108] In various embodiments, signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that implement device-to-device control with or without user intervention are used to transmit control signals between system 150 and display 176, speaker 178, or other peripheral devices 180. Output devices may be communicatively coupled to system 150 via dedicated connections through corresponding interfaces 164, 166, and 168. Alternatively, output devices may be connected to system 150 via communication interface 160 using communication channel 162. Display 176 and speaker 178 may be integrated into a single unit along with other components of system 150 in an electronic device, such as a television. In various embodiments, display interface 164 includes a display driver, such as a timing controller (TCon) chip.

[0109] For example, if the RF input section 172 is part of a separate set-top box, the display 176 and speaker 178 can alternatively be separate from one or more of the other components. In various embodiments where the display 176 and speaker 178 are external components, the output signal can be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0110] System 150 may include one or more sensor devices 168. Examples of sensor devices that may be used include one or more GPS sensors, gyroscope sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors can be used to determine information such as the user's position and orientation. Where system 150 is used as a control module (such as control modules 124, 132) for an extended reality display, the user's position and orientation can be used to determine how to render image data so that the user perceives the correct portion of a virtual object or scene from the correct perspective. In the case of a head-mounted display device, the position and orientation of the device itself can be used to determine the user's position and orientation for rendering virtual content. In the case of other display devices (such as telephones, tablets, computer monitors, or televisions), other inputs can be used to determine the user's position and orientation for rendering content. For example, a user can select and / or adjust the desired viewpoint and / or viewing direction by using a touchscreen, keypad or keyboard, trackball, joystick, or other inputs. Where the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation can be selected and / or adjusted based on the movement of the display device for rendering content.

[0111] The embodiments can be executed by processor 152 or by computer software implemented by hardware or a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, memory 154 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 152 can be of any type suitable for the technical environment and can encompass one or more microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0112] Block-based video coding Like HEVC, VVC is built within a block-based hybrid video coding framework. Figure 2A A block diagram of a block-based hybrid video coding system 200 is given. Variations of this encoder 200 are envisioned, but for clarity, the encoder 200 is described below without describing all anticipated variations.

[0113] Before being encoded, the video sequence may undergo pre-coding processing 204, such as applying color transformations to the input color image (e.g., a conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing remapping of the input image components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization with one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0114] The input video signal 202, including the image to be encoded, is partitioned 206 and processed block by block, for example, in units of CUs. Different CUs can have different sizes. In VTM-1.0, CUs can be up to 128x128 pixels. However, unlike HEVC, which partitions blocks solely based on quadtrees, in VTM-1.0, coding tree units (CTUs) are divided into CUs to adapt to different local features based on quadtrees / binary / tritrees. Furthermore, the concept of multiple partitioning unit types in HEVC is removed, so in VVC-1.0, there is no longer a separation between CUs, prediction units (PUs), and transform units (TUs); instead, each CU is always used as a basic unit for both prediction and transform, without further partitioning. In the multi-type tree structure, CTUs are first partitioned using a quadtree structure. Then, each quadtree leaf node can be further partitioned using binary and ternary tree structures. Different partitioning types can be used, such as quadtree partitioning, vertical binary partitioning, horizontal binary partitioning, vertical ternary partitioning, and horizontal ternary partitioning.

[0115] exist Figure 2AIn the encoder, spatial prediction 208 and / or temporal prediction 210 can be performed. Spatial prediction (or "intra-frame prediction") uses pixels from samples of already encoded neighboring blocks (called reference blocks) (located in the same video picture / slice) to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also known as "inter-frame prediction" or "motion-compensated prediction") uses reconstructed pixels from already encoded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. A temporal prediction signal for a given CU can be signaled by one or more motion vectors (MVs), which indicate the amount and direction of motion between the current CU and its temporal reference. Furthermore, if multiple reference pictures are supported, a reference picture index can be sent additionally, which is used to identify which reference picture in the reference picture repository 212 the temporal prediction signal originates from.

[0116] The mode decision block 214 in the encoder selects the optimal prediction mode, for example, based on a rate-distortion optimization method. This selection can be made after spatial and / or temporal prediction. Intra-frame / inter-frame decisions can be indicated, for example, by a prediction mode flag. The prediction block is subtracted from the current video block 216 to generate the prediction residual. The prediction residual is decorrelated using transform 218 and quantization 220. (For some blocks, the encoder can bypass both transform and quantization, in which case the residual can be directly encoded without applying the transform or quantization process.) The quantized residual coefficients are inversely quantized 222 and inversely transformed 224 to form the reconstructed residual, which is then added back to the prediction block 226 to form the reconstructed signal of the CU. Further in the in-loop filtering, such as deblocking / SAO (sample adaptive offset) filtering, 228 can be applied to the reconstructed CU to reduce coding artifacts, which is then placed in the reference picture repository 212 and used to encode future video blocks. In order to form the output video bitstream 230, the coding mode (intra-frame or inter-frame), prediction mode information, motion information, and quantized residual coefficients are sent to the entropy coding unit (108) for further compression and packing to form the bitstream.

[0117] Figure 2B A block diagram of a block-based video decoder 250 is given. In decoder 250, the bitstream is decoded by decoder elements, as described below. Video decoder 250 typically performs operations similar to... Figure 2A The encoding process described herein is the opposite of the decoding process. Encoder 200 typically also performs video decoding as part of the encoded video data.

[0118] Specifically, the decoder's input includes a video bitstream 252, which can be generated by the video encoder 200. The video bitstream 252 is first unpacked and entropy decoded at the entropy decoding unit 254 to obtain transform coefficients, motion vectors, and other encoded information. Picture partitioning information indicates how the pictures are partitioned. Therefore, the decoder can partition 256 pictures based on the decoded picture partitioning information. The encoding mode and prediction information are sent to the spatial prediction unit 258 (if intra-frame coding) or the temporal prediction unit 260 (if inter-frame coding) to form prediction blocks. Residual transform coefficients are sent to the inverse quantization unit 262 and the inverse transform unit 264 to reconstruct the residual blocks. The prediction blocks and residual blocks are then added together at 266 to generate reconstructed blocks. The reconstructed blocks can be further filtered through the in-loop filter 268 before being stored in the reference picture repository 270 for predicting future video blocks.

[0119] The decoded image 272 can further undergo post-decoding processing 274, such as inverse color transformation (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which performs the inverse of the remapping process performed in pre-encoding processing 204. Post-decoding processing can utilize metadata derived in pre-encoding processing and signaled in the bitstream. The decoded video can be sent to display device 276. Display device 276 can be a device independent of decoder 250, or decoder 250 and display device 276 can be components of the same device.

[0120] The various methods and other aspects described in this disclosure can be used to modify modules of the video encoder 200 or decoder 250. Furthermore, the systems and methods disclosed herein are not limited to VVC or HEVC and can be applied to, for example, other standards and recommendations (whether pre-existing or future developments), and any extensions to such standards and recommendations (including VVC and HEVC). Unless otherwise stated or technically excluded, the aspects described in this disclosure can be used alone or in combination.

[0121] Figure 3A This is a schematic side view illustrating an exemplary waveguide display that can be used with extended reality (XR) applications according to some embodiments. An image is projected by an image generator 302. The image generator 302 can use one or more of a variety of techniques for projecting images. For example, the image generator 302 can be a laser beam scanning (LBS) projector, a liquid crystal display (LCD), a light-emitting diode (LED) display (including organic LED (OLED) or micro LED (µLED) displays), a digital light processor (DLP), a liquid crystal on silicon (LCoS) display, or other types of image generators or light engines.

[0122] The light representing the image 312 generated by the image generator 302 is coupled into the waveguide 304 by a diffraction input coupler 306. The input coupler 306 diffracts the light representing the image 312 into one or more diffraction orders. For example, ray 308, which is part of the bottom of the image, is diffracted by the input coupler 306, and one of the diffraction orders 310 (e.g., second order) is at an angle that allows it to propagate through the waveguide 304 via total internal reflection. The image generator 302 displays the image as instructed by a control module 324, which operates to render image data, video data, point cloud data, or other displayable data.

[0123] At least a portion of the light 310, already coupled into waveguide 304 by diffraction input coupler 306, is coupled out of the waveguide by diffraction output coupler 314. At least some of the light coupled out of waveguide 304 replicates the incident angle of the light coupled into the waveguide. For example, in the illustration, output coupled rays 316a, 316b, and 316c replicate the angle of input coupled ray 308. Because the light leaving the output coupler replicates the direction of the light entering the input coupler, the waveguide essentially replicates the original image 312. The user's eye 318 can focus on the replicated image.

[0124] exist Figure 3A In the example, output coupler 314 outputs only a portion of the coupled light on each reflection, allowing a single input beam (such as beam 308) to generate multiple parallel output beams (such as beams 316a, 316b, and 316c). In this way, even if the eye is not perfectly aligned with the center of the output coupler, at least some of the light from each part of the image may reach the user's eye. For example, if eye 318 moves downwards, beam 316c may enter the eye even if beams 316a and 316b do not, so the user can still perceive the bottom of image 312 despite the positional shift. Therefore, output coupler 314 partially functions as an outgoing pupil dilator in the vertical direction. The waveguide may also include one or more additional outgoing pupil dilators (…). Figure 3A (Not shown in the image) to expand the exit pupil in the horizontal direction.

[0125] In some embodiments, waveguide 304 is at least partially transparent relative to light originating from outside the waveguide display. For example, at least some of the light 320 from a real-world object (such as object 322) passes through waveguide 304, allowing the user to see the real-world object when using the waveguide display. Multiple diffraction orders and thus multiple images will exist as the light 320 from the real-world object also passes through diffraction grating 314. To minimize the visibility of multiple images, it is desirable that the zeroth-order diffraction (not deflected by 314) has high diffraction efficiency for both light 320 and the zeroth order, while higher diffraction orders have lower energy. Therefore, in addition to expanding and output coupling the virtual image, output coupler 314 is preferably configured to allow the zeroth order of the real image to pass through. In such embodiments, the image displayed by the waveguide display may appear to be superimposed on the real world.

[0126] Figure 3B This is a schematic side view illustrating exemplary alternative display types that can be used with extended reality applications according to some embodiments. In the XR head-mounted display device 330, a control module 332 controls a display 334 (which may be an LCD) to display images. The head-mounted display includes a partially reflective surface 336 that reflects (and in some embodiments, both reflects and focuses) the image displayed on the LCD to make the image visible to the user. The partially reflective surface 336 also allows at least some external light to pass through, thereby allowing the user to see their surroundings.

[0127] Figure 3C This is a schematic side view illustrating exemplary alternative display types that can be used with extended reality applications according to some embodiments. In the XR head-mounted display device 340, a control module 342 controls a display 344 (which may be an LCD) to display an image. The image is focused by one or more lenses of a display optics 346 to make the image visible to the user. Figure 3C In this example, external light does not reach the user's eyes directly. However, in some such embodiments, an external camera 348 may be used to capture images of the external environment and display such images on a display 344 along with any virtual content that may also be displayed.

[0128] The embodiments described herein are not limited to any particular type or structure of XR display device.

[0129] As discussed in this article, avatar instances can be encoded using human-readable formats, such as avatars encoded for 3D scene representations.

[0130] Many formats exist for encoding 3D content, including mesh rigs, such as deformable 3D objects using weights. Formats like FBX, USD, or glTF can be found. However, these formats are not known to be specifically designed for avatar data. Therefore, applications must interpret the data to determine whether it can be correlated with avatars in any of the currently available formats, and how to parse and utilize the data. Currently, there is no known open format specifically for avatars.

[0131] Recently, MPEG proposed the MPEG_node_avatar extension for glTF ( [SD] Regarding potential improvements to CDAM2 23090-14 document—Update to Appendix H of the Reference Avatar MPEG Conference #143, Input Document m63997 (July 2023) (“ m63997 The extension defines nodes as avatars. With this extension, applications can learn, for example: (1) which nodes in the file represent avatars (via the "isAvatar" attribute), (2) what type of avatar they represent (via the "type" attribute), and (3) the components of an avatar (via the "mappings" attribute). While this information is helpful, some avatar-related information is still missing, such as the semantic definition of the deformable target, parametric textures, or different levels of detail. Furthermore, because glTF is specifically designed for hardware optimization, some information is sometimes lost. For example, glTF only processes triangles: according to information available from the registry... <dot>chronos <dot>org / glTF / specs / 2.0 / glTF-2.0 <dot>PDF glTF 2.0 Specification (Khronos 3D Format Working Group (2021)) glTF 2.0 The glTF cannot store a grid of quadrilaterals. The introduction of the avatar concept in MPEG also has limitations in streaming capabilities. Avatars are constrained to be embedded in scene description type formats, rather than having the self-contained, streaming-optimized, and interoperable formats that the MPEG standard relies on.

[0132] This article introduces a new file format called Avatar JSON Exchange File (AJIF). This format uses the ".ajif" extension and stores information about avatar instances. Encoding and Decoding Figure 4 This is a process diagram illustrating an exemplary top-level encoding and decoding process according to some embodiments. In exemplary process 400, avatar data 402 may be encoded 404 and stored in AJIF format 406. The AJIF-formatted data 406 may be decoded by decoder 408 to obtain avatar data 410.

[0133] Avatar data is a set of formal information supported by any computer, such as strings, images, or tensors. Avatar data can be formatted to conform to certain specifications. For example, in some embodiments, all integers can be 64-bit signed integers. Therefore, if the original data does not fully conform to this format (such as 32-bit unsigned integers), some processing can be performed to format and / or construct the avatar data. Depending on the options of some embodiments, encoding may preserve all avatar data without loss, lose some avatar data, or lose some floating-point precision.

[0134] Avatar represents data In some embodiments, integer values ​​are 64-bit signed integers, floating-point values ​​are IEEE 754 64-bit floating-point numbers, and string content is encoded using UTF-8. Images are RGBA with floating-point values. In some embodiments, the axis order of 2D coordinates is (always) (X, Y) or (U, V), and the axis order of 3D coordinates is (always) (X, Y, Z).

[0135] For some implementations, the unit vector format for each axis is the same regarding the orientation of 2D and 3D coordinates, for example, (X, Y) or (U, V) for 2D and (X, Y, Z) for 3D; and the direction of the axis should be indicated. AJIF uses a right-handed coordinate system. AJIF defines +Y as up, +Z as front, and -X as right. The front of an AJIF asset faces +Z.

[0136] The AJIF format can store multiple avatar instances, each representing the same avatar but with different levels of detail (LOD). For some instances, each level of detail of an avatar instance may contain one or more of the following data: high-level avatar information; a list of the same avatar with different levels of detail; geometry; skeletal skinning; and controllers.

[0137] Advanced avatar information may include the following optional fields: Name, Identifier, Age, and Gender.

[0138] For a list of different levels of detail for the same avatar, each level may include the following optional fields: Mesh, Map, Skeleton, and Controller. The Mesh field may include one or more geometries, as defined below. The Map field may include a list of strings with associated geometries to identify body parts. The Skeleton field may include one or more skeleton skins, as defined below. The Controller field may include one or more controllers, as defined below.

[0139] The geometry can include the following items: vertices, faces, normals, blend shapes, and textures. Vertices can include a list of 3D floating-point coordinates corresponding to vertex coordinates. Faces can include a list of tuples, where each value in the tuple is a vertex index. Tuples can have 3 to 256 values. The normal field can include: a list of 3D floating-point coordinates corresponding to vertex normal coordinates; and / or a list of tuples, where each value in the tuple is a normal index. Tuples can have 3 to 256 values. The blend shape field can include a list of blend shapes, where each blend shape includes the following items: displacement, name, code syntax (CS), and side. Displacement can include a list of 3D floating-point displacements, each displacement associated with a vertex. The name field can be a string representing the name of the blend shape. The CS field can be a string with the corresponding code syntax ("facs:AU1"). The side field can be a string with a positional hint (such as "left", "top", etc.).

[0140] The texture field can include a list of texture data, each consisting of a type, uvs, face, image, and basis. The type can include texture types such as albedo, normal, roughness, metallicity, specular, height, opacity, ambient occlusion, refraction, emissivity, and / or gloss. The uvs term can include a list of 2D floating-point coordinates corresponding to UV coordinates. The face can be a list of tuples, where each value in the tuple is a uv index. Tuples can have 3 to 256 values. The image can be a texture image or a floating-point tensor [image width, image height, 4]. The basis can include a texture displacement map, such as a list of image displacements, like a floating-point tensor [image width, image height, image channel]. Each displacement can be associated with a string describing that displacement.

[0141] A skeleton skin can include the following items: root joint, joint list, and skin weight list. The skin weight list can include: the mesh used by the avatar, and a list of tuples referencing the joints in the joint list, the vertices in the mesh, and scalar weights between 0 and 1. Each joint can include: a map, sub-joints, a transformation, and a 4x4 floating-point matrix. The map can be a string used to identify the joint's body position / semantics. The transformation can perform scaling, then rotation, then translation. The 4x4 floating-point matrix can be used for reverse binding.

[0142] A controller can include mappings, inputs, and channels. Mappings can be strings with corresponding code syntax. Inputs can be a list of incrementing input weights. Channels can be a list of controller channels. Each controller channel can include: an interpolation method, the part to be animated, and a list of output value sets. The interpolation method can be "LINEAR," "STEP," or "CUBICSPLINE." The part to be animated can include: transformations of one or more geometries; transformations of one or more skeletal joints; blend shape weights of one or more geometries; texture displacement weights of one or more textures; weights from another controller; and a list of output value sets, where each value set corresponds to a value from an input weight. The number and meaning of values ​​in each value set must be consistent with the part to be animated and the interpolation method.

[0143] AJIF structure The AJIF structure has the contents shown in Table 1 below.

[0144]

[0145] Table 1. In Table 1, all items / content except the last two ("metadata" and "asset") are collections of items. Each collection is a list of items that can be referenced by the item's index in the list, where the first index is zero. For example, if " nodes "If the set contains three items, then the reference index of the first item is 0; the reference index of the second item is 1; and the reference index of the third item is 2."

[0146] Single file mode Figure 5 This illustrates an exemplary .ajif file mapping according to some embodiments. For some embodiments, the default encoding mode is based on a main JSON file (the .ajif file) and one or more other files containing (ultimately) additional data such as vertices or images. Using base64 encoding, all additional data can be placed within the .ajif file. The .ajif file can reference portions of a file to retrieve data. The referenced file can be the .ajif file itself, such as... Figure 5 As shown. Therefore, additional data can be placed after the JSON content in the .ajif file. Using this structure, the JSON data is correctly encoded, and the end of the JSON data can be determined using the fact that the root of the JSON data is a JSON object. Figure 5 The "Unused" field can be anything (such as zero-padding or non-AJIF data), which AJIF does not use.

[0147] All content in a .ajif file can be encoded, and additional information can be included. In this mode, the beginning of the .ajif file ( Figure 5 The "JSON" field (502) in the code is the main JSON content, followed by a byte with a value of zero. This terminating byte ( Figure 5 The "0" in the JSON field defines the end of the JSON portion, which does not need to be valid JSON data. Eight bytes can follow the byte with a value of zero. Figure 5 The "extra data size" (in the context of the binary data 506) includes a 64-bit little-endian unsigned integer 504 that defines the size of the "extra data" binary data 506. In some embodiments, the exact number of extra data bytes may be cached in memory. In some embodiments, one or more unused bytes 508 may exist.

[0148] AJIF Features and Benefits The AJIF format is dedicated exclusively to encoding a single avatar. Unlike general scene description formats, it is not necessary to parse a file to determine that the file is for avatars. For many general formats, parsing is mandatory to determine whether the file contains one or more avatars. For some embodiments of AJIF, there is one and only one avatar, so no parsing is required.

[0149] AJIF provides avatar-related information that may not be found in general formats and most avatar-specific formats. AJIF is not limited to human avatars. In addition, a part of the body or the entire body can be encoded. AJIF is an open format.

[0150] Encoding is not constrained by hardware considerations. The primary purpose of AJIF is not limited to renderers, but can be used by all computer graphics applications. Since the avatar representation is based on a general avatar data representation, the decoded avatar representation can be converted for internal application use.

[0151] Data can be merged into a single file in multiple ways or dispersed across multiple files. This degree of freedom can improve user experience and save resources depending on the use case. For example, all extra data related to lod can be placed in a separate file. Thus, the application can only load the files related to the required lod.

[0152] RawData Definition RawData items may include low-level content such as tensors or images, as shown in Table 2.

[0153]

[0154] Table 2. uri Definition The "uri" attribute contains or references binary content. The use and interpretation of this content depends on other attributes. If the "uri" attribute starts with "data:application / ajif-buffer;base64,", the subsequent characters are the base64 encoding of the data. For example, the uri "data:application / ajif-buffer;base64,TWFueSBoYW5kcyBtYWtlIGxpZ2h0IHdvcmsu" contains the base64-encoded data "TWFueSBoYW5kcyBtYWtlIGxpZ2h0IHdvcmsu". By interpreting this data field as a UTF-8 string, the data field can be decoded as "Many hands make light work.".

[0155] If "uri" begins with "http:" or "https:", then "uri" is interpreted as a URL, and data must be downloaded from that URL. If the "uri" attribute does not begin with "data:", "http:", or "https:", then "uri" contains a file path relative to the AJIF path. The file contains the data. For other "uri" values, decoding is undefined.

[0156] The "urii" attribute can be the same as the "uri" attribute in the glTF buffer, and can follow the specification of Berners-Lee, T., etc., Uniform Resource Identifier (URI): General Syntax, STD 66, RFC 3986, DOI 10.17487 / RFC3986 (January 2005), available in the rfc-editor. <dot>Obtainable from org / info / rfc3986 ("RFC 3986"). For some embodiments, the encoding is a subset of RFC 3986.

[0157] Definitions of offset and byteLength (byte length) If the "offset" and "byteLength" properties do not exist, all binary content provided by the "uri" property will be used.

[0158] If the "byteLength" attribute exists but the "offset" attribute does not exist, only the first "byteLength" bytes of the binary content provided by the "uri" attribute are used. For example, if "uri" provides 3 bytes (23, 17, 240), "byteLength" is 2, and there is no "offset", then the content to be considered is (23, 17).

[0159] If the "offset" attribute exists but the "byteLength" attribute does not exist, the first "offset" bytes are ignored in the binary content provided by the "uri" attribute. For example, if the "uri" provides 3 bytes (23, 17, 240), the "offset" is 1, and there is no "byteLength", then the content to be considered is (17, 240).

[0160] If both the "offset" and "byteLength" attributes are present, then only the "byteLength" bytes starting from "offset" are considered in the binary content provided by the "uri" attribute. For example, if "uri" provides 3 bytes (23, 17, 240), "offset" is 1 and "byteLength" is 1, then the content to be considered is (17).

[0161] Dense Tensor Coding If "type" is "dense", then the item contains a tensor (or a multidimensional array). In this case, all values ​​of the tensor are used to represent the tensor.

[0162] The "dims" attribute defines the tensor dimensions from left to right. The total number of values ​​in the tensor is the product of these dimensions. For example, if "dims" is [2, 7, 4], then the first dimension is 2, the second dimension is 7, and the third dimension is 4. In this example, the total number of values ​​is 2. 7 4 = 56.

[0163] The "dtype" attribute defines the type of the values ​​in the tensor. If "dtype" is "i8", the values ​​in the tensor are 8-bit signed integers. If "dtype" is "u8", the values ​​are 8-bit unsigned integers. If "dtype" is "i16", the values ​​are 16-bit signed little-endian integers. If "dtype" is "u16", the values ​​are 16-bit unsigned little-endian integers. If "dtype" is "i32", the values ​​are 32-bit signed little-endian integers. If "dtype" is "u32", the values ​​are 32-bit unsigned little-endian integers. If "dtype" is "i64", the values ​​are 64-bit signed little-endian integers. If "dtype" is "u64", the values ​​are 64-bit unsigned little-endian integers. If "dtype" is "f16", the values ​​are IEEE 754 16-bit floating-point numbers. If "dtype" is "f32", the value is an IEEE 754 32-bit floating-point number. If "dtype" is "f64", the value is an IEEE 754 64-bit floating-point number.

[0164] The tensor value is decoded from the binary content obtained from "uri", "offset", and "byteLength". The values ​​in the binary content are ordered in row-major order. For example, if the binary content contains bytes [3, 17, 23, 1] and "dtype" is "u16", the decoded value is [3 + (17...). 256), 23 + (1 256)]= [4355,279。

[0165] Sparse tensor coding If "type" is "sparse", then the item contains a tensor (or a multidimensional array). In this case, the tensor is represented as a list of 2-tuples (index, value), where the index is the position of the value within the tensor. The index is the 1D index of a cell within the tensor. For tensors with more than one dimension, knowing that the values ​​are sorted row-majorly, the 1D index is obtained by "flattening" the tensor. For example, if the tensor has dimensions [a, b, c], then the 1D index of the 3D index (x, y, z) is (x...). b + y) c + z.

[0166] The "count" property defines the number of 2-tuples (index, value). This property is optional and can be used to know the size of a list without decoding the binary content.

[0167] The "dims" attribute defines the tensor dimensions from left to right. The total number of values ​​in the tensor is the product of these dimensions. For example, if "dims" is [2, 7, 4], then the first dimension is 2, the second dimension is 7, and the third dimension is 4. The total number of values ​​in this example is 2. 7 4 = 56.

[0168] The "dtype" attribute defines the type of the values ​​in the tensor. If "dtype" is "i8", the values ​​in the tensor are 8-bit signed integers. If "dtype" is "u8", the values ​​are 8-bit unsigned integers. If "dtype" is "i16", the values ​​are 16-bit signed little-endian integers. If "dtype" is "u16", the values ​​are 16-bit unsigned little-endian integers. If "dtype" is "i32", the values ​​are 32-bit signed little-endian integers. If "dtype" is "u32", the values ​​are 32-bit unsigned little-endian integers. If "dtype" is "i64", the values ​​are 64-bit signed little-endian integers. If "dtype" is "u64", the values ​​are 64-bit unsigned little-endian integers. If "dtype" is "f16", the values ​​are IEEE 754 16-bit floating-point numbers. If "dtype" is "f32", the value is an IEEE 754 32-bit floating-point number. If "dtype" is "f64", the value is an IEEE 754 64-bit floating-point number.

[0169] The "itype" property defines the type of the index in the 2-tuple list. If "itype" is "i8", the value in the tensor is an 8-bit signed integer. If "itype" is "u8", the value in the tensor is an 8-bit unsigned integer. If "itype" is "i16", the value in the tensor is a 16-bit signed little-endian integer. If "itype" is "u16", the value in the tensor is a 16-bit unsigned little-endian integer. If "itype" is "i32", the value in the tensor is a 32-bit signed little-endian integer. If "itype" is "u32", the value in the tensor is a 32-bit unsigned little-endian integer. If "itype" is "i64", the value in the tensor is a 64-bit signed little-endian integer. If "itype" is "u64", the value in the tensor is a 64-bit unsigned little-endian integer.

[0170] The 2-tuple (index, value) of the sparse tensor is decoded from the binary content obtained by "uri", "offset", and "byteLength". "count" tuples are decoded one after another, for example (index1, value1, index2, value2, ...). For example, if the binary content contains bytes [29, 3, 17, 7, 23, 1], "dtype" is "u16" and "itype" is "i8", then the decoded tuple is [(29, 4355), (7, 279)].

[0171] Image encoding If "type" is "image", then the item contains an image, such as a 2D array of pixel values ​​in a specific color space.

[0172] If "mimeType" does not exist, then the "dims", "ctype", and "dtype" properties must exist. The "dims" property is the image size [width, height]. "ctype" defines the pixel format. If "ctype" is "RGB", there are red, green, and blue pixels. If "ctype" is "RGBA", there are red, green, blue, and alpha pixels.

[0173] The "dtype" attribute defines the type of the pixel value. If "dtype" is "i8", the value in the tensor is an 8-bit signed integer. If "dtype" is "u8", the value in the tensor is an 8-bit unsigned integer. If "dtype" is "i16", the value in the tensor is a 16-bit signed little-endian integer. If "dtype" is "u16", the value in the tensor is a 16-bit unsigned little-endian integer. If "dtype" is "i32", the value in the tensor is a 32-bit signed little-endian integer. If "dtype" is "u32", the value in the tensor is a 32-bit unsigned little-endian integer. If "dtype" is "i64", the value in the tensor is a 64-bit signed little-endian integer. If "dtype" is "u64", the value in the tensor is a 64-bit unsigned little-endian integer. If "dtype" is "f16", the value is an IEEE 754 16-bit floating-point number. If "dtype" is "f32", the value is an IEEE 754 32-bit floating-point number. If "dtype" is "f64", the value is an IEEE 754 64-bit floating-point number.

[0174] For example, if "ctype" is "RGB" and "dtype" is "f32", then each pixel has three 32-bit floating-point values.

[0175] The binary content obtained from "uri", "offset", and "byteLength" contains the pixel values ​​of the image. The values ​​in the binary content are sorted in row-major order.

[0176] For example, if the binary content contains the bytes [3, 17, 23, 1, 3, 7, 9, 8, 2, 7, 5, 3], "dims" is (2, 2), "ctype" is "RGB", and "dtype" is "u8", then the decoded values ​​are as follows: Pixel (0,0) has red=3, green=17, and blue=23. Pixel (0,1) has red=1, green=3, and blue=7. Pixel (1,0) has red=9, green=8, and blue=2. Pixel (1,1) has red=7, green=5, and blue=3.

[0177] If "mimeType" exists, the binary content obtained from "uri", "offset", and "byteLength" contains an image encoded with the corresponding mimeType. For example, if "mimeType" is "image / png", the binary content is an image encoded using the PNG file format.

[0178] For formats that encode the dimensions of an image, the "dims" attribute is not required. Otherwise, "dims" contains the dimensions of the image. For formats that encode the pixel format of an image, the "ctype" attribute is not required. Otherwise, "ctype" contains the pixel format. For formats that encode the type of pixel values, the "dtype" attribute is not required. Otherwise, "dtype" contains the type of pixel values.

[0179] Tuple encoding If "type" is "tuples", this item contains a list of tuples. Each tuple has values ​​of the same type, can have different numbers of values, and can have zero or more values. If the "count" property is present, the "count" property contains the number of tuples in the list.

[0180] The "dtype" attribute defines the type of the values ​​in the tuple. If "dtype" is "i8", the values ​​in the tensor are 8-bit signed integers. If "dtype" is "u8", the values ​​in the tensor are 8-bit unsigned integers. If "dtype" is "i16", the values ​​in the tensor are 16-bit signed little-endian integers. If "dtype" is "u16", the values ​​in the tensor are 16-bit unsigned little-endian integers. If "dtype" is "i32", the values ​​in the tensor are 32-bit signed little-endian integers. If "dtype" is "u32", the values ​​in the tensor are 32-bit unsigned little-endian integers. If "dtype" is "i64", the values ​​in the tensor are 64-bit signed little-endian integers. If "dtype" is "u64", the values ​​in the tensor are 64-bit unsigned little-endian integers. If "dtype" is "f16", the values ​​are IEEE 754 16-bit floating-point numbers. If "dtype" is "f32", the value is an IEEE 754 32-bit floating-point number. If "dtype" is "f64", the value is an IEEE 754 64-bit floating-point number.

[0181] The "itype" property defines the type of the index in the tuple. If "itype" is "i8": the value in the tensor is an 8-bit signed integer. If "itype" is "u8": the value in the tensor is an 8-bit unsigned integer. If "itype" is "i16": the value in the tensor is a 16-bit signed little-endian integer. If "itype" is "u16": the value in the tensor is a 16-bit unsigned little-endian integer. If "itype" is "i32": the value in the tensor is a 32-bit signed little-endian integer. If "itype" is "u32": the value in the tensor is a 32-bit unsigned little-endian integer. If "itype" is "i64": the value in the tensor is a 64-bit signed little-endian integer. If "itype" is "u64": the value in the tensor is a 64-bit unsigned little-endian integer.

[0182] The binary content obtained from "uri", "offset", and "byteLength" contains tuples. Each tuple is encoded as (tuple size, tuple value 1, tuple value 2, ...). For example, if the binary content contains bytes [1, 3, 17, 2, 23, 1, 18, 29], "dtype" is "u16", and "itype" is "u8", then there are two tuples: (4355) and (279, 7442).

[0183] Geometry definition The Geometry object defines a grid with associated properties, as shown in Table 3.

[0184]

[0185] Table 3. The "vertices" property references a tensor V, which has the vertices of a grid, where V(i) = (x, y, z) are the X, Y, Z coordinates of the vertex at index i.

[0186] The "normals" property references a tensor N, which has vertex normals of the grid, where N(i) = (x, y, z) are the X, Y, Z coordinates of the normal at index i.

[0187] The "uv" attribute references a list of tensors U(j) with vertices of a grid, where U(j)(i) = (u, v) are the U and V coordinates of the uv at index i in the set of uvs at index j.

[0188] The "faces" property contains the index of each face in the grid (see below). Faces definition).

[0189] The "blendshapes" property contains a list of blend shapes (or deformation targets). See below. Blendshapes definition.

[0190] The "textures" property contains a list of textures. See below. Textures definition.

[0191] Faces definition The Faces object defines the index of each face in the mesh, as shown in Table 4.

[0192]

[0193] Table 4. Each property of the Faces object references an item (or list of items) in the "data" collection, representing the index of each face in the mesh. Each of these items must have the same layout; for example, they must all conform to the same topology. If all faces have the same number of edges, each item must be a tensor of dimension [number of faces, number of edges], or a list of tuples with the same number of tuples, all of which have three values. If faces are allowed to have different numbers of edges, each item must be a list of tuples with the same number of tuples, each of which has the same number of values. Each item has the same layout but not necessarily the same values.

[0194] If it exists, the "vertices" property references an entry with the vertex indices for each face. If the "vertices" property does not exist, the default vertices for each face are sorted in ascending order, for example, (0, 1, 2), (3, 4, 5, 6), (7, 8, 9, ...). If it exists, the "normals" property references the item with the normal index for each face. If the "normals" property does not exist, the normal index is equal to the vertex index.

[0195] If present, the "uv" attribute contains a list of references to items with UV indices for each face. If the "uv" attribute does not exist, the UV indices are equal to the vertex indices.

[0196] Blendshape definition The Blendshape object contains data and related information about the blend shape (or mesh deformation or deformation target), as shown in Table 5.

[0197]

[0198] Table 5. The "displacements" property references a floating-point tensor D, which contains the displacement vector for each vertex of the geometry. For example, D(i) = (x, y, z) is the displacement of vertex i.

[0199] The "mapping" attribute contains a string indicating the corresponding code syntax. For example, "facs:AU1" could indicate that the blend shape is based on the FACS blend shape "AU1" ( FACS Quick Reference Melinda Ozel <dot>com, available at melindaozel <dot>com / facs-cheat-sheet / (Last visited January 6, 2024) FACS ())

[0200] The "side" property contains a string that provides a hint about the location of the blended shape. For example, "left" could indicate that the blended shape involves the left side of the geometry.

[0201] Texture definition The Texture object defines the texture and related data, as shown in Table 6.

[0202]

[0203] Table 6. The "type" attribute defines the texture type and therefore its purpose. The "type" attribute can be based on PBR texture types, such as "albedo" or "spectral".

[0204] The "uv" attribute references the UV coordinates in the geometry (in the parent). Geometric shapes (in the object's "uv" list) and the UV face index in the geometry (in the parent) Geometric shapes The reference is an index in the list of "uv" values ​​within the object's "faces" property. For example, if the "uv" property is 1, then the "uv" property references the second item in both the "uv" list within the geometry and the "uv" list within the geometry's "faces" property.

[0205] The "image" property references an image from the "data" collection. If there is no "base" property, the texture image is that image. If there is a "base" property, the texture image is parameterized and can be calculated as shown in Formula 1:

[0206] in: The image is provided by the "image" attribute.

[0207] The displacement is provided by the item at index i in the "base" attribute.

[0208] It is a floating-point value provided by the application.

[0209] It is a function that ensures that texture values ​​are within the range of valid color values.

[0210] Texture Displacement Definition The TextureDisplacement object contains data about texture displacement (or texture target) and related data, as shown in Table 7.

[0211]

[0212] Table 7. The "displacements" property references a floating-point tensor in the "data" collection. The dimensions of this tensor must be the same as those of the parent collection. Texture The "image" attribute in the file provides consistent dimensions for the image. For example, if the image has a size of (width, height) and has RGB channels, then the displacement tensor must be (width, height, 3).

[0213] The "mapping" attribute describes the texture displacement.

[0214] Node definition Node objects contain data used to organize, merge, describe, and transform components, as shown in Table 8.

[0215]

[0216] Table 8. The "mapping" attribute indicates body parts, such as "full_body / upper_body / right_arm".

[0217] The "geometry" property references the geometry in the "geometries" collection.

[0218] The "children" property is a list of indexes that define the child nodes of this node. As long as there are no circular dependencies, the node's child nodes are correct.

[0219] The "scale" attribute defines the non-uniform scaling of the node, as given by scaling factors along the x, y, and z axes. If this attribute exists, the "matrix" attribute may not exist.

[0220] The "rotation" property defines a unit quaternion rotation of the nodes in the order (x, y, z, w), where w is a scalar. If this property exists, the "matrix" property may not exist.

[0221] The "translation" property defines the translation of a node along the x, y, and z axes. If this property exists, the "matrix" property may not exist.

[0222] If "scale", "rotation", or "translation" is present, they are applied in the following order: first scale, then rotate, and finally translate.

[0223] The "matrix" attribute defines a floating-point 4x4 transformation matrix stored in column-major order. If this attribute exists, "scale", "rotation", and "translation" may not exist.

[0224] Nodes, Mesh and Geometry A mesh within a node can have all the attributes found in a merge of the node's and its descendants' attributes. This is only possible if the node or at least one of its descendants references the geometry. Attribute merging is the concatenation of all attributes found during a depth-first traversal of the node hierarchy.

[0225] Skeleton definition The Skeleton object defines the skeleton, as shown in Table 9.

[0226]

[0227] Table 9. The "mapping" attribute defines the purpose of the skeleton.

[0228] The "root" property references a node in the "nodes" collection and defines the skeletal root. This node must be the nearest common root of the joint hierarchy, or its direct or indirect parent node.

[0229] The "joints" property is a list of references to nodes in the "nodes" collection. These nodes are used as joints in this skeleton.

[0230] "inverseBindMatrices" refers to the floating-point tensor B in the "data" collection. The tensor dimensions must match the "joints" attribute, for example, [joint count, 4, 4].

[0231] The "skinnings" attribute is Skinnings A list of objects (see below).

[0232] Skinning Definition Skinning The skin joint weights in the object definition geometry are shown in Table 10.

[0233]

[0234] Table 10. The "mapping" attribute defines the skinning semantics on the mesh.

[0235] The "mesh" attribute references a node in "nodes". This node, or at least one of its descendants, must reference a geometry. The resulting mesh is a concatenation of all geometries found during a depth-first traversal of the node hierarchy.

[0236] The "weights" property references a floating-point tensor in the "data" collection. This tensor has two dimensions: the first dimension is the number of vertices in the mesh provided by the "mesh" property, and the second dimension is the number of joints in the parent bone.

[0237] Multiple skinning interpolation Linear blended skin is defined as shown in Formula 2:

[0238] in This is the obtained vertex position. Component These are the scalar weights defined in the "weights" attribute of Table 10. Components The static / bound pose of the vertices is defined by the "mesh" attribute in Table 10. Components It's bone. The transformation matrix is ​​defined as "inverseBindMatrices" as mentioned above. Components Is with vertex The number of associated bones. The sum of the effects of all bones on a vertex should equal 1, as shown in Formula 3:

[0239] In the presence of multiple skinning weight matrices, Formula 4 can be derived:

[0240] in Therefore, linear blending skin becomes a linear combination of positions. In practice, each It comes from a static posture The displacement vectors are assumed to be linearly combined, as shown in the previous formula (Formula 4). Therefore, the attribute "skinnings" in Table 9 is the weight matrix. The list can be linearly combined as shown in Formula 5:

[0241] Each of them It is the weight matrix in the first formula The displacement vector resulting from the change, and It is the number of matrices to be combined.

[0242] This method facilitates the separation of animation using linear blending skinning and leverages the ability to blend different schemes into a single model, such as combining body deformation at joints with muscle deformation. For this purpose, scalar properties are listed in Table 10. Used to measure each weight matrix The effect is shown in Formula 6:

[0243] Controller definition The Controller object defines the components used to animate the avatar, as shown in Table 11.

[0244]

[0245] Table 11. The "mapping" attribute identifies the controller type or semantics.

[0246] The "input" attribute references a 1D tensor T with incrementing values, such that for any j > i, T[j] >= T[i]. The "input" attribute defines the input weights. If it does not exist, the input weights are [0, 1].

[0247] The "channels" property is a list of channels (see below).

[0248] The controller animates the avatars by applying all the transformations in the channel.

[0249] Channel definition Channel objects define data used to animate a part of an avatar or another controller, as shown in Table 12.

[0250]

[0251] Table 12. The channel targets an avatar or a component in another controller. For node scaling, "nodes" must reference the node, and "path" must point to "scale". In this case, the weight is 3, and the node scaling is animated by the channel. For node rotation, "nodes" must reference the node, and "path" must point to the "rotation" value. In this case, the weight is 4, and the node rotation is animated by the channel. For node translation, "nodes" must reference the node, and "path" must point to the "translation" value. In this case, the weight is 3, and the node translation is animated by the channel. For node matrices, "nodes" must reference the node, and "path" must point to "matrix". In this case, the weight is 16, and the node's matrix transformation is animated by the channel.

[0252] For the blend shape weights of nodes, "nodes" must reference nodes that reference the geometry or at least one descendant of the geometry. In this case, the weight size is the total number of blend shapes found in the mesh constructed through a depth-first traversal of the node hierarchy. The order of the weights, and therefore the correspondence between blend shapes and weight indices, also depends on the mesh constructed through a depth-first traversal of the node hierarchy. Channels enable the animation of blend shape weights.

[0253] For a node's texture displacement weight, "nodes" must reference a node that references a texture with displacement or at least one descendant of a texture with displacement. In this case, the weight is the total number of texture displacements found in the mesh constructed through a depth-first traversal of the node hierarchy. The order of the weights, and therefore the correspondence between texture displacements and weight indices, also depends on the mesh constructed through a depth-first traversal of the node hierarchy. Channels enable the animation of texture displacement weights.

[0254] For the weight of another controller, "controller" must reference another controller in the "controllers" collection / set. Such references must not result in circular dependencies. If the "controller" property exists, the "nodes" and "path" properties must not exist (or are ignored in some embodiments). If the "nodes" property exists, "path" must exist and "controller" must not exist (or can be ignored in some embodiments). The "interpolation" property defines the interpolation method.

[0255] Given input values ​​(defined in the parent controller), weight magnitudes (depending on the target component, as described above), interpolation methods, and output values ​​(defined in the "output" attribute), any output weight can be calculated for any input weight between the minimum and maximum input values. This method and algorithm are similar to... glTF 2.0 The methods and algorithms defined in the specification are the same.

[0256] Once the output weights are calculated for the input weights, they are used by the component targeted by the "controller" or ("nodes" + "path"). For example, if the channel is aimed at rotating nodes, the output weights have four numbers for rotating the nodes.

[0257] LOD definition LOD contains all the data for an incarnation at a given level of detail, as shown in Table 13.

[0258]

[0259] Table 13. The "lod" attribute indicates the level of detail. Lower values ​​correspond to higher levels of detail. For example, a lod value of 0 indicates more detail than a lod value of 1.

[0260] The "mesh" attribute references a node in the "nodes" collection. This node, or one of its descendants, must reference the geometry. The avatar mesh under this LOD is a combination of all attributes found in the node and its descendants (see Section 3.7).

[0261] The "skeletons" list references the skeletons in the "skeletons" collection. Skeletons can be used to animate avatar meshes under this LOD.

[0262] The "controller" list refers to the controllers in the "controllers" collection. Controllers can be used to animate the avatar mesh under this LOD.

[0263] Metadata definition The "metadata" property contains general information about the avatar, as shown in Table 14.

[0264]

[0265] Table 14. The "name" attribute defines the name used to describe the avatar.

[0266] The "id" attribute defines a unique value that identifies the avatar. This value is not the user's ID, but rather a value used to distinguish the avatar from any other avatar within a context such as an application or virtual world.

[0267] The "age" attribute defines the avatar's age.

[0268] The "gender" attribute defines the gender (or multiple genders) of the avatar. The value of this attribute is not limited to "male" and "female" and can be extended to any related value.

[0269] Asset Definition The "asset" attribute contains encoding information, as shown in Table 15.

[0270]

[0271] Table 15. The "version" property defines the version of the file. The "copyright" property contains a copyright message suitable for displaying acknowledgments to the content creators. The "generator" property indicates how the file was generated.

[0272] AJIF Solution Example Figure 6 This is a schematic diagram illustrating an exemplary avatar mesh model according to some embodiments. Figure 6 An exemplary full-body mesh 600 is shown.

[0273] Figure 7 This is a schematic diagram illustrating an exemplary avatar skeletal model according to some embodiments. Figure 7 An exemplary skeletal model 700 with a set of joints 702 is shown.

[0274] Table 16 shows the following list of exemplary code examples, which illustrates the JSON format portion of an AJIF file.

[0275]

[0276] Table 16. The parser first decodes the "asset" root attribute and ensures that it is processing the version of the file. Since the "version" attribute indicates version "1.0", parsing can continue.

[0277] The parser analyzes the items in the "lod" root list. This item is named "main" in the "lod" root list and is found in the "name" attribute. The detail level is 0, and it is found in the "lod" attribute.

[0278] This LOD has a mesh structure due to the presence of the "mesh" attribute. The value of this attribute is "0," thus referencing the first node in the "nodes" list. The parser decodes this node and finds the "mapping" attribute with a value of "full_body." Applications can be written to treat this node as a full-body avatar. The parser also finds the "geometry" attribute in the first node, which references the first item in the root list of "geometries."

[0279] The parser decodes the first geometry in the "geometries" root list. The parser can find multiple attributes that reference data in the "data" root list. In each case, the parser can decode the item found in the "data" root list and associate the result with the referenced attribute. For example, the geometry's "vertices" attribute references the first item in the "data" root list. The parser collects the relevant binary data. In this case, the relevant binary data is read from the "0_positions.bin" file. Then, given the "type", "dims", and "dtype" attributes of the data item, a 32-bit floating-point tensor with dimensions [53698, 3] is constructed using the binary data. Finally, this tensor is used to define the vertices of the avatar's geometry. Vertices ("vertices" attribute), normals ("normals" attribute), uvs ("uv" list), and their corresponding face indices (found in the "faces" attribute) are decoded and added to the avatar's geometry.

[0280] The parser found two blend shapes in the "blendshapes" list of geometry. Their names are "AU1_Inner_Brow_Raiser_L" and "AU1_Inner_Brow_Raiser_R" ​​(decoded from the "name" attribute). Both are based on... FACS Canonical action unit 1 (decoded from the "mapping" attribute). The first blend shape updates the left side of the avatar, and the second blend shape updates the right side (decoded from the "side" attribute). The parser uses the "displacements" attribute to construct a sparse tensor for each blend shape. This sparse representation can remain as is, or it can be merged, for example, into a single dense tensor of dimensions [2, 53698, 3].

[0281] The parser decodes the texture from the first item in the geometry's "textures" list. The texture type is "albedo" (found in the "type" attribute). The texture uses the first uv definition of the geometry (UV coordinates + face index) because the texture's "uv" attribute is 0. The texture image is constructed from the data item at index 6 (found in the "image" attribute). Given the data item's "dtype" and "ctype" attributes, the parser loads the image file "0_albedo.png" and converts it to a 32-bit floating-point RGB image. Finally, the parser decodes the "base" list and obtains a 32-bit floating-point dense tensor with two dimensions [512, 512, 3] containing the texture displacement.

[0282] The parser has decoded the contents of the geometry item and collected all the corresponding data in an internal structure, such as a scene object found in a game engine. Since the first node references the geometry and has no child nodes, the resulting avatar mesh consists of only a single geometry.

[0283] Returning to the LOD item, resolving the "skeletons" list: there exists a single reference to the first bone in the root list of "skeletons". This bone has 25 joints referenced by the bone's "joints" property. The root of the bone is the node with index 1 (the "root" property). Each node referenced by these properties corresponds to a joint and contains a "mapping" property that defines the joint's position in the body. In most cases, there exists a "children" list that defines the child nodes of the nodes. The "children" list is used to build the hierarchy of joints. Most of them also contain "rotation", "scale", and "translation" properties: these are used to build the 3D position of each joint. This can be done recursively, starting from the root joint and knowing its position is (0, 0, 0).

[0284] Processing Model The following sections describe the components of the encoder and decoder architecture of the incarnation embodiment, as well as the previously defined properties.

[0285] encoder Figure 8 This is a process diagram illustrating an exemplary encoder architecture according to some embodiments. Figure 8 An encoder architecture 800 is shown, in which the encoder processes avatar data 802 in the format described above. For some embodiments, the output data from the encoder is in ".ajif" format, and / or for some embodiments, it is a binary compressed version of the same format. In the binary format, the data can be grouped (810) to allow independent access to each element of the avatar embodiment.

[0286] Figure 8 The format analysis block 804 and formatting block 806 described in the text are parsing mechanisms to extract relevant information from different file formats, so that the information can be packaged and compressed into a human-readable format by the binary compression module 808.

[0287] The formatting module 806 allows the generation of a human-readable format (AJIF) (which, in some embodiments, can be output as an AJIF file 812) instead of a binary format (which, in some embodiments, can be output as a binary file 814).

[0288] The binary compression module 808 uses the standard SPIHT algorithm and arithmetic coding (AC) to apply lossless compression and convert the data into a bit stream.

[0289] For ".json" files, binary compression 808 follows the RFC 8949 concise binary object representation standard.

[0290] Block 810 is used to organize binary compressed data into dedicated blocks for each element (such as data and geometry) of the embodiment. This can be used during transmission to request access to only a portion of the data.

[0291] decoder Figure 9 This is a process diagram illustrating an exemplary decoder architecture according to some embodiments. Figure 9 The decoder architecture 900 is shown, in which the input is formatted as a binary data file 902 and / or an AJIF file 904.

[0292] The input binary format data undergoes degrouping 906 and decompression 908 to conform to the same specifications used and presented in the encoder. Data is then extracted from the file and mapped to the "Avatar Data" format. After this step, the application has information related to the avatar embodiment. Data in Human-readable Format (AJIF) and "Avatar Data" format 912 can also be extracted. In some embodiments, AJIF formatted data and / or binary decompressed data may pass through an exchange file format block 910 to output avatar data 912.

[0293] Streaming Figure 10 This is a process diagram illustrating an exemplary process for streaming AJIF files according to some embodiments. Figure 10 An exemplary mechanism for a streaming pipeline that uses the AJIF file format to send avatar data is illustrated. The AJIF file format is compatible with streaming by arranging the avatar's components in packets, allowing applications to request specific attributes of the avatar, such as the avatar's level of detail, or just skeletal information or animation controllers.

[0294] Figure 10 This flexibility is described. Frame 1000 begins with one of the inputs from the receiving input list 1002 (video, image, mesh, or proprietary avatar format). Data from input 1002 is processed 1004 and mapped to AJIF format. "Avatar Data" 1006 conforming to AJIF format can be compressed for transmission, for example, encoded 1008. Encoded data (AJIF data 1014) can be received and decoded at the receiving end 1010. The AJIF format is a JSON-based format, and for binary compression, AJIF-formatted data can follow the JSON compression standard RFC 8949: Concise Binary Object Representation. Output processing block 1012 maps the content provided by the AJIF stream to application-specific requirements, such as those for rendering, interactive applications, video-based streaming applications, etc.

[0295] Video conferencing tools Figure 11 This is a process diagram illustrating an exemplary video conferencing architecture with AJIF format data according to some embodiments. Two people initiate a video call using a conventional flat panel display 2D device as output and one or more cameras as input devices. The AJIF format provides the ability to transfer avatar instances between users. Figure 11 An exemplary process 1100 is shown for a video conferencing use case that generates virtual representations of users using the AJIF format. Users can access a webcam and can send their own live video feeds. In this use case, two users 1102, 1106 both send AJIF files 1114 representing their chosen avatars. The goal of server 1104 is to decode 1108 and create 1110 appropriate functions for avatar manipulation, rendering, and / or animation 1112.

[0296] In this example, AJIF encoding includes "Geometries" data, which can contain vertices, faces, normals, UVs, blend shapes, and textures representing the avatar model. This "Geometries" data is transmitted by each user at the start of the stream, and the server engine creates the 3D geometry needed to render and animate the avatar representation at each end of the communication. Subsequent streams contain video feeds from each user, and the engine infers changes in their facial pose and expression to animate the avatar. If rendering is on the server side, the engine updates the avatar and scene and provides the rendered results to both clients. If rendering is on the client side, the engine provides avatar parameters (typically blend shape weights) to animate the avatar.

[0297] Workflow Plan Figure 12 This is a message sequence diagram illustrating an exemplary process for a video conferencing use case according to some embodiments. Figure 12 Some details of an exemplary process 1200 for the workflow and coordination between a user and an application are shown. In item 1 (1214), an initialization protocol is executed, including application initialization, user authentication, service discovery, and rendering device detection. In item 2 (1216), user 1210 sends an avatar AJIF file to initialize the avatar. In item 3 (1218), engine 1208 requests AJIF decoder 1206 to interpret the file. In item 4 (1220), the decoded AJIF file is sent to avatar creation block 1204 to create the avatar in the application format. In item 5 (1222), the avatar application-related structure and manipulators are returned to engine 1208 for real-time manipulation. In item 6 (1224), avatar animation parameters are sent to avatar animation block 1202 for real-time animation. In item 7 (1226), user 1210 moves their face by rotating or changing facial expressions. In item 8 (1228), the landmarks of facial pose and expression are sent to animation block 1202. In item 9 (1230), animation block 1202 converts the landmarks into blend shape weight coefficients. In item 10 (1232), animation block 1202 notifies engine 1208 to update the avatar and scene. In item 11 (1234), engine 1208 notifies client 1210 of scene changes. In item 12 (1236), 2D display 1212 is updated with the new changes related to the avatar and scene.

[0298] Immersive experience Figure 13A and Figure 13B A system diagram is formed illustrating a set of exemplary interfaces for an immersive experience with one or more users / avatars, according to some embodiments. One or more people (represented as User 0 (1302)...User N (1304)) initiate an application in an immersive environment using augmented reality (AR) or virtual reality (VR). The immersive environment application may use 3D rendering mechanisms such as HMD glasses, holographic displays, and / or immersive head-mounted devices to represent people, rather than video calls. Figure 13A and Figure 13B The architecture 1300 for use cases in immersive experiences is shown. The experience is defined as a virtual scene and stored in a dedicated database 1320, which can be offline or online.

[0299] A virtual scene contains a complete description of its components, such as virtual objects, interactive systems, avatar manipulators, user containers, virtual cameras, lighting, and so on. The server's goal is to manage the interaction between the user and the scene. Therefore, server 1312 can be responsible for rendering, animation, and / or scene interaction as needed, or, in some embodiments, this work can be separated between server 1312 and user applications 1306, 1308. In some embodiments, if computing resources permit, this work is performed by user applications 1305, 1308.

[0300] Each user 1302, 1304 has an application 1306, 1308 capable of rendering scene interactions, animate scene interactions, and / or performing scene interactions. User applications 1306, 1308 can directly perform user / avatar interactions or animations, where the server will need to request additional input from the device to perform the user / avatar interaction or animation. Therefore, user applications 1306, 1308 can process data related to animate creation. User applications 1306, 1308 store the avatar identity to be loaded into the virtual world in the cloud or on devices 1314, 1316. AJIF files allow this flexibility, and network 1310 can request that AJIF files be stored in avatar storage device 1318. In some embodiments, AJIF files may be stored directly in device memory. This data is sent to server 1312 to perform scene updates related to avatar interactions.

[0301] Virtual World Figure 14A and Figure 14B A system diagram is formed illustrating a set of exemplary interfaces for a virtual world with multiple online users / avatars, according to some embodiments.

[0302] Multiple people (1402, 1404) connect to the virtual world via video or immersive displays (2D video, HDMR glasses, or immersive head-mounted devices). Figure 14A and Figure 14B The architecture 1400 for use cases in the virtual world is shown.

[0303] The virtual world is defined as a virtual scene and stored in a dedicated database 1406. The avatar is represented as an AJIF file and stored in database 1408, and can be queried by users 1402, 1404 or server application 1410.

[0304] The system involves multiple servers, each with a specific function to enhance computing power and data distribution in large-scale and dense networks. Servers may include physical servers, rendering servers, and scene manager servers.

[0305] Physics server 1412 is responsible for handling 3D object collisions, lighting simulation, shadow casting, and other environmental physics simulations in the world. The physics server processes user input data to predict and correct physical interactions within the scene.

[0306] Render server 1414 renders the scene to the target device, taking into account physically simulated parameters such as lighting, shadows, object positions, and materials. The rendering server also renders spatial sound effects based on the physically simulated properties.

[0307] Scene Manager Server 1416 can manage scene parameters, such as dynamic and static parameters. Dynamic parameters can be used for interactive events that need to be initiated within a timestamp or due to the interactivity of certain objects. Static parameters can include, for example, objects or non-interactive objects in the scene related to the initialization phase.

[0308] The Scene Manager Server 1416 manages information related to users and their corresponding avatars, as well as how they are integrated into the scene. The Scene Manager Server also manages any assets that users / avatars can interact with, and the interactive space within the virtual world. Based on scene status and application requirements, the Scene Manager Server can request individual avatar components or complete avatar descriptions from the AJIF database.

[0309] Scene manager server 1416 may include client server and network server.

[0310] Client server 1418 can manage user interactions between users and objects and notify the network when updates are needed. Client server 1418 can handle the payload capacity in the network for client / user information streaming, resulting in high-speed bitstreams and high performance on the user application side. Client server 1418 can initialize communication between the user and server applications. Client server 1418 can request individual avatar components or complete avatar descriptions from the AJIF database based on user / client requirements that may differ from those of the server.

[0311] Network server 1420 can manage data synchronization between all servers and data distribution within the network. Network server 1420 synchronizes users at the same timestamp for all events. Network server 1420 can establish online communication protocols between users and servers. Network server 1420 can monitor stable connections and communication between all parties.

[0312] In some implementations, where local computing power is required to overcome network transmission difficulties, users are clients who include their own hardware and applications to facilitate server operations (such as rendering, physics simulation, avatar creation, and animation) and scene interaction. In such scenarios, such as multi-user online video games, server applications need to predict user behavior.

[0313] Asset creation workflow Figure 15 This is a message sequence diagram illustrating an exemplary process of asset creation workflow between a client and server application according to some embodiments. Figure 15 A use case for an asset creation pipeline with an exemplary workflow 1500 is illustrated. User client 1508 can be an artist or a regular user interacting with an interface ("compositer interface") to create an avatar model. This model allows for the generation of personalized avatar representations and demonstrates the availability of AJIF files. The goal of the avatar creation block is to validate and ensure that the data conforms to the format discussed above, and that the encoder is able to interpret the avatar data to generate an AJIF file.

[0314] In item 1 (1510), the initialization protocol is executed, including application initialization, user authentication, service discovery, and an interface for visualizing avatar attributes (compositer interface). In item 2 (1512), user 1508 inserts information related to their avatar identity, such as name, identifier, age, and gender. In item 3 (1514), a request to verify advanced information data is sent via computer interface 1506. In item 4 (1516), a request to encode data is sent to AJIF encoding block 1502. In item 5 (1518), as previously described, the advanced information data is encoded and structured.

[0315] In item 6 (1520), user 1508 selects an avatar mesh from a list of suggested avatars recommended by the synthesizer, or manually creates their own avatar by providing the following information: geometric data, such as vertices, faces, normals, and blend shapes; skeletal data, such as root joints, joints, and skin weights; and mapping data. In item 7 (1522), a request is sent to avatar creation block 1504 to verify the avatar mesh, mapping, and skeletal information data. In item 8 (1524), a request to encode the data is sent. In item 9 (1526), ​​the avatar mesh, mapping, and skeletal information data are encoded and structured as described above.

[0316] In item 10 (1528), by editing the controllers that animate geometry, bones, blending shapes, textures, or controller weights, user 1508 selects an avatar animation from a list of animations suggested by the compositor, or manually creates an animation via the compositor. In item 11 (1530), a request is sent from compositor interface block 1506 to avatar creation block 1504 to verify the avatar animation / controller information data. In item 12 (1532), a request to encode the data is sent. In item 13 (1534), the avatar animation / controller information data is encoded and structured by AJIF encoding block 1502 as previously described.

[0317] In item 14 (1536), user 1508 selects an avatar appearance from a list of appearances suggested by the synthesizer, or manually creates their own appearance by providing one or more of the following information: texture type, such as albedo, normal, specular reflection; UVS; face; texture image; and texture displacement. In item 15 (1538), a request is sent to verify the avatar appearance / texture information data. In item 16 (1540), a request is sent to encode the data. In item 17 (1542), the avatar appearance / texture information data is encoded and structured as described above.

[0318] In item 18 (1544), user 1508 requests that their selection be saved as an AJIF file for future use in other applications. In item 19 (1546), a request is sent to encode the avatar data. In item 20 (1548), a request is sent to encode and compress all avatar data. In item 21 (1550), the avatar information selected by the user is compressed. In item 22 (1552), the encoded and compressed (or uncompressed, depending on the user's selection) data is sent to the user as a .ajif file for loading in the virtual environment.

[0319] Figure 16 This is a diagram illustrating exemplary results of an asset creation process related to avatar representation according to some embodiments. Figure 16 The results of the asset creation process related to avatar representation are shown. Users integrate AJIF files into a virtual interactive world with realistic rendering. Figure 16 An exemplary output 1600 is shown, illustrating the creation and integration of avatar assets into a virtual world.

[0320] Figure 17 This is a flowchart illustrating an exemplary process for decoding Avatar JSON Exchange File (AJIF) data according to some embodiments. In some embodiments, exemplary process 1700 may include acquiring 1702 Avatar JSON Exchange File (AJIF) data. In some embodiments, exemplary process 1700 may also include decoding the AJIF data 1704 to generate avatar data. In some embodiments, exemplary process 1700 may also include acquiring animation parameters from the avatar data 1706. In some embodiments, exemplary process 1700 may also include acquiring 1708 user action data corresponding to a user's action. In some embodiments, exemplary process 1700 may also include generating updated avatar data based on the user action data 1710. In some embodiments, exemplary process 1700 may also include sending the updated avatar data 1712 to a client device.

[0321] Figure 18 This is a flowchart illustrating an exemplary process for encoding avatar JSON exchange file (AJIF) data according to some embodiments. In some embodiments, exemplary process 1800 may include generating 1802 avatar JSON exchange file (AJIF) data corresponding to an avatar. In some embodiments, exemplary process 1800 may also include sending the AJIF data 1804 to an application server. In some embodiments, exemplary process 1800 may also include obtaining 1806 user action data corresponding to a user's action. In some embodiments, exemplary process 1800 may also include sending 1808 user action data corresponding to a user's action to an application server. In some embodiments, exemplary process 1800 may also include receiving 1810 scene update data from an application server, wherein the scene update data includes updated avatar data corresponding to the user's action. In some embodiments, exemplary process 1800 may also include rendering 1812 an update to the scene based on the received scene update data.

[0322] While methods and systems according to some embodiments are generally discussed within the context of extended reality (XR), some embodiments can be applied to any XR scenario, such as virtual reality (VR) / mixed reality (MR) / augmented reality (AR) scenarios. Additionally, although the term "head-mounted display (HMD)" is used herein according to some embodiments, for some embodiments, some embodiments can be applied to wearable devices capable of supporting, for example, XR, VR, AR, and / or MR (which may or may not be attached to the head).

[0323] A first exemplary method according to some embodiments may include: acquiring avatar JSON exchange file (AJIF) data; decoding the AJIF data to generate avatar data; acquiring animation parameters from the avatar data; acquiring user action data corresponding to the user's actions; generating updated avatar data based on the user action data; and sending the updated avatar data to a client device.

[0324] Some embodiments of the first exemplary method may also include generating avatar animation rendering data using animation parameters.

[0325] In some embodiments of the first exemplary method, the user action data is facial landmark data.

[0326] For some embodiments of the first exemplary method, generating updated avatar data may include: obtaining a hybrid shape weight update corresponding to facial landmark data; and applying the hybrid shape weight update to the avatar data to generate updated avatar data.

[0327] For some embodiments of the first exemplary method, generating updated avatar data may include: obtaining vertex position data associated with the avatar from the avatar data; obtaining linear blend skin weights from the avatar data; obtaining a transformation matrix from the avatar data; and using the vertex position data, linear blend skin weights, and transformation matrix to generate updated avatar data.

[0328] In some embodiments of the first exemplary method, the animation parameters may include a channel object corresponding to a portion of the avatar.

[0329] Some embodiments of the first exemplary method may also include analyzing avatar data to determine the type of avatar represented in the avatar data.

[0330] Some embodiments of the first exemplary method may also include analyzing avatar data to determine an avatar portion mapping of the avatar data.

[0331] Some embodiments of the first exemplary method may also include analyzing avatar data to determine at least one node in the avatar data associated with the avatar.

[0332] For some embodiments of the first exemplary method, AJIF data may include data of at least one of the following data types associated with the avatar: original avatar data, geometry data, texture data, skeleton data, controller data, and level of detail (LOD) data.

[0333] For some embodiments of the first exemplary method, AJIF data may include data of at least one of the following data types: file node data, metadata, and file asset data.

[0334] In some embodiments of the first exemplary method, AJIF data corresponds to two or more files.

[0335] In some embodiments of the first exemplary method, AJIF data may include image data associated with the avatar.

[0336] A first exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the methods listed above.

[0337] A second exemplary method according to some embodiments may include: generating avatar JSON exchange file (AJIF) data corresponding to an avatar; sending the AJIF data to an application server; obtaining user action data corresponding to a user's action; sending the user action data corresponding to a user's action to the application server; receiving scene update data from the application server, wherein the scene update data includes updated avatar data corresponding to the user's action; and rendering an update to the scene based on the received scene update data.

[0338] In some embodiments of the second exemplary method, the user action data is facial landmark data.

[0339] A second exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the methods listed above.

[0340] A third exemplary method according to some embodiments may include: acquiring avatar JSON exchange file (AJIF) data; decoding the AJIF data to generate avatar data; acquiring animation parameters from the avatar data; acquiring user action data corresponding to the user's actions; generating updated avatar data based on the user action data; and rendering the updated avatar data to a display device.

[0341] Some embodiments of the third exemplary method may also include generating avatar animation rendering data using animation parameters.

[0342] In some embodiments of the third exemplary method, the user action data is facial landmark data.

[0343] For some embodiments of the third exemplary method, generating updated avatar data may include: obtaining a hybrid shape weight update corresponding to facial landmark data; and applying the hybrid shape weight update to the avatar data to generate updated avatar data.

[0344] For some embodiments of the third exemplary method, generating updated avatar data may include: obtaining vertex position data associated with the avatar from the avatar data; obtaining linear blend skin weights from the avatar data; obtaining a transformation matrix from the avatar data; and using the vertex position data, linear blend skin weights, and transformation matrix to generate updated avatar data.

[0345] In some embodiments of the third exemplary method, the animation parameters may include a channel object corresponding to a portion of the avatar.

[0346] Some embodiments of the third exemplary method may also include analyzing avatar data to determine the type of avatar represented in the avatar data.

[0347] Some embodiments of the third exemplary method may also include analyzing avatar data to determine an avatar portion mapping of the avatar data.

[0348] Some embodiments of the third exemplary method may also include analyzing avatar data to determine at least one node in the avatar data associated with the avatar.

[0349] For some embodiments of the third exemplary method, AJIF data includes data of at least one of the following data types associated with the avatar: original avatar data, geometry data, texture data, skeleton data, controller data, and level of detail (LOD) data.

[0350] For some embodiments of the third exemplary method, AJIF data may include data of at least one of the following data types: file node data, metadata, and file asset data.

[0351] In some embodiments of the third exemplary method, AJIF data corresponds to two or more files.

[0352] In some embodiments of the third exemplary method, AJIF data may include image data associated with the avatar.

[0353] A third exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform the method as described in any one of claims 18 to 30.

[0354] A fourth exemplary method according to some embodiments may include: acquiring avatar JSON exchange file (AJIF) data; decoding the AJIF data to generate avatar data; acquiring animation parameters from the avatar data; receiving facial landmark data corresponding to a user's facial movements; generating updated avatar data based on the facial landmark data; and sending the updated avatar data to a client device.

[0355] A fourth exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the methods listed above.

[0356] A fifth exemplary method according to some embodiments may include: acquiring identity data about an avatar; generating advanced information data using the identity data; encoding the advanced information data; acquiring skeletal data about the avatar; generating mesh, mapping, and skeletal information data using the skeletal data; encoding the mesh, mapping, and skeletal information data; acquiring animation data about the avatar; generating controller information data using the animation data; encoding the controller information data; acquiring appearance data about the avatar; generating texture information data using the appearance data; encoding the texture information data; receiving a request to save avatar data; compressing the encoded advanced information data, the encoded mesh, mapping, and skeletal information data, the encoded controller information data, and the encoded texture information data to generate compressed avatar data; and sending the compressed avatar data.

[0357] A fifth exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform any of the methods listed above.

[0358] A sixth exemplary device according to some embodiments may include at least one processor configured to perform any of the methods listed above.

[0359] A seventh exemplary device according to some embodiments may include a computer-readable medium storing instructions for causing one or more processors to perform any of the methods listed above.

[0360] An eighth exemplary device according to some embodiments may include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing at least one processor to perform any of the methods listed above.

[0361] According to some embodiments, the signal may include updated avatar data generated according to any of the methods listed above.

[0362] This disclosure describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described in a specific manner and are generally described in a way that may sound limiting, at least for the purpose of illustrating the various features. However, this is for the purpose of clarity of description and does not limit the disclosure or scope of these aspects. In fact, all the different aspects can be combined and interchanged to provide other aspects. Furthermore, these aspects can also be combined and interchanged with aspects described in previous documents.

[0363] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are specifically shown, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of implementation. At least one aspect relates generally to video encoding and decoding, and at least one other aspect relates generally to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions thereon stored thereon for encoding or decoding video data according to any of the methods, and / or computer-readable storage media having bitstreams generated according to any of the methods stored thereon.

[0364] In this disclosure, the terms "reconstruction" and "decoding" are used interchangeably, as are the terms "pixel" and "sample," and the terms "image," "picture," and "frame." Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0365] The terms HDR (High Dynamic Range) and SDR (Standard Dynamic Range) generally convey specific values ​​of dynamic range to those skilled in the art. However, it is also intended to include additional embodiments in which references to HDR should be understood as meaning "higher dynamic range" and references to SDR should be understood as meaning "lower dynamic range". Such additional embodiments are not bound by any specific values ​​of dynamic range that may generally be associated with the terms "high dynamic range" and "standard dynamic range".

[0366] This document describes various methods, each of which includes one or more steps or actions for implementing the described method. The order and / or use of specific steps and / or actions may be modified or combined unless the correct operation of the method requires a specific order. Additionally, in various embodiments, terms such as "first," "second," etc., may be used to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless specifically required, the use of such terms does not imply a sequence of operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding, but may occur, for example, before, during, or in a time period overlapping with the second decoding.

[0367] For example, various numerical values ​​may be used in this disclosure. Specific values ​​are for illustrative purposes, and the aspects described are not limited to these specific values.

[0368] The embodiments described herein can be implemented by computer software, either by a processor or other hardware or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, the processor can be of any type suitable for the technical environment and can encompass one or more microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0369] Various implementations involve decoding. As used in this disclosure, "decoding" can encompass all or part of a process performed, for example, on a received encoded sequence, to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also includes, or alternatively includes, processes performed by a decoder of the various embodiments described in this disclosure, such as extracting an image from a tiled (packed) image, determining an upsampling filter to use and then upsampling the image, and flipping the image back to its intended orientation.

[0370] As another example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Based on the specific context of the description, it will be clear whether the phrase "decoding process" is intended to specifically refer to a subset of operations or to refer to the broader decoding process.

[0371] Various implementations involve encoding. Similar to the discussion above regarding "decoding," "encoding" as used in this disclosure can include, for example, all or part of the processing performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential coding, transform, quantization, and entropy coding. In various embodiments, such processes also include, or alternatively include, processes performed by a decoder of the various embodiments described in this disclosure.

[0372] As another example, in one embodiment, "encoding" refers only to entropy encoding; in another embodiment, "encoding" refers only to differential encoding; and in yet another embodiment, "encoding" refers to a combination of entropy encoding and differential encoding. Based on the specific context of the description, it will become clear whether the phrase "encoding process" is intended to specifically refer to a subset of operations or to refer to a broader encoding process.

[0373] When a diagram is presented as a flowchart, it should be understood that a block diagram of the corresponding device is also provided. Similarly, when a diagram is presented as a block diagram, it should be understood that a flowchart of the corresponding method / process is also provided.

[0374] Various embodiments involve rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is typically considered, often due to constraints on computational complexity. Rate-distortion optimization is generally formulated as minimizing a rate-distortion function, which is a weighted sum of code rate and distortion. Different approaches exist to address the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all encoding options, including all considered modes or encoding parameter values, where a comprehensive evaluation of the encoding cost and associated distortion of the reconstructed signal is performed after encoding and decoding. Faster methods can also be used to save encoding complexity, particularly by calculating approximate distortion based on prediction or prediction of the residual signal rather than the reconstructed signal. A hybrid of these two approaches can also be used, such as by using approximate distortion for only some possible encoding options and full distortion for others. Other methods evaluate only a subset of possible encoding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization is not necessarily a comprehensive evaluation of encoding cost and associated distortion.

[0375] The embodiments and aspects described herein can be implemented, for example, in methods or processes, apparatuses, software programs, data streams, or signals. Even if discussed only in the context of a single embodiment (e.g., discussed only as a method), embodiments of the discussed features can also be implemented in other forms (e.g., apparatuses or programs). Apparatuses can be implemented, for example, with suitable hardware, software, and firmware. Methods can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as computers, cellular phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate information communication between end users.

[0376] The references to "an embodiment" or "an embodiment" or "an implementation" or "implementation", as well as other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the phrases "in an embodiment" or "in an embodiment" or "in an implementation" or "in an implementation", as well as any other variations, appearing throughout this disclosure, do not necessarily refer to the same embodiment.

[0377] Additionally, this disclosure may relate to "determining" various types of information. Determining information may include one or more of the following: for example, estimated information, calculated information, predicted information, or information retrieved from memory.

[0378] Furthermore, this disclosure may relate to "accessing" various types of information. Accessing information may include one or more of the following: for example, receiving information, retrieving information (e.g., retrieving from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0379] Additionally, this disclosure may relate to "receiving" various types of information. Like "access," the intent to receive is a broad term. Receiving information may include one or more of the following: for example, accessing information or retrieving information (e.g., retrieving from memory). Furthermore, "receiving" is generally referred to in one or more ways during operation, such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0380] It should be understood that, for example, in the cases of "A / B", "A and / or B", and "at least one of A and B", the use of any of the following " / ", "and / or", and "at least one" is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As yet another example, in the cases of "A, B, and / or C" and "at least one of A, B, and C", this wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). This can be extended to as many as the items listed.

[0381] Furthermore, as used herein, the term "signaling" specifically refers to instructing the corresponding decoder to do something. For example, in some embodiments, the encoder signals a particular parameter among several parameters for region-based filter parameter selection for artifact removal filtering. In this way, in embodiments, the same parameter is used at both the encoder and decoder sides. Thus, for example, the encoder can send (explicitly signal) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling can be used without sending (implicitly signaling) to simply allow the decoder to know and select the specific parameter. Bit savings are implemented in various embodiments by avoiding the transmission of any actual functionality. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to send information to the corresponding decoder. Although the verb form of the word "signal" has been referred to above, the word "signal" can also be used as a noun herein.

[0382] Implementation methods can generate various signals that are formatted to carry information, such as information that can be stored or transmitted. The information may include, for example, instructions for performing a method, or data generated by one of the described implementation methods. For example, the signal may be formatted to carry a bit stream of the described embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding the data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0383] Several embodiments are described. Features of these embodiments may be provided individually or in any combination across various claim classes and types. Furthermore, embodiments may include one or more of the following features, means, or aspects individually or in any combination across various claim classes and types: Adjust the residual at the encoder according to any of the embodiments discussed.

[0384] A bitstream or signal that includes one or more of the described syntactic elements or their variants.

[0385] A bitstream or signal, which includes syntax for conveying information generated according to any embodiment of the described embodiments.

[0386] Syntactic elements are inserted into the signaling, which enable the decoder to adjust the residuals in a manner corresponding to that used by the encoder.

[0387] Create and / or send and / or receive and / or decode bitstreams or signals, which include one or more of the described syntactic elements or their variants.

[0388] Create and / or send and / or receive and / or decode according to any of the described embodiments.

[0389] A method, process, apparatus, medium for storing instructions, medium for storing data, or signal according to any of the described embodiments.

[0390] A TV, set-top box, mobile phone, tablet computer or other electronic device that performs adjustments to the filter parameters according to any of the described embodiments.

[0391] A TV, set-top box, mobile phone, tablet computer or other electronic device that performs adjustment of filter parameters according to any of the described embodiments and displays (e.g., using a monitor, screen or other type of display) the resulting image.

[0392] A TV, set-top box, mobile phone, tablet computer or other electronic device that selects channels (e.g., using a tuner) to receive signals including encoded images and performs adjustments to filter parameters according to any of the described embodiments.

[0393] A TV, set-top box, mobile phone, tablet computer or other electronic device that receives signals including encoded images over the air (e.g., using an antenna) and performs adjustments to filter parameters according to any of the described embodiments.

[0394] It should be noted that the various hardware elements, one or more of which are described in the embodiments, are referred to as "modules," which implement (i.e., perform, execute, etc.) the various functions described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices) that a person skilled in the art would consider suitable for a given implementation. Each described module may also include executable instructions for performing one or more functions described as being performed by the respective module, and it should be noted that these instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., and may be stored in any suitable non-transitory computer-readable medium such as RAM, ROM, etc., commonly referred to as RAM, ROM, etc.

[0395] Although the features and elements have been described above in specific combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with other features and elements. Furthermore, the methods described herein can be implemented in a computer program, software, or firmware incorporated into a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM discs and digital multifunction disks (DVDs). The processor associated with the software can be used to implement a wireless transceiver for a WTRU, terminal, base station, RNC, or any host computer.< / dot> < / dot> < / dot> < / dot> < / dot> < / dot>

Claims

1. A method comprising: Retrieve data from the avatar JSON exchange file (AJIF); The AJIF data is decoded to generate avatar data; Obtain animation parameters from the avatar data; Obtain user action data corresponding to the user's actions; Updated avatar data is generated based on the user action data; as well as The updated avatar data is sent to the client device.

2. The method of claim 1, further comprising: Use the animation parameters to generate avatar animation rendering data.

3. The method of any one of claims 1 to 2, wherein the user action data is facial landmark data.

4. The method of claim 3, wherein generating the updated avatar data comprises: Obtain the hybrid shape weight update corresponding to the facial landmark data; as well as The hybrid shape weight update is applied to the avatar data to generate the updated avatar data.

5. The method of any one of claims 1 to 2, wherein generating the updated avatar data comprises: Obtain vertex position data associated with the avatar from the avatar data; Obtain linear blending skin weights from the avatar data; Obtain the transformation matrix from the avatar data; as well as The updated avatar data is generated using the vertex position data, the linear blending skin weights, and the transformation matrix.

6. The method of any one of claims 1 to 5, wherein the animation parameters include a channel object corresponding to a portion of the avatar.

7. The method according to any one of claims 1 to 6, further comprising: Analyze the avatar data to determine the type of avatar represented in the avatar data.

8. The method according to any one of claims 1 to 7, further comprising: Analyze the avatar data to determine the avatar portion mapping of the avatar data.

9. The method according to any one of claims 1 to 8, further comprising: Analyze the avatar data to determine at least one node in the avatar data that is associated with the avatar.

10. The method of any one of claims 1 to 9, wherein the AJIF data includes data of at least one of the following data types associated with the avatar: original avatar data, geometry data, texture data, skeleton data, controller data, and level of detail (LOD) data.

11. The method of any one of claims 1 to 10, wherein the AJIF data includes at least one of the following data types: file node data, metadata, and file asset data.

12. The method of any one of claims 1 to 11, wherein the AJIF data corresponds to two or more files.

13. The method of any one of claims 1 to 12, wherein the AJIF data includes image data associated with the avatar.

14. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform the method as described in any one of claims 1 to 13.

15. A method comprising: Generate JSON exchange file (AJIF) data corresponding to the avatar; Send the AJIF data to the application server; Obtain user action data corresponding to the user's actions; Send the user action data corresponding to the user's action to the application server; Receive scene update data from the application server. The scene update data includes updated avatar data corresponding to the user's action; and The scene update is rendered based on the received scene update data.

16. The method of claim 15, wherein the user action data is facial landmark data.

17. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform the method as described in any one of claims 15 to 16.

18. A method comprising: Retrieve data from the avatar JSON exchange file (AJIF); The AJIF data is decoded to generate avatar data; Obtain animation parameters from the avatar data; Obtain user action data corresponding to the user's actions; Updated avatar data is generated based on the user action data; as well as The updated avatar data is rendered onto the display device.

19. The method of claim 18, further comprising: Use the animation parameters to generate avatar animation rendering data.

20. The method of any one of claims 18 to 19, wherein the user action data is facial landmark data.

21. The method of claim 20, wherein generating the updated avatar data comprises: Obtain the hybrid shape weight update corresponding to the facial landmark data; as well as The hybrid shape weight update is applied to the avatar data to generate the updated avatar data.

22. The method of any one of claims 18 to 19, wherein generating the updated avatar data comprises: Obtain vertex position data associated with the avatar from the avatar data; Obtain linear blending skin weights from the avatar data; Obtain the transformation matrix from the avatar data; as well as The updated avatar data is generated using the vertex position data, the linear blending skin weights, and the transformation matrix.

23. The method of any one of claims 18 to 22, wherein the animation parameters include a channel object corresponding to a portion of the avatar.

24. The method of any one of claims 18 to 23, further comprising: Analyze the avatar data to determine the type of avatar represented in the avatar data.

25. The method of any one of claims 18 to 24, further comprising: Analyze the avatar data to determine the avatar portion mapping of the avatar data.

26. The method of any one of claims 18 to 25, further comprising: Analyze the avatar data to determine at least one node in the avatar data that is associated with the avatar.

27. The method of any one of claims 18 to 26, wherein the AJIF data includes data of at least one of the following data types associated with the avatar: original avatar data, geometry data, texture data, skeletal data, controller data, and level of detail (LOD) data.

28. The method of any one of claims 18 to 27, wherein the AJIF data includes at least one of the following data types: file node data, metadata, and file asset data.

29. The method of any one of claims 18 to 28, wherein the AJIF data corresponds to two or more files.

30. The method of any one of claims 18 to 29, wherein the AJIF data includes image data associated with the avatar.

31. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform the method as described in any one of claims 18 to 30.

32. A method comprising: Retrieve data from the avatar JSON exchange file (AJIF); The AJIF data is decoded to generate avatar data; Obtain animation parameters from the avatar data; Receive facial landmark data corresponding to the user's facial movements; Updated avatar data is generated based on the facial landmark data; as well as The updated avatar data is sent to the client device.

33. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform the method of claim 32.

34. A method comprising: Obtain identity data about the avatar; Use the identity data to generate advanced information data; The advanced information data is encoded; Obtain skeletal data about the avatar; The bone data is used to generate mesh, mapping, and bone information data; The mesh, mapping, and skeleton information data are encoded; Obtain animation data related to the avatar; The animation data is used to generate controller information data; The controller information data is encoded; Obtain appearance data about the avatar; Use the appearance data to generate texture information data; The texture information data is encoded; Receive a request to save avatar data; Compress encoded high-level information data, encoded mesh, mapping and skeleton information data, encoded controller information data, and encoded texture information data to generate compressed avatar data; as well as Send the compressed avatar data.

35. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to perform the method of claim 34.

36. An apparatus comprising at least one processor configured to perform the method as claimed in any one of claims 1 to 13, 15, 16, 18 to 30, 32 and 34.

37. An apparatus comprising a computer-readable medium storing instructions for causing one or more processors to perform the method as described in any one of claims 1 to 13, 15, 16, 18 to 30, 32 and 34.

38. An apparatus comprising at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform the method as described in any one of claims 1 to 13, 15, 16, 18 to 30, 32 and 34.

39. A signal comprising updated avatar data generated according to any one of claims 1 to 13, 18 to 30 and 32.

Images