Method of transmitting processing unit information associated with split points

Abstract

Systems, methods, and tools may be disclosed for processing processing unit information associated with a split point. A first device may include a processor. A first device may be configured to determine processing unit information associated with a processing unit of a second device. The processing unit information may be associated with the intermediate data. The processing unit information may include at least an indication of a processing unit type to be used by the second device to process the intermediate data and / or a processing unit identifier associated with a processing unit of the second device. The device may be configured to send processing unit information and / or intermediate data to a second device. The device may be configured to receive a processing unit state associated with the intermediate data from the second device.

Description

[0001] Cross-references to related applications. Background Technology

[0002] Video coding systems can be used to compress digital video signals, for example, to reduce the storage and / or transmission bandwidth required for such signals. Video coding systems can include, for example, block-based, wavelet-based, and / or object-based systems. Summary of the Invention

[0003] Systems, methods, and tools for processing processing unit information associated with a split point may be disclosed. A wireless transmit / receive unit (WTRU) (e.g., a first device) may include a processor. The first device may be configured to determine processing unit information associated with a processing unit of a second device. The processing unit information may be associated with intermediate data. The processing unit information may include at least an indication of the type of processing unit to be used by the second device to process the intermediate data and / or a processing unit identifier associated with a processing unit of the second device. The device may be configured to send the processing unit information and / or the intermediate data to the second device. The device may be configured to receive processing unit status associated with the intermediate data from the second device.

[0004] The first device can be configured to receive indications of hardware capabilities associated with the second device. Processing unit information can be determined based on the hardware capabilities associated with the second device. The first device can be configured to send processing unit information to the second device as part of metadata. The second device can use the metadata to decode or process intermediate data, and / or select or identify processing units of the second device. The first device can be configured to send processing unit recommendations to the second device as part of metadata. Processing unit recommendations can be based on processing power.

[0005] Metadata may be transmitted via a different channel than the intermediate data. The first device may be configured to receive inference results from the second device. The inference results may be associated with a score. The processing unit status associated with the intermediate data may include an indication that the processing unit of the second device is unavailable. The first device may be configured to receive a response message and / or inference results from the second device based on the indication that the processing unit of the second device is unavailable. The response message may indicate that the processing unit of the second device is unavailable and / or may indicate processing unit information. The processing unit status may include an indication of at least one of the following: processing unit load, processing unit type, or processing unit identifier.

[0006] A method can be performed by a first device. The method may include determining processing unit information associated with a processing unit of a second device. The processing unit information may be associated with intermediate data. The processing unit information may include at least an indication of a processing unit type to be used by the second device to process the intermediate data and / or a processing unit identifier associated with a processing unit of the second device. The method may include sending the processing unit information and / or the intermediate data to the second device. The method may include receiving a processing unit status associated with the intermediate data from the second device.

[0007] The method may include receiving an indication of hardware capabilities associated with a second device. Processing unit information may be determined based on the hardware capabilities associated with the second device. The method may include sending the processing unit information to the second device as part of metadata. The second device may use the metadata to decode or process intermediate data and / or select or identify processing units of the second device. The method may include sending processing unit recommendations to the second device as part of metadata. Processing unit recommendations may be based on processing power.

[0008] Metadata may be transmitted via a different channel than the intermediate data. The method may include receiving an inference result from the second device. The inference result may be associated with a score. The processing unit status associated with the intermediate data may include an indication that the processing unit of the second device is unavailable. The method may include receiving a response message and / or an inference result from the second device based on the indication that the processing unit of the second device is unavailable. The response message may indicate that the processing unit of the second device is unavailable and / or indicate processing unit information. The processing unit status may include an indication of at least one of the following: processing unit load, processing unit type, or processing unit identifier.

[0009] A wireless transmit / receive unit (WTRU) (e.g., a second device) may include a processor. The second device may be configured to receive processing unit information and / or intermediate data from a first device. The processing unit information may be associated with a processing unit of the second device. The processing unit information may be associated with the intermediate data. The processing unit information may include at least an indication of the type of processing unit to be used by the second device to process the intermediate data and / or a processing unit identifier associated with a processing unit of the second device. The second device may be configured to determine a processing unit status in response to receiving the processing unit information and / or intermediate data. The second device may be configured to send the processing unit status to the first device.

[0010] The second device can be configured to send an indication of hardware capabilities associated with the second device to the first device. The indication of hardware capabilities can be associated with the processing unit type of the second device. The second device can be configured to receive processing unit information from the first device as part of metadata. The second device can be configured to decode intermediate data based on the received metadata. The second device can be configured to identify the processing unit of the second device based on the metadata. The second device can be configured to receive processing unit recommendations from the first device as part of metadata. Processing unit recommendations can be associated with the amount of processing power required to process the intermediate data. The metadata can be received via a different channel than the intermediate data.

[0011] The second device can be configured to send an inference result to the first device. The inference result can be associated with a score. The processing unit status associated with the intermediate data can include an indication that the processing unit of the second device is unavailable. The second device can be configured to send a response message and / or an inference result based on the indication that the processing unit of the second device is unavailable. The response message can indicate the unavailability of the processing unit of the second device and / or processing unit information. The processing unit status can include an indication of at least one of the following: processing unit load, processing unit type, or processing unit identifier.

[0012] The method performed by the second device may include receiving processing unit information and / or intermediate data from the first device. The processing unit information may be associated with a processing unit of the second device. The processing unit information may be associated with the intermediate data. The processing unit information may at least include an indication of a processing unit type to be used by the second device to process the intermediate data and / or a processing unit identifier associated with a processing unit of the second device. The method may include determining a processing unit status in response to receiving the processing unit information and / or intermediate data. The method may include sending the processing unit status to the first device.

[0013] The method may include sending an indication of hardware capabilities associated with a second device to a first device. The indication of hardware capabilities may be associated with a processing unit type of the second device. The method may include receiving processing unit information from the first device as part of metadata. The method may include decoding intermediate data based on the received metadata. The method may include identifying the processing unit of the second device based on the metadata. The method may include receiving a processing unit recommendation from the first device as part of metadata. The processing unit recommendation may be associated with the amount of processing power required to process the intermediate data. The metadata may be received via a different channel than the intermediate data.

[0014] The method may include sending an inference result to a first device. The inference result may be associated with a score. The processing unit status associated with intermediate data may include an indication that the processing unit of the second device is unavailable. The method may include sending a response message and / or an inference result based on the indication that the processing unit of the second device is unavailable. The response message may indicate the unavailability of the processing unit of the second device and / or processing unit information. The processing unit status may include an indication of at least one of the following: processing unit load, processing unit type, or processing unit identifier.

[0015] A computer program product stored on a non-transitory computer-readable medium may include program code instructions for implementing the steps of the methods described herein when executed by at least one processor. The computer program may include program code instructions for implementing the steps of the methods described herein when executed by a processor. Video data may include information representing encoded output generated according to one of the methods described herein.

[0016] A device (e.g., a WTRU) may be configured with a processor to perform one or more actions. The device may receive a set of processing units and processing unit information associated with each corresponding processing unit in the set of processing units. The device may determine the amount of processing power required to process intermediate data. The device may select a processing unit from the set of processing units based on the processing unit information and the amount of processing power. The device may send an instruction indicating the selected processing unit.

[0017] In the example, processing unit information may include the processing unit type associated with each corresponding processing unit in the set of processing units.

[0018] In the example, processing unit information may include a processing unit identifier associated with each corresponding processing unit in the set of processing units.

[0019] In the example, the indication may include a processing unit identifier associated with the selected processing unit.

[0020] In the example, the device can receive a status update of the load associated with at least one of the processing units in the set of processing units.

[0021] In the example, the device can send processing unit recommendations as metadata.

[0022] The systems, methods, and tools described herein may relate to decoders. In some examples, the systems, methods, and tools described herein may relate to encoders. In some examples, the systems, methods, and tools described herein may relate to signals (e.g., from an encoder and / or received by a decoder). Computer-readable media may include instructions for causing one or more processors to perform the methods described herein. A computer program product may include instructions that, when executed by one or more processors, cause one or more processors to perform the methods described herein. Attached Figure Description

[0023] Figure 1A This is a system diagram illustrating an example communication system in which one or more of the disclosed embodiments may be implemented.

[0024] Figure 1B This illustrates that, according to an embodiment, it is possible to Figure 1A The system diagram shows an example wireless transmit / receive unit (WTRU) used within the communication system.

[0025] Figure 1C This illustrates that, according to an embodiment, it is possible to Figure 1A The system diagram shows an example radio access network (RAN) and an example core network (CN) used within the communication system shown.

[0026] Figure 1D This illustrates that, according to an embodiment, it is possible to Figure 1A The system diagram shows another example RAN and another example CN used in the communication system shown.

[0027] Figure 2 An example video encoder is shown.

[0028] Figure 3 An example video decoder is shown.

[0029] Figure 4 An example of a system in which various aspects and examples can be implemented is shown.

[0030] Figure 5A An example architecture for split inference between WTRU and network is depicted, where the media data source is in WTRU.

[0031] Figure 5B An example architecture for split inference between WTRU and the network is depicted, where the media data source is in the network.

[0032] Figure 5C An example is depicted of messages exchanged between the WTRU and network devices to manage dynamic split points.

[0033] Figure 6This is a block diagram depicting an example of information exchange between processing units.

[0034] Figure 7 An example message exchange between a first device and a second device is depicted, wherein each of the first device or the second device is a WTRU or a network device.

[0035] Figure 8 Example methods for parsing and / or processing processing unit types and / or processing unit IDs are described.

[0036] Figure 9 It is a block diagram depicting an example of metadata and / or processing unit information. Detailed Implementation

[0037] A more detailed understanding can be obtained from the following description, which is given by way of example in conjunction with the accompanying drawings.

[0038] Figure 1A This is a diagram illustrating an example communication system 100 in which one or more of the disclosed embodiments may be implemented. Communication system 100 may be a multiple access system providing content such as voice, data, video, messaging, broadcasting, etc., to multiple wireless users. Communication system 100 enables multiple wireless users to access such content by sharing system resources including wireless bandwidth. For example, communication system 100 may employ one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero Tail Unique Word DFT-Spread Spectrum OFDM (ZT UW DTS-s OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtered OFDM, Filter Bank Multicarrier (FBMC), and the like.

[0039] like Figure 1AAs shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106 / 115, public switched telephone network (PSTN) 108, Internet 110, and other networks 112. However, it will be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, and 102d may be any type of device configured to operate and / or communicate in a wireless environment. As examples, WTRUs 102a, 102b, 102c, and 102d, any of which may be referred to as a “station” and / or “STA”, may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of industrial and / or automated processing chains), consumer electronic devices, devices operating on commercial and / or industrial wireless networks, and the like. Any of WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as WTRUs.

[0040] The communication system 100 may also include base station 114a and / or base station 114b. Each of base stations 114a and 114b may be any type of device configured to wirelessly interface with at least one of WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks, such as CN 106 / 115, the Internet 110, and / or other networks 112. As an example, base stations 114a and 114b may be base transceiver stations (BTS), node Bs, eNode Bs, home node Bs, home eNode Bs, gNBs, NR node Bs, site controllers, access points (APs), wireless routers, and the like. Although each of base stations 114a and 114b is depicted as a single element, it will be appreciated that base stations 114a and 114b may include any number of interconnected base stations and / or network elements.

[0041] Base station 114a may be part of RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as base station controllers (BSCs), radio network controllers (RNCs), relay nodes, etc. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. Cells may provide coverage for radio services to a specific geographic area that may be relatively fixed or may change over time. Cells may also be divided into cell sectors. For example, the cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver per sector of the cell. In embodiments, base station 114a may employ multiple-input multiple-output (MIMO) technology and may use multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0042] Base stations 114a and 114b can communicate with one or more of WTRUs 102a, 102b, 102c, and 102d via air interface 116. Air interface 116 can be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). Any suitable radio access technology (RAT) can be used to establish air interface 116.

[0043] More specifically, as described above, the communication system 100 can be a multiple access system and can employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, base stations 114a and WTRUs 102a, 102b, and 102c in RAN 104 / 113 can implement radio technologies such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which can use Wideband CDMA (WCDMA) to establish air interfaces 115 / 116 / 117. WCDMA can include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA can include High-Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0044] In the embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which can use Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-A Pro) to establish air interface 116.

[0045] In the embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as NR radio access, which can use New Radio (NR) to establish air interface 116.

[0046] In the embodiments, base station 114a and WTRUs 102a, 102b, and 102c can implement multiple radio access technologies. For example, base station 114a and WTRUs 102a, 102b, and 102c can, for instance, use the dual connectivity (DC) principle to jointly implement LTE radio access and NR radio access. Therefore, the air interface utilized by WTRUs 102a, 102b, and 102c may be characterized by transmissions to / from multiple types of base stations (e.g., eNBs and gNBs) and / or multiple types of radio access technologies.

[0047] In other embodiments, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as IEEE 802.11 (i.e., Wi-Fi), IEEE 802.16 (i.e., Global Microwave Access Interoperability (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rate Evolution of GSM (EDGE), GSM EDGE (GERAN), and the like.

[0048] Figure 1ABase station 114b can be, for example, a wireless router, a home node B, a home eNode B, or an access point, and can utilize any suitable RAT to facilitate wireless connectivity in a local area, such as a commercial location, home, vehicle, campus, industrial facility, air corridor (e.g., for drone use), road, and the like. In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.11 to establish a wireless local area network (WLAN). In another embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, base station 114b and WTRUs 102c, 102d can utilize cellular-based RATs (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish picocells or femtocells. Figure 1A As shown, base station 114b can have a direct connection to the Internet 110. Therefore, it is not required that base station 114b access the Internet 110 via CN 106 / 115.

[0049] RAN 104 / 113 can communicate with CN 106 / 115, which can be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRUs 102a, 102b, 102c, and 102d. Data can have different Quality of Service (QoS) requirements, such as different throughput requirements, latency requirements, fault tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like. CN 106 / 115 can provide call control, billing services, location-based services, prepaid calling, internet connectivity, video distribution, and / or perform advanced security functions such as user authentication. Although Figure 1A Although not shown, it should be understood that RAN 104 / 113 and / or CN 106 / 115 can communicate directly or indirectly with other RANs that use the same RAT as or a different RAT than RAN 104 / 113. For example, in addition to being connected to RAN 104 / 113, which can utilize NR radio technology, CN 106 / 115 can also communicate with another RAN (not shown) that uses GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0050] CN 106 / 115 can also serve as a gateway for WTRU 102a, 102b, 102c, 102d to access PSTN 108, the Internet 110, and / or other networks 112. PSTN 108 may include a circuit-switched telephone network providing Common Old-Style Telephone Service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices using common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) from the TCP / IP Internet Protocol suite. Network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs, which may use the same RAT as RAN 104 / 113 or a different RAT.

[0051] Some or all of the WTRUs 102a, 102b, 102c, and 102d in the communication system 100 may include multi-mode capabilities (e.g., WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). For example, Figure 1A The WTRU 102c shown can be configured to communicate with a base station 114a that can employ cellular-based radio technology and a base station 114b that can employ IEEE 802 radio technology.

[0052] Figure 1B This is a system diagram illustrating example WTRU 102. (See diagram below.) Figure 1B As shown, WTRU 102 may, among other things, include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a Global Positioning System (GPS) chipset 136, and / or other peripheral devices 138. It should be understood that WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with the embodiments.

[0053] Processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like. Processor 118 may perform signal encoding, data processing, power control, input / output processing, and / or any other functionality that enables WTRU 102 to operate in a wireless environment. Processor 118 may be coupled to transceiver 120, and transceiver 120 may be coupled to transmitting / receiving element 122. Although Figure 1B The processor 118 and transceiver 120 are depicted as separate components, but it should be understood that the processor 118 and transceiver 120 may be integrated together in an electronic package or chip.

[0054] Transmitting / receiving element 122 can be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via air interface 116. For example, in one embodiment, transmitting / receiving element 122 can be an antenna configured to transmit and / or receive RF signals. In another embodiment, transmitting / receiving element 122 can be a transmitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, transmitting / receiving element 122 can be configured to transmit and / or receive both RF signals and optical signals. It should be understood that transmitting / receiving element 122 can be configured to transmit and / or receive any combination of wireless signals.

[0055] Although the transmitting / receiving element 122 is in Figure 1B While depicted as a single element, WTRU 102 may include any number of transmit / receive elements 122. More specifically, WTRU 102 may employ MIMO technology. Thus, in one embodiment, WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals via air interface 116.

[0056] Transceiver 120 can be configured to modulate signals to be transmitted by transmitting / receiving element 122 and demodulate signals received by transmitting / receiving element 122. As described above, WTRU 102 can have multi-mode capability. Therefore, transceiver 120 can include multiple transceivers for enabling WTRU 102 to communicate via various RATs, such as, for example, NR and IEEE 802.11.

[0057] The processor 118 of WTRU 102 may be coupled to a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) unit or an organic light-emitting diode (OLED) display unit) and may receive user input data therefrom. The processor 118 may also output user data to the speaker / microphone 124, keypad 126, and / or display / touchpad 128. Additionally, the processor 118 may access information and store data therein from any type of suitable memory, such as non-removable memory 130 and / or removable memory 132. Non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. Removable memory 132 may include a subscriber identity module (SIM) card, memory stick, secure digital storage (SD) card, and the like. In other embodiments, the processor 118 may access information and store data in memory that is not physically located on WTRU 102, such as on a server or home computer (not shown).

[0058] The processor 118 may receive power from the power supply 134 and may be configured to distribute and / or control power to other components in the WTRU 102. The power supply 134 may be any suitable device for powering the WTRU 102. For example, the power supply 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0059] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) about the current location of the WTRU 102. In addition to or instead of information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) via air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It should be understood that the WTRU 102 may acquire location information using any suitable location determination method while remaining consistent with the embodiments.

[0060] The processor 118 may also be coupled to other peripheral devices 138, which may include one or more software and / or hardware modules providing additional features, functionality, and / or wired or wireless connectivity. For example, peripheral devices 138 may include accelerometers, electronic compasses, satellite transceivers, digital cameras (for photos and / or video), Universal Serial Bus (USB) ports, vibration devices, television transceivers, hands-free headsets, and Bluetooth. ® Modules, FM radio units, digital music players, media players, video game player modules, internet browsers, virtual reality and / or augmented reality (VR / AR) devices, activity trackers, and the like. Peripheral device 138 may include one or more sensors. Sensors may be one or more of the following: gyroscope, accelerometer, Hall effect sensor, magnetometer, orientation sensor, proximity sensor, temperature sensor, time sensor; geolocation sensor; altimeter, light sensor, touch sensor, magnetometer, barometer, gesture sensor, biometric sensor, and / or humidity sensor.

[0061] WTRU 102 may include a full-duplex radio for which transmission and reception of some or all signals associated with a specific subframe for both UL (e.g., for transmission) and downlink (e.g., for reception) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit to reduce and / or substantially eliminate self-interference via hardware (e.g., a choke) or via a processor (e.g., a separate processor (not shown) or via processor 118). In embodiments, WTRU 102 may include a half-duplex radio for which transmission and reception of some or all signals associated with a specific subframe for either UL (e.g., for transmission) or downlink (e.g., for reception) may be concurrent and / or simultaneous.

[0062] Figure 1C This is a system diagram illustrating RAN 104 and CN 106 according to an embodiment. As described above, RAN 104 can use E-UTRA radio technology to communicate with WTRUs 102a, 102b, and 102c via air interface 116. RAN 104 can also communicate with CN 106.

[0063] RAN 104 may include eNode-B 160a, 160b, 160c, but will be appreciated that RAN 104 may include any number of eNode-Bs while remaining consistent with the embodiments. Each of eNode-B 160a, 160b, 160c may include one or more transceivers for communicating with WTRU 102a, 102b, 102c via air interface 116. In one embodiment, eNode-B 160a, 160b, 160c may implement MIMO technology. Thus, for example, eNode-B 160a may use multiple antennas to transmit radio signals to and / or receive radio signals from WTRU 102a.

[0064] Each of the eNode-B 160a, 160b, and 160c can be associated with a specific cell (not shown) and can be configured to handle radio resource management decisions, handover decisions, user scheduling in the UL and / or DL, and the like. Figure 1C As shown, eNode-B 160a, 160b, and 160c can communicate with each other via the X2 interface.

[0065] Figure 1C The CN 106 shown may include a Mobility Management Entity (MME) 162, a Serving Gateway (SGW) 164, and a Packet Data Network (PDN) Gateway (or PGW) 166. While each of the foregoing elements is depicted as part of CN 106, it should be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0066] The MME 162 can connect to each of the eNode-Bs 162a, 162b, and 162c in RAN 104 via the S1 interface and can be used as a control node. For example, the MME 162 can be responsible for authenticating users of WTRUs 102a, 102b, and 102c, bearer activation / deactivation, selecting a specific serving gateway during the initial attachment of WTRUs 102a, 102b, and 102c, and the like. The MME 162 can provide control plane functions for handover between RAN 104 and other RANs (not shown) employing other radio technologies such as GSM and / or WCDMA.

[0067] The SGW 164 can connect to each of the eNode Bs 160a, 160b, and 160c in RAN 104 via the S1 interface. The SGW 164 can typically route and forward user data packets to / from WTRUs 102a, 102b, and 102c. The SGW 164 can perform other functions such as anchoring the user plane during inter-eNode B handover, triggering paging when DL data is available for WTRUs 102a, 102b, and 102c, managing and storing the context of WTRUs 102a, 102b, and 102c, and so on.

[0068] The SGW 164 can connect to the PGW 166, which can provide WTRU 102a, 102b, and 102c with access to packet-switched networks such as the Internet 110, thereby facilitating communication between WTRU 102a, 102b, and 102c and IP-enabled devices.

[0069] CN 106 can facilitate communication with other networks. For example, CN 106 can provide WTRUs 102a, 102b, and 102c with access to circuit-switched networks such as PSTN 108 to facilitate communication between WTRUs 102a, 102b, and 102c and traditional landline communication equipment. For example, CN 106 may include, or be able to communicate with, an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that serves as an interface between CN 106 and PSTN 108. Furthermore, CN 106 can provide WTRUs 102a, 102b, and 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers.

[0070] Despite Figure 1A-1D While the WTRU is described as a wireless terminal, it is conceivable that in some representative embodiments, such a terminal may (e.g., temporarily or permanently) use a wired communication interface with a communication network.

[0071] In a representative embodiment, the other network 112 may be a WLAN.

[0072] In Infrastructure Basic Services Set (BSS) mode, a WLAN may have an access point (AP) for the BSS and one or more stations (STAs) associated with the AP. The AP may have access or an interface to a distribution system (DS) or another type of wired / wireless network that carries traffic into and / or out of the BSS. Traffic originating outside the BSS and destined for a STA can reach and be delivered to the STA via the AP. Traffic originating from a STA and destined for a destination outside the BSS can be sent to the AP for delivery to the appropriate destination. Traffic between STAs within the BSS can be sent via the AP, for example, where a source STA can send traffic to the AP, and the AP can deliver the traffic to the destination STA. Traffic between STAs within the BSS can be considered and / or referred to as peer-to-peer traffic. Peer-to-peer traffic can be sent between a source STA and a destination STA (e.g., directly between the source STA and the destination STA) using Direct Link Establishment (DLS). In some representative embodiments, the DLS may use 802.11e DLS or 802.11z Tunneling DLS (TDLS). WLANs using Standalone BSS (IBSS) mode may not have access points (APs), and STAs within the IBSS or using the IBSS (e.g., all STAs) can communicate directly with each other. The IBSS communication mode may sometimes be referred to as the "ad-hoc" communication mode in this document.

[0073] When operating in 802.11ac infrastructure mode or a similar mode, the AP can transmit beacons on a fixed channel, such as the primary channel. The primary channel can be of fixed width (e.g., a bandwidth of 20 MHz) or dynamically set via signaling. The primary channel can be the operating channel of the BSS and can be used by the STA to establish a connection with the AP. In some representative embodiments, carrier-sense multiple access (CSMA / CA) with collision avoidance can be implemented, for example, in an 802.11 system. For CSMA / CA, each STA, including the AP, can sense the primary channel. If the primary channel is sensed / detected and / or determined to be busy by a particular STA, that particular STA can back off. A single STA (e.g., only one station) can transmit at any given time within a given BSS.

[0074] High-throughput (HT) STAs can communicate using a 40 MHz wide channel, for example, by combining a primary 20 MHz channel with adjacent or non-adjacent 20 MHz channels.

[0075] Very High Throughput (VHT) STAs can support wide channels of 20 MHz, 40 MHz, 80 MHz, and / or 160 MHz. 40 MHz and / or 80 MHz channels can be formed by combining consecutive 20 MHz channels. A 160 MHz channel can be formed by combining eight consecutive 20 MHz channels or by combining two non-consecutive 80 MHz channels; this can be referred to as an 80+80 configuration. For the 80+80 configuration, after channel coding, the data passes through a segmented parser, which divides the data into two streams. Inverse Fast Fourier Transform (IFFT) processing and time-domain processing can be performed separately for each stream. The streams can be mapped onto two 80 MHz channels, and the data can be transmitted by the transmitting STA. At the receiver of the receiving STA, the above operations for the 80+80 configuration can be reversed, and the combined data can be sent to the Media Access Control (MAC).

[0076] The sub-1 GHz operating mode is supported by 802.11af and 802.11ah. The channel operating bandwidth and carrier are reduced in 802.11af and 802.11ah compared to those used in 802.11n and 802.11ac. 802.11af supports 5 MHz, 10 MHz, and 20 MHz bandwidths in the TV white space (TVWS) spectrum, and 802.11ah supports 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz bandwidths using non-TVWS spectrum. According to a representative embodiment, 802.11ah may support instrument-type control / machine-type communication, such as MTC devices in macro coverage areas. MTC devices may have certain capabilities, such as limited capabilities, including support (e.g., only support) certain and / or limited bandwidths. MTC devices may include batteries with a battery life exceeding a threshold (e.g., to maintain a very long battery life).

[0077] Multiple channels and channel bandwidths can be supported. WLAN systems such as 802.11n, 802.11ac, 802.11af, and 802.11ah include channels that can be designated as the primary channel. The primary channel can have a bandwidth equal to the maximum common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel can be set and / or limited by the STAs supporting the minimum bandwidth operating mode among all STAs operating in the BSS. In the 802.11ah example, for STAs that support (e.g., only support) the 1 MHz mode (e.g., MTC type devices), the primary channel can be 1 MHz wide, even if the AP and other STAs in the BSS support 2 MHz, 4 MHz, 8 MHz, 16 MHz, and / or other channel bandwidth operating modes. Carrier sensing and / or Network Allocation Vector (NAV) settings can depend on the status of the primary channel. If the primary channel is busy, for example, because an STA (which only supports the 1 MHz operating mode) is transmitting to the AP, then all available frequency bands can be considered busy even if most frequency bands remain idle and are likely available.

[0078] In the United States, the available frequency bands for 802.11ah are from 902 MHz to 928 MHz. In South Korea, the available frequency bands are from 917.5 MHz to 923.5 MHz. In Japan, the available frequency bands are from 916.5 MHz to 927.5 MHz. Depending on the country code, the total bandwidth available for 802.11ah ranges from 6 MHz to 26 MHz.

[0079] Figure 1D This is a system diagram illustrating RAN 113 and CN 115 according to an embodiment. As described above, RAN 113 can employ NR radio technology to communicate with WTRUs 102a, 102b, and 102c via air interface 116. RAN 113 can also communicate with CN 115.

[0080] RAN 113 may include gNBs 180a, 180b, and 180c; however, it should be understood that RAN 113 may include any number of gNBs while remaining consistent with the embodiments. Each of gNBs 180a, 180b, and 180c may include one or more transceivers for communicating with WTRUs 102a, 102b, and 102c via air interface 116. In one embodiment, gNBs 180a, 180b, and 180c may implement MIMO technology. For example, gNBs 180a and 180b may utilize beamforming to transmit signals to and / or receive signals from gNBs 180a, 180b, and 180c. Therefore, gNB 180a may, for example, use multiple antennas to transmit radio signals to and / or receive radio signals from WTRU 102a. In embodiments, gNBs 180a, 180b, and 180c can implement carrier aggregation technology. For example, gNB 180a can transmit multiple component carriers to WTRU 102a (not shown). A subset of these component carriers can be on unlicensed spectrum, while the remaining component carriers can be on licensed spectrum. In embodiments, gNBs 180a, 180b, and 180c can implement Cooperative Multipoint (CoMP) technology. For example, WTRU 102a can receive cooperative transmissions from gNBs 180a and 180b (and / or gNB 180c).

[0081] WTRUs 102a, 102b, and 102c can communicate with gNBs 180a, 180b, and 180c using transmissions associated with scalable numberology. For example, OFDM symbol spacing and / or OFDM subcarrier spacing can vary for different transmissions, different cells, and / or different portions of the radio transmission spectrum. WTRUs 102a, 102b, and 102c can communicate with gNBs 180a, 180b, and 180c using subframes or transmission time intervals (TTIs) of various or scalable lengths (e.g., containing different numbers of OFDM symbols and / or absolute times of varying durations).

[0082] gNBs 180a, 180b, and 180c can be configured to communicate with WTRUs 102a, 102b, and 102c in standalone and / or non-standalone configurations. In standalone configuration, WTRUs 102a, 102b, and 102c can communicate with gNBs 180a, 180b, and 180c without also accessing other RANs (e.g., eNode-Bs 160a, 160b, and 160c). In standalone configuration, WTRUs 102a, 102b, and 102c can utilize one or more gNBs 180a, 180b, and 180c as mobility anchors. In standalone configuration, WTRUs 102a, 102b, and 102c can use signals in unlicensed frequency bands to communicate with gNBs 180a, 180b, and 180c. In a non-standalone configuration, WTRUs 102a, 102b, and 102c can communicate with / connect to gNBs 180a, 180b, and 180c, while also communicating with / connecting to another RAN such as eNode-Bs 160a, 160b, and 160c. For example, WTRUs 102a, 102b, and 102c can implement DC principles to communicate substantially simultaneously with one or more gNBs 180a, 180b, and 180c and one or more eNode-Bs 160a, 160b, and 160c. In a non-standalone configuration, eNode-B 160a, 160b, and 160c can be used as mobility anchors for WTRU 102a, 102b, and 102c, and gNB 180a, 180b, and 180c can provide additional coverage and / or throughput for serving WTRU 102a, 102b, and 102c.

[0083] Each of gNBs 180a, 180b, and 180c can be associated with a specific cell (not shown) and can be configured to handle radio resource management decisions, handover decisions, user scheduling in UL and / or DL, network slicing support, dual connectivity, interoperability between NR and E-UTRA, routing of user plane data to User Plane Functions (UPF) 184a and 184b, routing of control plane information to Access and Mobility Management Functions (AMF) 182a and 182b, and the like. Figure 1D As shown, gNB 180a, 180b, and 180c can communicate with each other via the Xn interface.

[0084] Figure 1DThe CN 115 shown may include at least one AMF 182a, 182b, at least one UPF 184a, 184b, at least one Session Management Function (SMF) 183a, 183b, and possibly a Data Network (DN) 185a, 185b. While each of the foregoing elements is depicted as part of the CN 115, it should be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0085] AMF 182a and 182b can connect to one or more of gNB 180a, 180b, and 180c in RAN 113 via the N2 interface and can be used as control nodes. For example, AMF 182a and 182b can be responsible for authenticating users of WTRU 102a, 102b, and 102c, supporting network slicing (e.g., handling different PDU sessions with different requirements), selecting specific SMF 183a and 183b, managing registration areas, terminating NAS signaling, mobility management, and the like. AMF 182a and 182b can use network slicing to customize CN support for WTRU 102a, 102b, and 102c based on the type of service being used by WTRU 102a, 102b, and 102c. For example, different network slices can be established for different use cases, such as services relying on Ultra Reliable Low Latency (URLLC) access, services relying on Enhanced Massive Mobile Broadband (eMBB) access, services for Machine Type Communication (MTC) access, and / or the like. AMF 182a and 182b provide control plane functions for switching between RAN 113 and other RANs (not shown) that employ other radio technologies, such as LTE, LTE-A, LTE-A Pro, and / or non-3GPP access technologies, such as WiFi.

[0086] SMFs 183a and 183b can connect to AMFs 182a and 182b in CN 115 via the N11 interface. SMFs 183a and 183b can also connect to UPFs 184a and 184b in CN 115 via the N4 interface. SMFs 183a and 183b can select and control UPFs 184a and 184b, and configure the routing of traffic through UPFs 184a and 184b. SMFs 183a and 183b can perform other functions such as managing and allocating WTRU IP addresses, managing PDU sessions, controlling policy enforcement and QoS, providing downlink data notifications, and so on. PDU session types can be IP-based, non-IP-based, Ethernet-based, and so on.

[0087] UPF 184a and 184b can be connected via an N3 interface to one or more gNBs 180a, 180b, and 180c in RAN 113. This N3 interface provides WTRU 102a, 102b, and 102c with access to packet-switched networks, such as the Internet 110, to facilitate communication between WTRU 102a, 102b, and 102c and IP-enabled devices. UPF 184 and 184b can perform other functions such as routing and forwarding packets, enforcing user plane policies, supporting multi-homed PDU sessions, handling user plane QoS, buffering downlink packets, providing mobility anchoring, and so on.

[0088] CN 115 can facilitate communication with other networks. For example, CN 115 may include, or be able to communicate with, an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) serving as an interface between CN 115 and PSTN 108. Furthermore, CN 115 can provide WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers. In one embodiment, WTRUs 102a, 102b, 102c may be connected to local data networks (DNs) 185a, 185b via the N3 interface to UPFs 184a, 184b and the N6 interface between UPFs 184a, 184b and DNs 185a, 185b.

[0089] Given Figure 1A-1D and to Figure 1A-1D The corresponding descriptions herein refer to the following: WTRU 102a-d, Base Station 114a-b, eNode-B 160a-c, MME 162, SGW 164, PGW 166, gNB 180a-c, AMF 182a-b, UPF 184a-b, SMF 183a-b, DN 185a-b, and / or one or more other devices described herein. One or more of the functions described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more of the functions described herein. For example, an emulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0090] Simulation devices can be designed to perform tests on one or more other devices in a laboratory environment and / or in a carrier network environment. For example, one or more simulation devices may perform one or more or all of their functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices within the communication network. One or more simulation devices may perform one or more or all of their functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. Simulation devices may be directly coupled to another device for testing purposes and / or may be used to perform tests via over-the-air wireless communication.

[0091] One or more simulation devices may perform one or more (including all) functions without being implemented / deployed as part of a wired and / or wireless communication network. For example, simulation devices may be used in test environments within test laboratories and / or non-deployed (e.g., testing) wired and / or wireless communication networks to perform testing of one or more components. One or more simulation devices may be test equipment. Simulation devices may transmit and / or receive data using direct RF coupling and / or wireless communication via RF circuitry (e.g., which may include one or more antennas).

[0092] This application describes several aspects, including tools, features, examples, models, methods, etc. Many of these aspects are described in a specific manner, and many of these aspects, which at least demonstrate individual characteristics, are generally described in a way that may sound restrictive. However, this is for the purpose of clarity and does not limit the application or scope of those aspects. In fact, all the different aspects can be combined and interchanged to provide other aspects. Furthermore, these aspects can also be combined and interchanged with those described in earlier filings.

[0093] The aspects described and contemplated in this application can be implemented in many different forms. Figures 5-9 described herein provide some examples, but other examples are contemplated. The discussion of Figures 5-9 does not limit the breadth of implementations. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to the transmission of generated or encoded bitstreams. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having bitstreams generated according to any of the described methods stored thereon.

[0094] In this application, the terms “reconstructed” and “decoded” are used interchangeably, the terms “pixel” and “sample” are used interchangeably, and the terms “image”, “picture” and “frame” are used interchangeably.

[0095] This document describes various methods, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Additionally, terms such as "first," "second," etc., may be used in various examples to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless specifically required, the use of such terms does not imply an ordering of the modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding and may occur, for example, before, during, or in a time period overlapping with the second decoding.

[0096] The various methods and other aspects described in this application can be used to modify, for example... Figure 2 and Figure 3 The video encoder 200 and decoder 300 shown herein are modules, such as a decoding module. Furthermore, the subject matter disclosed herein can be applied to, for example, any type, format, or version of video encoding, whether described in standards or recommendations, whether pre-existing or future-developed, and any extensions to such standards and recommendations. Unless otherwise stated or technically excluded, the aspects described in this application may be used alone or in combination.

[0097] Various numerical values ​​are used in the examples described in this application. These and other specific values ​​are for illustrative purposes and the aspects described are not limited to these specific values.

[0098] Figure 2 This is a diagram illustrating an example video encoder. Variations of the example encoder 200 are envisioned, but for clarity, encoder 200 is described below, without describing all anticipated variations.

[0099] Before being encoded, the video sequence may undergo pre-coding (201), such as applying color transformations to the input color image (e.g., a conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing remapping of the input image components to obtain a signal distribution more resilient to compression (e.g., using histogram equalization with one of the color components). Metadata may be associated with pre-processing and attached to the bitstream.

[0100] In encoder 200, the image is encoded by encoder elements as described below. The image to be encoded is segmented (202) and processed in units, for example, coding units (CUs). Each unit is encoded using, for example, intra-frame or inter-frame modes. When a unit is encoded in intra-frame mode, intra-frame prediction (260) is performed. In inter-frame mode, motion estimation (275) and compensation (270) are performed. The encoder determines (205) which of the intra-frame or inter-frame modes is used to encode the unit and indicates the intra-frame / inter-frame decision by, for example, a prediction mode flag. For example, the prediction residual is calculated by subtracting (210) the prediction block from the original image block.

[0101] The predicted residual is then transformed (225) and quantized (230). The quantized transform coefficients, along with the motion vector and other syntax elements, are entropy encoded (245) to output a bitstream. The encoder can skip the transform and apply the quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is directly encoded without applying either the transform or quantization process.

[0102] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and prediction block are combined (255) to reconstruct the image block. An in-loop filter (265) is applied to the reconstructed image to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coded artifacts. The filtered image is stored at the reference image buffer (280).

[0103] Figure 3 This is a diagram illustrating an example video decoder. In the example decoder 300, the bitstream is decoded by decoder elements, as described below. The video decoder 300 typically performs operations similar to... Figure 2 The decoding passes are the reciprocal of the encoding passes described in the document. Encoder 200 typically also performs video decoding as part of the encoded video data.

[0104] Specifically, the input to the decoder includes a video bitstream that can be generated by the video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, motion vectors, and other encoded information. Image segmentation information indicates how to segment the image. Therefore, the decoder can segment (335) the image based on the decoded image segmentation information. The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residual. The decoded prediction residual and prediction block are combined (355) to reconstruct the image block. The prediction block (370) can be obtained from intra-frame prediction (360) or motion-compensated prediction (i.e., inter-frame prediction) (375). An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored at a reference image buffer (380).

[0105] The decoded image may undergo further post-decoding processing (385), such as inverse color transformation (e.g., conversion from YCbCr4:2:0 to RGB 4:4:4) or inverse remapping of the remapping process performed in pre-encoding processing (201). Post-decoding processing may utilize metadata derived in pre-encoding processing and signaled in the bitstream. In the example, the decoded image (e.g., after applying an in-loop filter (365) and / or after post-decoding processing (385), if post-decoding processing is used) may be sent to a display device for presentation to the user.

[0106] Figure 4 This is a diagram illustrating examples of systems in which the various aspects and examples described herein can be implemented. System 400 may be embodied as a device including the various components described below and configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 400 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one example, the processing and encoder / decoder elements of system 400 are distributed across multiple ICs and / or discrete components. In various examples, system 400 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various examples, system 400 is configured to implement one or more aspects described in this document.

[0107] System 400 includes at least one processor 410 configured to execute instructions loaded thereon for implementing various aspects, such as those described in this document. Processor 410 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 400 includes at least one memory 420 (e.g., a volatile memory device and / or a non-volatile memory device). System 400 includes a storage device 440, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 440 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0108] System 400 includes an encoder / decoder module 430 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 430 may include its own processor and memory. The encoder / decoder module 430 represents one or more modules that may be included in a device to perform encoding and / or decoding functions. It is well known that a device may include one or both encoding and decoding modules. Alternatively, the encoder / decoder module 430 may be implemented as a separate element of system 400, or it may be incorporated into processor 410 as a combination of hardware and software known to those skilled in the art.

[0109] Program code to be loaded onto processor 410 or encoder / decoder 430 to execute the various aspects described in this document may be stored in storage device 440 and subsequently loaded onto memory 420 for execution by processor 410. Depending on various examples, one or more of processor 410, memory 420, storage device 440, and encoder / decoder module 430 may store one or more various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0110] In some examples, the memory within processor 410 and / or encoder / decoder module 430 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other examples, external memory (e.g., the processing device could be processor 410 or encoder / decoder module 430) is used for one or more of these functions. External memory can be memory 420 and / or storage device 440, such as volatile memory and / or non-volatile flash memory. In several examples, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one example, fast external volatile memory such as RAM is used as working memory for video encoding and decoding operations.

[0111] As indicated in box 445, inputs to the components of system 400 can be provided through various input devices. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives RF signals transmitted over the air, for example by a broadcaster, (ii) component (COMP) input terminals (or a collection of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 4 Other examples not shown include composite video.

[0112] In various examples, the input device of block 445 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or band-limiting a signal to a certain frequency band), (ii) down-converting the selected signal, (iii) band-limiting it again to a narrower frequency band to select a signal band that, in some examples, may be referred to as a channel, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and / or (vi) demultiplexing to select a desired data packet stream. The RF section of various examples includes one or more elements for performing these functions, such as a frequency selector, signal selector, band limiter, channel selector, filter, downconverter, demodulator, error corrector, and demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box example, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to the desired frequency band. Various examples rearrange the order of the above (and other) components, remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as, for example, inserting amplifiers and analog-to-digital converters. In various examples, the RF section includes an antenna.

[0113] USB and / or HDMI terminals may include corresponding interface processors for connecting system 400 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented as needed, for example, within a separate input processing IC or within processor 410. Similarly, various aspects of USB or HDMI interface processing may be implemented as needed, either within a separate interface IC or within processor 410. Demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 410 and encoder / decoder 430, which operate in combination with memory and storage elements to process the data streams as needed for presentation on the output device.

[0114] Various components of system 400 can be provided within an integrated housing, in which the various components can be interconnected and transmit data between them using a suitable connection arrangement 425, such as an internal bus known in the art, including inter-IC (I2C) bus, wiring and printed circuit board.

[0115] System 400 includes a communication interface 450 that enables communication with other devices via a communication channel 460. The communication interface 450 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 460. The communication interface 450 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 460 may be implemented, for example, within a wired and / or wireless medium.

[0116] In various examples, data is streamed or otherwise provided to system 400 using a wireless network such as a Wi-Fi network, for example, IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). Wi-Fi signals in these examples are received via a communication channel 460 and a communication interface 450 suitable for Wi-Fi communication. The communication channel 460 in these examples is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other examples use a set-top box to provide streaming data to system 400, delivering data via an HDMI connection in input box 445. Still other examples use an RF connection in input box 445 to provide streaming data to system 400. As mentioned above, various examples provide data in non-streaming methods. Additionally, various examples use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth® networks.

[0117] System 400 can provide output signals to various output devices, including a display 475, a speaker 485, and other peripheral devices 495. Various examples of the display 475 include one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a flexible display, and / or a foldable display. The display 475 can be used in televisions, tablet computers, laptop computers, cellular phones (mobile phones), or other devices. The display 475 can also be integrated with other components (e.g., as in a smartphone) or separate (e.g., as in an external monitor for a laptop computer). In various examples, other peripheral devices 495 include one or more of a standalone digital video disc (or digital multifunction disc) (DVD, for both terms), a disc player, a stereo system, and / or a lighting system. Various examples utilize one or more peripheral devices 495 that provide functionality based on the output of system 400. For example, a disc player performs the function of playing the output of system 400.

[0118] In various examples, signaling such as AV.Link, Consumer Electronic Control (CEC), or other communication protocols enabling device-to-device control with or without user intervention is used to transmit control signals between system 400 and display 475, speaker 485, or other peripheral devices 495. Output devices can be communicatively coupled to system 400 via dedicated connections through corresponding interfaces 470, 480, and 490. Alternatively, output devices can be connected to system 400 via communication interface 450 using communication channel 460. Display 475 and speaker 485 can be integrated into a single unit with other components of system 400 in electronic devices such as, for example, televisions. In various examples, display interface 470 includes display drivers, such as, for example, a timing controller (TCon) chip.

[0119] Display 475 and speaker 485 may alternatively be separated from one or more other components, for example, if the RF section of input 445 is part of a separate set-top box. In various examples where display 475 and speaker 485 are external components, output signals may be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0120] These examples can be executed by computer software implemented by processor 410, or by hardware, or a combination of hardware and software. As a non-limiting example, the examples can be implemented by one or more integrated circuits. Memory 420 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. As a non-limiting example, processor 410 can be of any type suitable for the technical environment and can include one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0121] Various implementations involve decoding. As used in this application, "decoding" can encompass all or part of a process performed, for example, on a received encoded sequence to produce a final output suitable for display. In various examples, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding.

[0122] As another example, in one example, "decoding" refers only to entropy decoding; in another example, "decoding" refers only to differential decoding; and in yet another example, "decoding" refers to a combination of entropy decoding and differential decoding. It will be clear, and is considered well understood by those skilled in the art, whether the phrase "decoding process" is intended to specifically refer to a subset of operations or generally to the broader decoding process, based on the specific context of the description.

[0123] Various implementations involve encoding. In a manner similar to the discussion above regarding “decoding,” the term “encoding” as used in this application can encompass all or part of the process performed, for example, on an input video sequence to produce an encoded bitstream. In various examples, such a process includes one or more processes typically performed by an encoder, such as segmentation, differential coding, transform, quantization, and entropy coding.

[0124] As another example, in one example, "encoding" refers only to entropy encoding; in another example, "encoding" refers only to differential encoding; and in yet another example, "encoding" refers to a combination of differential and entropy encoding. It will be clear, and is considered well understood by those skilled in the art, whether the phrase "encoding process" is intended to specifically refer to a subset of operations or generally to a broader encoding process, depending on the context of the specific description.

[0125] Note that the syntax elements used in this article are descriptive terms, such as encoding syntax related to host senders, message types, split points, etc. Therefore, they do not preclude the use of other syntax element names.

[0126] When the accompanying drawings are presented as flowcharts, it should be understood that they also provide block diagrams of the corresponding apparatus. Similarly, when the accompanying drawings are presented as block diagrams, it should be understood that they also provide flowcharts of the corresponding methods / processes.

[0127] The implementations and aspects described herein can be implemented, for example, in a method or process, apparatus, software program, data stream, or signal. Even if discussed only in the context of a single implementation form (e.g., discussed only as a method), the implementation of the features in question can also be implemented in other forms (e.g., apparatus or program). An apparatus can be implemented, for example, in appropriate hardware, software, and firmware. A method can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. A processor also includes communication devices, such as, for example, a computer, a cellular phone, a portable / personal digital assistant (“PDA”), and other devices that facilitate information communication between end users.

[0128] References to “an example” or “an example” or “an implementation” or “an implementation” and their variations mean that a particular feature, structure, characteristic, and such is included in at least one example in connection with the example. Therefore, the phrases “in an example” or “in the example” or “in an implementation” or “in the implementation” appearing throughout this application and any other variations do not necessarily refer to the same example.

[0129] Additionally, this application may involve "determining" various types of information. Determining information may include, for example, one or more of estimation information, calculation information, prediction information, or information retrieved from memory. Obtaining may include receiving, retrieving, constructing, generating, and / or determining.

[0130] Furthermore, this application may involve "accessing" various types of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or one or more of these.

[0131] Additionally, this application may involve "receiving" various types of information. Like "access," receiving is intended to be a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) or one or more of it. Furthermore, "receiving" is generally involved in one or more ways during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0132] It should be understood that, for example, the use of any of the above " / ", "and / or", and "...at least one of A and B" in the cases of "A / B", "A and / or B", and "at least one of A and B" is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the cases of "A, B and / or C" and "at least one of A, B, and C", this wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many items as listed.

[0133] Furthermore, as used herein, the word "signal" refers, among other things, to instructing something to the corresponding decoder. In this way, in the examples, the same parameter can be used on both the encoder and decoder sides. Thus, for example, the encoder can transmit (explicit signaling) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various examples by avoiding the transmission of any actual functionality. It should be understood that signaling can be implemented in many ways. For example, in various examples, one or more syntax elements, flags, and the like are used to signal information to the corresponding decoder. While the foregoing refers to the verb form of the word "signal," the word "signal" can also be used as a noun in this document.

[0134] It will be apparent to those skilled in the art that implementations can generate various signals that are formatted to carry information, for example, that can be stored or transmitted. This information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bit stream of the described example. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, signals can be transmitted via a variety of different wired or wireless links. Signals may be stored on, or accessed or received from, a processor-readable medium.

[0135] This document describes numerous examples. Features of the examples may be provided individually or in any combination across various claim classes and types. Furthermore, examples may include one or more of the features, devices, or aspects described herein, individually or in any combination across various claim classes and types. For example, features described herein may be implemented in a bitstream or signal including information generated as described herein. This information may allow a decoder to decode the bitstream, an encoder, a bitstream, and / or a decoder according to any of the described embodiments. For example, features described herein may be implemented by creating and / or transmitting and / or receiving and / or decoding a bitstream or signal. For example, features described herein may be implemented as a method, process, apparatus, medium storing instructions, medium storing data, or signal. For example, features described herein may be implemented by a TV, set-top box, cellular phone, tablet computer, or other electronic device performing decoding. The TV, set-top box, cellular phone, tablet computer, or other electronic device may display (e.g., using a monitor, screen, or other type of display) the resulting image (e.g., an image reconstructed from a residual of a video bitstream). The TV, set-top box, cellular phone, tablet computer, or other electronic device may receive a signal including an encoded image and perform decoding.

[0136] As used in this article, WTRU can refer to User Equipment (UE).

[0137] Figure 5A and 5B An example architecture (e.g., 3GPP SA4 AI4media) for split inference is described for a model (e.g., an artificial intelligence and / or machine learning (AI / ML model) consisting of n layers (1…n) between the network and WTRU). The first inference may process the first part of the model (e.g., the first part) (e.g., layers 1…k), and / or the second inference may process the second part of the model (e.g., the second part) (e.g., layers k+1…n).

[0138] like Figure 5A and 5B The example architecture described may include the delivery and access capabilities of intermediate data between the WTRU and the network (e.g., for example, for...). Figures 5A-5B The two different scenarios shown. Figure 5A An example split inference between WTRU and the network is depicted, where the media data source is in WTRU. Figure 5B An example split inference between WTRU and the network is depicted, where the media data source is in the network.

[0139] In the example, such as Figure 5AAs shown, the media data source can originate from a WTRU associated with the network. If / when the media data source originates from a WTRU, the first part of the inference can be performed by the WTRU, and the second part of the inference can be performed by the network. The resulting output data can be sent back to the WTRU (e.g., from the network).

[0140] In the example, such as Figure 5B As shown, the media data source can originate from a network. The media data source can be received from the network by the WTRU. If / when the media data source is received from the network or via the WTRU, the first part of the AI / ML model can be executed by the network (e.g., on the network side), and the second part can be executed by the WTRU (e.g., on the WTRU).

[0141] WTRU and / or the network can select the type of processing unit on which the model (e.g., AI / ML model) will run (e.g., if / when a model to be run is selected). The type of processing unit can be CPU, TPU, GPU, FPGA and / or at least one of these.

[0142] The first part of the AI / ML model can be compiled and / or loaded into a machine learning framework. The machine learning framework can be configured and / or be able to infer the input media data. The resulting intermediate data can be passed to the second part of the AI / ML model (e.g., to complete the inference).

[0143] Messages can be defined to wrap data exchange within an AI / ML split pipeline. These messages may include (e.g., consist of) information following intermediate data. This information can assist remote nodes (e.g., the network) in processing the intermediate data (e.g., by indicating one or more of the following: the model to use, the split point to set, the amount of intermediate data to be copied, or how the intermediate data may have been encoded).

[0144] Information and intermediate data can be transmitted via data structures. Example data structures (e.g., for transmitting information and / or intermediate data) may include at least one of the following: a model identifier (e.g., Model_RefID), an indication of whether the split point has changed from a previous message (e.g., ChangeOfSplitPoint), a split point used by the WTRU (e.g., SplitPointID), the next split point used by the WTRU (e.g., NextSplitPointID), the length of the intermediate data (e.g., IntermediateDataLength), the array dimension of the intermediate data (e.g., IntermediateDataDim), an indication of whether the data is compressed and / or the compression algorithm used (e.g., EncodingMethod), a sequence identifier of the input data (e.g., SequenceNumber), a timestamp of the intermediate data (e.g., TimeStamp), or the intermediate data itself (e.g., IntermediateData). Example data structures (e.g., for transmitting information and intermediate data) may be represented as follows: Current IntermediateDataWrapper: Struct IntermediateDataWrapper { int Model_RefID; # Model identifier bit ChangeOfSplitPoint; # Set to 1 if the splitpoint has changed from a previous message. int SplitPointID; # Split point used by WTRU int NextSplitpointID; # The next split point used by WTRU (optional, for example). int IntermediateDataLength; # Length of the intermediate data Dim IntermediateDataDim; # Array dimension of intermediate data byte EncodingMethod; # Indicates whether the data is compressed and which compression algorithm is used. int SequenceNumber; # Sequence identifier for input data double TimeStamp; # Timestamp of intermediate data Byte IntermediateData[] # Intermediate data}; In the example of in-band intermediate data transmission, the transmission information can be transmitted together with the intermediate data in-band (e.g., on the same channel).

[0145] In the example, information can be transmitted from intermediate data out-of-band (e.g., separately) (e.g., on another channel).

[0146] Figure 5C An example of dynamic split point management is described. Messages can be exchanged between the WTRU and network devices (e.g., operators, AI / ML application providers, and / or the like) to manage dynamic split points. A local decision module (e.g., within the WTRU) can be used to set the optimal split point based on requirements (e.g., bandwidth, latency, energy, and / or the like), local information, and / or the range of split points.

[0147] A local decision module included as part of a first device (e.g., a WTRU, network device, and / or the like) or a decision module included as part of a second device (e.g., a WTRU, network device, and / or the like) (e.g., a remote decision module) can determine where to split (e.g., determine the optimal split point) based on requirements (e.g., latency or predictive scoring) and / or current conditions (e.g., available bandwidth, energy level, available processing units, and / or the like).

[0148] Conditions can change. To maintain the initially defined requirements (e.g., if conditions change), a local decision module and / or a decision module (e.g., a remote decision module) can determine to change the processing unit (e.g., change to a processing unit on another device, such as from a processing unit on a first device to a processing unit on a second device). Instructions including information indicating the change of processing unit can be transmitted (e.g., from the first device to the second device).

[0149] The first device can determine (e.g., calculate) the processing power required to process intermediate data to be sent to the second device. The first device can transmit information (e.g., processing unit recommendations) to the second device based on the determined processing power. The transmitted information may include (e.g., contain) processing unit information, such as type (e.g., processing unit type, such as TPU, GPU, CPU, FPGA and / or the like), and / or one or more processing unit identifiers.

[0150] The first device may (e.g., from the second device) request and / or receive a list of processing units (e.g., at the second device) and their corresponding capabilities. The first device may determine and / or select one or more processing units based on (e.g., satisfying) requirements (e.g., bandwidth, latency, energy, and / or the like) (e.g., from the list of processing units). The first device may (e.g., to the second device) transmit one or more identifiers of the selected processing unit(s).

[0151] The second device can provide the first device with the real-time status of one or more processing units (e.g., processing units in a list of processing units). For example, the second device can send the load of (one or more) processing units (e.g., to the first device).

[0152] Processing unit information can be transmitted from the first device to the second device.

[0153] The first device can provide processing unit information (e.g., as metadata) to be transmitted to the second device. The second device (e.g., receiving the processing unit information) can process (e.g., compute) the processing unit information to determine and / or select processing units to process data (e.g., AI / ML models or subsets of AI / ML models).

[0154] Processing unit information may include one or more of the following: processing unit capability, processing unit type, processing unit identifier, processing capability identifier, parameter number (e.g., the number of parameters), parameter precision (e.g., the precision of (one or more) parameters), multiply-accumulate (MAC) operation (e.g., Kilo), temporary memory (e.g., MB), or processing unit real-time status.

[0155] Processing unit information may include processing unit capabilities. Processing unit capabilities describe the capacity required for the processing unit to process intermediate data transmitted from the first device to the second device. Processing unit capabilities may include processing unit speed, random access memory (RAM), cache memory, and / or power consumption.

[0156] Processing unit speed (e.g., floating-point operations per second (flops) per clock cycle) can define the number of operations per clock cycle required for the processing unit to compute intermediate data. RAM is the amount of memory available for processing intermediate data (e.g., the minimum amount of memory required to process intermediate data). Cache memory can be the amount of memory used to perform inference on intermediate data (e.g., the minimum amount of cache (e.g., L1 / L2) memory required to perform inference on intermediate data). Energy consumption can be an estimated energy consumption for processing intermediate data.

[0157] The processing unit information may include the processing unit type. The processing unit type may be one or more of the following: CPU, GPU, TPU, FPGA vision processing unit (VPU), or quantum processing unit (QPU).

[0158] Processing unit information may include a processing unit identifier. The processing unit identifier may identify a processing unit associated with a processing unit capability. The first device may (e.g., from the second device) request and / or receive a list of (one or more) processing units with corresponding capabilities. The first device may determine the processing unit capabilities (e.g., for each processing unit), select a processing unit (e.g., from the list of (one or more) processing units), and / or send a processing unit identifier (e.g., associated with the selected processing unit). The processing unit identifier may be sent to the second device along with intermediate data. In the example, “GPU ID0”, “GPU ID1”, “GPU ID3”, “GPU ID4”, “CPUID0”, or processing unit list numbers [1,2,3,4] may be sent and / or received.

[0159] Processing unit information may include the real-time status of the processing unit. The real-time status of the processing unit may describe the live state of the corresponding processing unit (e.g., the processing unit load and / or the memory occupied).

[0160] The processing unit information may include the processing unit status (e.g., received by the first device from the second device).

[0161] Processing unit information can include the processing unit type. Processing unit information can define the processing unit type (e.g., CPU, GPU, TPU, FPGA vision processing unit (VPU), or quantum processing unit (QPU)).

[0162] Processing unit information may include a processing capacity identifier. The processing capacity identifier can be used to indicate (e.g., the allocated and / or recommended processing capacity of the processing unit). Processing capacity can be negotiated during the configuration phase. Processing capacity and / or settings can be stored (e.g., during the configuration phase). Processing capacity can be identified using a processing capacity identifier.

[0163] Processing unit information may include parameter numbers (e.g., the number of parameters). The number of parameters can indicate the number of parameters in the neural network (e.g., the total number).

[0164] Processing unit information may include parameter precision. Parameter precision may be indicated as the number of bits (e.g., the number of bits) used to store (e.g., one) a parameter. In the example, a first indicator (e.g., "I") may be used to indicate an integer parameter. In the example, a second indicator (e.g., "F") may be used to indicate a floating-point number. In the example, if the proposed method uses a 16-bit integer to represent the parameter, the 16-bit integer may be reported as 16(I) or 16I or 16Int.

[0165] Processing unit information may include multiply-accumulate (MAC) operations (e.g., Kilo). In the worst case of the inference phase, a MAC operation may be multiple MAC operations (e.g., per pixel or generally). A MAC operation adds the product of two numbers to an accumulator. Processing unit information may include temporary memory (e.g., MB). Temporary memory may represent memory used to store (e.g., all) the output feature maps of intermediate layers (e.g., forward pass).

[0166] The second device may (e.g., to the first device) provide real-time status of the processing unit. The real-time status of the processing unit may include one or more of the following: processing unit load, total processing unit load, selected processing unit type, or selected processing unit identifier.

[0167] Processing unit load can identify the load of a processing unit (e.g., as a percentage from 0% to 100%). Overall processing unit load can identify the load of the overall processing units (e.g., GPUs from 1 to 4). In the example, the overall processing unit load can be expressed as a percentage (e.g., from 0% to 100%). The selected processing unit type can define the current processing unit type used to process the model portion (e.g., the AI / ML model portion). The selected processing unit identifier can identify the current processing unit used to process the model portion (e.g., the AI / ML model portion). The selected processing unit identifier can belong to a list of candidate processing units shared between devices (e.g., "GPU ID0", "GPU ID1", "GPU ID3", "GPU ID4", "CPU ID0", or processing unit list numbers [1,2,3,4]).

[0168] Figure 6 This is a block diagram depicting an example of information exchange between processing units.

[0169] Processing unit messages and / or metadata may include processing unit information. Processing unit messages may be used by a local decision module (e.g., hosted by a first device) and an AI / ML model manager (e.g., hosted by a second device) to exchange information (e.g., about the capabilities of the first and / or second devices, such as processing capacity). The exchanged information may be compiled by the local decision module of the first device. The local decision module can use the exchanged information to determine the appropriate processing unit (e.g., the processing unit best capable of processing intermediate data).

[0170] Metadata associated with intermediate data may include instructions for AI / ML processing modules (e.g., on a second device) to decode and process the intermediate data.

[0171] This document provides one or more features associated with control data used to provide information about the processing unit (e.g., before or during split inference).

[0172] The first device can send messages to the second device to receive and / or obtain (e.g., acquire) status information regarding the processing usage and / or requirements of the second device. The AI / ML model manager on the second device (e.g., such as...) Figure 6 The AI / ML model manager (as shown) can process (e.g., dispose of) messages. The AI / ML model manager can respond with information about the usage and / or availability of processing units (e.g., CPU, GPU, TPU, FPGA, VPU, and / or the like) on the second device. The first device can receive information about the processing units and / or availability on the device. The local decision module can process (e.g., dispose of) the response. The local decision module can compile the information included in the response. The local decision module can determine and / or select the optimal split point and / or (one or more) optimal processing units to use (e.g., on the first device and / or the second device). In the example, a decision module (e.g., a remote decision module) can determine and / or select the set of (one or more) optimal processing units and / or split points to use (e.g., on the first device and / or the second device). In the example, the second device can send an instruction to the first device regarding the selected set of optimal processing units and / or split points.

[0173] One or more processing units (e.g., identified processing units) on the second device may be transmitted to the first device in metadata, or may be associated with intermediate data (e.g., directly associated) (e.g., in a 2-byte form that can describe the type and ID of the processing unit to be used).

[0174] The messages exchanged between the first device and the second device may include the following example structure: ProcessingUnit message: { Host_sender: <Host_name>, MessageType: <Message_type> Model_RefID: <Model_RefID> SplitPoints:[{ <splitpointid> , <processingunittype> , <processingunitid> , <architecturetype>}] ProcessingUnitInfos: [ { ProcessingUnitType: <processingunittype>,ProcessingUnitID: <processingunitid>, ProcessingUnitLoad: <processingunitload>}, … {ProcessingUnitID: <processingunitid>, ProcessingUnitLoad: <processingunitload>} ] } For example: <Host_name> It can be either "network" or "WTRU".

[0175] <Message_Type> It can be either a "request" or a "response".

[0176] <Model_RefID> : Can be a unique model identifier.

[0177] SplitPoints: If the message type is "Request", it can be a list of split points with the requested processing unit, or if the message type is "Response", it can be a list of split points with updated configuration.

[0178] <splitpointid>It can be a unique identifier for the split point.

[0179] <processingunittype>It can be "CPU", "GPU", "TPU", or "FPGA".

[0180] <processingunitid>It can be "GPU ID0", "GPU ID1", "GPU ID3", "GPU ID4", "CPU ID0", "TPU", or "FPGA".

[0181] <architecturetype>It can be the main type of split-point architecture that can influence the choice of processing units (e.g., CNN, FC, …).

[0182] <processingunitload>This can be the load on the processing unit (e.g., in percentages from 0% to 100%).

[0183] Figure 7 An example message exchange between a first device and a second device (e.g., the second device includes an AI / ML model manager operating thereon) is depicted. In the example, messages can be exchanged by performing one or more of the following as described herein.

[0184] like Figure 7 As shown, at point 1, a first device using a processing unit request can request information from a second device. This processing unit request may include one or more of the following: a request for information associated with a split point, a request for information associated with a processing unit, a request for information associated with the processing unit's capabilities, or a request for information associated with the processing unit's real-time status.

[0185] In the example, a request for information associated with a split point may include a list of split points that the first device can expect information about. An example request for information associated with a split point may include one or more of the following: Host_name (e.g., "WTRU"), Message_type (e.g., "request"), or SplitPoints (e.g., [{SplitPointID1}, ..., {SplitPointIDn}]).

[0186] In the example, a request for information associated with a processing unit (e.g., associated with a split point) may include a list of split points transmitted by the first device and the proposed processing unit. A request for information associated with a processing unit may include one or more of the following: Host_name (e.g., "WTRU"), Message_type (e.g., "request"), or SplitPoints (e.g., [{SplitPointID1, ProcessingUnitType1}, ..., {SplitPointIDn, ProcessingUnitTypen}]).

[0187] In the example, a request that includes information associated with a processing unit capability may include one or more of the following: Host_name (e.g., "WTRU"), Message_type (e.g., "request"), or processing unit capability (e.g., as described herein).

[0188] In the example, a request that includes information associated with the real-time state of the processing unit may include one or more of the following: Host_name (e.g., "WTRU"), Message_type (e.g., "request"), or the real-time state of the processing unit (e.g., as described herein).

[0189] (For example, the AI / ML model manager of a second device) can receive (e.g., collect) requests. These requests may include indications of requested information regarding one or more split points and / or one or more processing units. Figure 7 As shown, at point 2, the AI / ML model manager can determine, obtain, or calculate a response that includes the requested information (e.g., a response based on the requested information).

[0190] At point 3, the AI / ML model manager can respond to the request (e.g., provide the requested information). The response may include one or more of the following: information about the split point, information about the processing unit (e.g., associated with the split point), information about the processing unit's capabilities, or information about the processing unit's real-time status.

[0191] In the example, a response that includes information associated with the split point may include one or more of the following: Host_name (e.g., "Network"), Message_type (e.g., "Response"), or SplitPoint (e.g., [{SplitPointID1, ProcessingUnitType1, ArchiveType1}, ..., {SplitPointIDn, ProcessingUnitTypen, ArchiveTypen}]).

[0192] In the example, a response that includes information associated with a processing unit (e.g., associated with a split point) may include one or more of the following: Host_name (e.g., "Network"), Message_type (e.g., "Response"), or SplitPoints (e.g., [{SplitPointID1, ProcessingUnitType1, ProcessingUnitID1, ArchitectureType1}, ..., {SplitPointIDn, ProcessingUnitTypen, ProcessingUnitIDn, ArchitectureTypen}]).

[0193] In the example, a response that includes information associated with the processing unit's capabilities may include one or more of the following: Host_name (e.g., "Network"), Message_type (e.g., "Response"), or ProcessingUnitInfos: [{ProcessingUnitType: ProcessingUnitType1, ProcessingUnitID: ProcessingUnitIDi>, ProcessingUnitSpeed: <processingunitspeed>,ProcessingUnitRAM: <processingunitram>,ProcessingUnitCache:<ProcessingUnitCacheL1,ProcessingUnitCacheL2>,ProcessingUnitEnergyConsumption: <processingunitenergyconsumption>}, ...}]).

[0194] In the example, a response that includes information associated with the real-time state of the processing unit may include one or more of the following: Host_name (e.g., "Network"), Message_type (e.g., "Response"), or the real-time state of the processing unit (e.g., [{ProcessingUnitType: ProcessingUnitType1, ProcessingUnitID: ProcessingUnitID1, ProcessingUnitLoad: ProcessingUnitLoad1}...]).

[0195] This document may provide one or more features associated with metadata used to provide information about processing units.

[0196] The first device can provide processing unit information as metadata (e.g., to be transmitted to the second device). The receiving device (e.g., the second device) can obtain or compute processing information to select which processing unit will process the data (e.g., an AI / ML model or a subset of AI / ML models).

[0197] Processing information may include processing unit type (e.g., ProcessingUnitType) and / or processing unit ID (e.g., ProcessingUnitID).

[0198] The processing unit type (e.g., ProcessingUnitType) is a parameter that can be transmitted (e.g., as 8 bits / 1 byte). In the examples, the processing unit type value can include one or more of the following: 0x00 (e.g., CPU), 0x01 (e.g., GPU), 0x02 (e.g., TPU), or 0x03 (e.g., FPGA). It should be understood that the example processing unit type values ​​described herein are not an exhaustive list, as one or more processing units (e.g., VPU, QPU, and / or the like) may be provided.

[0199] The processing unit ID (e.g., ProcessingUnitID) is a parameter that can be transmitted (e.g., as 8 bits / 1 byte). In the example, in addition to the processing unit type (e.g., ProcessingUnitType) parameter, the processing unit ID (e.g., ProcessingUnitID) can also be used to send information. For example, 256 entities of the selected (e.g., chosen) processing unit type (e.g., ProcessingUnitType) can be possible. For example, ProcessingUnitType=0x01 and ProcessingUnitID=0x02 (e.g., the selected processing unit of the second device can compile and / or load the split model of "GPU:2").

[0200] The processing unit type and / or processing unit ID (e.g., ProcessingUnitType and / or ProcessingUnitID) can be parameters (e.g., they are not required). The second device may use the processing unit type and / or processing unit ID if / when it exists. The second device may not set the recommended processing unit type and / or processing unit ID if the recommended processing unit is unavailable (e.g., if / when the second device receives the processing unit type and / or processing unit ID from the first device).

[0201] An intermediate data wrapper that includes processing unit information can be used. The intermediate data wrapper can include one or more of the following: model identifier, whether the split point has changed from a previous message, the next split point (e.g., used by the first device), the length of the intermediate data, the array dimension of the intermediate data, an indication of whether the data is compressed, an indication of whether a compression algorithm is used, an input data sequence identifier, a timestamp of the intermediate data, processing unit type, processing unit ID, or intermediate data. In the example, an intermediate data wrapper including processing unit information can be represented as follows: Struct IntermediateDataWrapper { int Model_RefID; # Model identifier bit ChangeOfSplitPoint; # 1 if the splitpoint has changed from a previous message. int SplitPointID; # Split point used by the first device int NextSplitpointID; # The next split point used by the first device (optional) int IntermediateDataLength; # Length of the intermediate data Dim IntermediateDataDim; # Array dimension of intermediate data byte EncodingMethod; # Indicates whether the data is compressed and which algorithm is used for compression. int SequenceNumber; # Sequence identifier for input data double TimeStamp; # Timestamp of intermediate data Byte ProcessingUnitType; # 0: CPU, 1: GPU, 2: TPU, 3: FPGA Byte ProcessingUnitID; # 0: GPU ID0, 1: GPU ID1, 2: GPU ID2, 3: GPU ID3 Byte IntermediateData[] # Intermediate data }; This article can provide one or more features associated with the processing unit type metadata (e.g., for selecting the desired processing unit).

[0202] Figure 8 Example methods for parsing and / or processing processing unit types and processing unit IDs are described.

[0203] The first splitting model can run on a processing unit (e.g., on a first device). The first splitting model can generate intermediate data from the inference. This intermediate data can be transmitted over a network to a second device. The second device can complete (e.g., terminate) the inference process. The second device can send the inference results (e.g., a score) back to the first device.

[0204] Intermediate data may be sent along with information instructing (e.g., to a second device) how to process the intermediate data. Processing unit type and / or processing unit ID may be transmitted (e.g., in addition to the intermediate data and / or in addition to (e.g., any) existing information). The second device may parse the intermediate data and / or existing information.

[0205] like Figure 8 As shown, at position 1, the processing unit type value can be read.

[0206] At point 2, the second device can determine whether a processing unit type exists. If a processing unit type exists, the example method proceeds to point 3. If no processing unit type exists, the example method proceeds to point 5. At point 5, the second device can remain on the current processing unit (e.g., the current processing unit type and / or the current processing unit ID).

[0207] At point 3, the second device can read the processing unit ID.

[0208] At point 4, the second device can determine whether a processing unit ID exists. If the processing unit ID does not exist, the example method proceeds to point 6, where the second device can use a default processing unit (e.g., ID=0). If the processing unit ID exists, the example method proceeds to point 7.

[0209] At point 7, the second device can apply (e.g., a new) processing unit type and processing unit ID to the split model. The AI / ML model can be loaded onto (e.g., a new) processing unit. The second device can send a message (e.g., a response to the first device). This message can indicate (e.g., whether a new) processing unit is in use. This message can be associated with (e.g., including) the final inference result, or it can be sent to the first device as a separate message.

[0210] Figure 9 It is a block diagram depicting examples of metadata and / or processing unit information (e.g., how intermediate data and associated information can be transferred from one device to another, such as from a first device to a second device).

[0211] In the example, the application can run on a first device. The application may rely on an AI / ML service. The AI / ML service can implement a distributed inference method. The distributed inference method may include an AI / ML model. The AI / ML model may include one or more layers (e.g., n layers (1…n)) between the second device and the first device. A first AI / ML inference engine can process a first part of the AI / ML model (e.g., partition M1 (e.g., layers 1…k)), and a second AI / ML inference engine can process a second part of the AI / ML model (e.g., partition M2 (e.g., layers k+1…n)). M1 can run on the first device, and M2 can run on the second device.

[0212] A decision module (e.g., located on a first device) may be designated as a local decision module. A decision model (e.g., a local decision model) may ingest (e.g., receive) input data, such as privacy information, split point ranges, and / or requirements (e.g., bandwidth, latency, energy, and / or the like).

[0213] The decision-making module (e.g., a local decision model) can send a message to the AI / ML model manager (e.g., on a second device). This message can request information about the hardware capabilities of the second device. Messages can be sent regularly (e.g., periodically) or individually (e.g., at any time).

[0214] The second device can send a response (e.g., an acknowledgment message) that includes hardware capabilities.

[0215] The decision module (e.g., a local decision module on the first device) can compile the received information (e.g., private and / or local information, split point range, requirements and / or hardware capabilities of the second device).

[0216] Decision modules (e.g., local decision modules) can deliver split points to split function modules.

[0217] The splitting module manages the splitting of AI / ML models. The splitting module can update models (e.g., models running in AI / ML inference engines).

[0218] In addition to the splitting point, the decision module (e.g., the local decision module) can deliver processing unit information to the AI / ML processing module (e.g., the first device). Processing unit information (e.g., dedicated to the second device) can be sent to the intermediate data delivery module.

[0219] The processing unit information may include information about the type of processing unit to be used on the first device (e.g., locally) and / or the type of processing unit to be used on the second device.

[0220] The decision module (e.g., a local decision module) can run (e.g., continuously) and evaluate the input. The decision module can deliver (e.g., new) split points and / or (e.g., new) processing unit information (e.g., if needed).

[0221] (For example, the AI / ML inference engine of the first device) can process M1 from the input data and can deliver intermediate data (for example, intended for the second device) for processing M2.

[0222] Intermediate data can be sent to an intermediate data delivery function, which can prepare it for transmission (e.g., to a second device). Preparation may include serializing, encoding, packing, and / or encapsulating the data.

[0223] The intermediate data delivery function can associate intermediate data with information for a second device to decode and process. The information for the second device to decode and process can be enriched with bytes (e.g., 2 bytes): processing unit type (e.g., ProcessingUnitType) and / or processing unit ID (e.g., ProcessingUnitID).

[0224] The processing unit type (e.g., ProcessingUnitType) can define the type of processing unit that the second device can (e.g., should) use to process intermediate data.

[0225] The processing unit ID (e.g., ProcessingUnitID) can define the identifier of the processing unit (e.g., if / when there are multiple processing units of the same type (e.g., devices), such as GPU:0, GPU:2 and / or the like).

[0226] like Figure 9 As shown, metadata may include information data used by a second device to decode and / or process intermediate data. Metadata may be transmitted in-band (e.g., along with intermediate data) or out-of-band (e.g., on a channel separate from the channel on which the intermediate data is transmitted).

[0227] Intermediate data and metadata can be read by intermediate data access functions (e.g., on a second device).

[0228] Split point information can be transmitted to the AI / ML splitting module. The AI / ML splitting module can manage the splitting of the AI / ML model (e.g., if needed).

[0229] Processing unit information (e.g., processing unit type and / or processing unit ID) can be transferred to the AI / ML inference engine. This processing unit information can be used by the AI / ML framework to compile and / or load the split model. In this example, the AI / ML framework could be Tensorflow and / or PyTorch. If the specified processing unit is unavailable, the processing unit information can be transferred back to the first device (e.g., with the inference result).

[0230] Intermediate data can be sent to the AI / ML inference engine (e.g., to complete the inference process when processing the second part M2 of the model). The inference results can be transferred to the results delivery module.

[0231] The results delivery module can prepare inference results. Preparation may include packaging or encapsulating (e.g., inference results).

[0232] The processing unit state can be transmitted (e.g., along with the inference result). If the processing unit state is transmitted (e.g., along with the inference result), the processing unit state can indicate (e.g., to the first device and its local decision module) whether the selection of the processing unit is relevant.

[0233] Although features and elements are described herein in specific combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with other features and elements. Furthermore, the methods described herein can be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable media include electronic signals (transmitted via wired or wireless connections) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROMs and / or digital multifunction discs (DVDs). The processor associated with the software can be used to implement a radio frequency transceiver for use in WTRUs, UEs, terminals, base stations, RNCs, and / or any host computer.< / processingunitenergyconsumption> < / processingunitram> < / processingunitspeed> < / processingunitload> < / architecturetype> < / processingunitid> < / processingunittype> < / splitpointid> < / processingunitload> < / processingunitid> < / processingunitload> < / processingunitid> < / processingunittype> < / architecturetype> < / processingunitid> < / processingunittype> < / splitpointid>

Claims

1. A first device, comprising: The processor is configured as follows: Determine processing unit information associated with a processing unit of the second device, wherein the processing unit information is associated with intermediate data, and wherein the processing unit information includes at least an indication of the type of processing unit to be used by the second device to process the intermediate data and a processing unit identifier associated with the processing unit of the second device. Send processing unit information and intermediate data to the second device; as well as Receive the processing unit status associated with the intermediate data from the second device.

2. The first device of claim 1, wherein the processor is further configured to receive from the second device an indication of hardware capabilities associated with the second device, wherein processing unit information is determined based on the hardware capabilities associated with the second device.

3. The first device according to any one of claims 1-2, wherein, The processor is further configured to send processing unit information to the second device as part of metadata.

4. The first device according to claim 3, wherein, The metadata will be used by a second device to decode or process intermediate data.

5. The first device according to any one of claims 3-4, wherein, The metadata will be used by the second device to select or identify the processing unit of the second device.

6. The first device according to any one of claims 3-5, wherein, The processor is further configured to send processing unit recommendations to the second device as part of metadata, wherein the processing unit recommendations are based on processing power.

7. The first device according to any one of claims 3-6, wherein, Metadata is sent via a different channel than intermediate data.

8. The first device according to any one of claims 1-7, wherein, The processor is further configured to receive inference results from a second device, wherein the inference results are associated with a score.

9. The first device according to any one of claims 1-7, wherein, The processing unit status associated with the intermediate data includes an indication that the processing unit of the second device is unavailable.

10. The first device according to claim 9, wherein, The processor is further configured to receive a response message and inference result from the second device based on an indication that the processing unit of the second device is unavailable, wherein the response message indicates that the processing unit of the second device is unavailable and processing unit information.

11. The first device according to any one of claims 1-10, wherein, The processing unit status includes an indication of at least one of the following: processing unit load, processing unit type, or processing unit identifier.

12. A method performed by a first device, comprising: Determine processing unit information associated with a processing unit of the second device, wherein the processing unit information is associated with intermediate data, and wherein the processing unit information includes at least an indication of the type of processing unit to be used by the second device to process the intermediate data and a processing unit identifier associated with the processing unit of the second device. Send processing unit information and intermediate data to the second device; as well as Receive the processing unit status associated with the intermediate data from the second device.

13. The method according to claim 12, wherein, The method further includes receiving an indication of hardware capabilities associated with the second device, wherein the processing unit information is determined based on the hardware capabilities associated with the second device.

14. The method according to any one of claims 12-13, wherein the method further comprises sending processing unit information to the second device as part of metadata.

15. The method according to claim 14, wherein, The metadata will be used by a second device to decode or process intermediate data.

16. The method according to any one of claims 14-15, wherein, The metadata will be used by the second device to select or identify the processing unit of the second device.

17. The method according to any one of claims 14-16, wherein, The method further includes sending a processing unit recommendation to a second device as part of metadata, wherein the processing unit recommendation is based on processing power.

18. The method according to any one of claims 14-17, wherein, Metadata is sent via a different channel than intermediate data.

19. The method according to any one of claims 12-18, wherein, The method further includes receiving an inference result from a second device, wherein the inference result is associated with a score.

20. The method according to any one of claims 12-18, wherein, The processing unit status associated with the intermediate data includes an indication that the processing unit of the second device is unavailable.

21. The method of claim 20, wherein the method further comprises receiving a reply message and an inference result from the second device based on an indication that the processing unit of the second device is unavailable, wherein the reply message indicates that the processing unit of the second device is unavailable and processing unit information.

22. The method according to any one of claims 12-21, wherein, The processing unit status includes an indication of at least one of the following: processing unit load, processing unit type, or processing unit identifier.

23. A second device, comprising: The processor is configured as follows: The first device receives processing unit information and intermediate data, wherein the processing unit information is associated with a processing unit of the second device, wherein the processing unit information is associated with the intermediate data, and wherein the processing unit information includes at least an indication of the type of processing unit to be used by the second device to process the intermediate data and a processing unit identifier associated with the processing unit of the second device. In response to receiving processing unit information and intermediate data, the processing unit status is determined; as well as Send the processing unit status to the first device.

24. The second device according to claim 23, wherein, The processor is further configured to send an indication of hardware capabilities associated with the second device to the first device, wherein the indication of hardware capabilities is associated with the processing unit type of the second device.

25. The second device according to any one of claims 23-24, wherein, The processor is further configured to receive processing unit information from the first device as part of metadata.

26. The second device according to claim 25, wherein, The processor is further configured to decode intermediate data based on the received metadata.

27. The second device according to any one of claims 25-26, wherein, The processor is further configured to identify the processing unit of the second device based on metadata.

28. The second device according to any one of claims 25-27, wherein, The processor is further configured to receive a processing unit recommendation from the first device as part of metadata, wherein the processing unit recommendation is associated with the amount of processing power required to process the intermediate data.

29. The second device according to any one of claims 25-28, wherein, Metadata is received via a different channel than intermediate data.

30. The second device according to any one of claims 23-29, wherein, The processor is further configured to send inference results to the first device, wherein the inference results are associated with the score.

31. The second device according to any one of claims 23-29, wherein, The processing unit status associated with the intermediate data includes an indication that the processing unit of the second device is unavailable.

32. The second device according to claim 31, wherein, The processor is further configured to send a response message and an inference result based on an indication that the processing unit of the second device is unavailable, wherein the response message indicates the unavailability of the processing unit of the second device and processing unit information.

33. The second device according to any one of claims 23-32, wherein, The processing unit status includes an indication of at least one of the following: processing unit load, processing unit type, or processing unit identifier.

34. A method performed by a second device, comprising: The first device receives processing unit information and intermediate data, wherein the processing unit information is associated with a processing unit of the second device, wherein the processing unit information is associated with the intermediate data, and wherein the processing unit information includes at least an indication of the type of processing unit to be used by the second device to process the intermediate data and a processing unit identifier associated with the processing unit of the second device. In response to receiving processing unit information and intermediate data, the processing unit status is determined; as well as Send the processing unit status to the first device.

35. The method according to claim 34, wherein, The method further includes sending an indication of hardware capabilities associated with the second device to the first device, wherein the indication of hardware capabilities is associated with the processing unit type of the second device.

36. The method according to any one of claims 34-35, wherein, The method further includes receiving processing unit information from the first device as part of metadata.

37. The method according to claim 36, wherein, The method further includes decoding intermediate data based on the received metadata.

38. The method according to any one of claims 36-37, wherein, The method further includes a processing unit that identifies a second device based on metadata.

39. The method according to any one of claims 36-38, wherein, The method further includes receiving a processing unit recommendation from a first device as part of metadata, wherein the processing unit recommendation is associated with the amount of processing power required to process intermediate data.

40. The method according to any one of claims 36-39, wherein, Metadata is received via a different channel than intermediate data.

41. The method according to any one of claims 34-40, wherein, The method further includes sending an inference result to a first device, wherein the inference result is associated with a score.

42. The method according to any one of claims 34-40, wherein, The processing unit status associated with the intermediate data includes an indication that the processing unit of the second device is unavailable.

43. The method according to claim 42, wherein, The method further includes sending a response message and an inference result based on an indication that the processing unit of the second device is unavailable, wherein the response message indicates the unavailability of the processing unit of the second device and processing unit information.

44. The method according to any one of claims 34-43, wherein, The processing unit status includes an indication of at least one of the following: processing unit load, processing unit type, or processing unit identifier.

45. A computer program product stored on a non-transitory computer-readable medium and comprising program code instructions for implementing, when executed by at least one processor, the steps of the method according to claims 12 to 22 or at least one of claims 34 to 44.

46. ​​A computer program comprising program code instructions configured to, when executed by a processor, implement the steps of the method according to at least one of claims 12 to 22 or 34 to 44.

47. Video data comprising information representing an encoded output generated by any one of the methods according to claims 12 to 22 or 34 to 44.

Images