Method for transmitting processing unit information associated with a division point

Abstract

Systems, methods, and means for processing processing unit information associated with a split point may be disclosed. The first device may include a processor. The first device may be configured to determine processing unit information associated with a processing unit of a second device. The processing unit information may be associated with intermediate data. The processing unit information may include at least an indication of the type of processing unit to be used by the second device to process the intermediate data, and / or a processing unit identifier associated with a processing unit of the second device. The device may be configured to transmit processing unit information and / or intermediate data to the second device. The device may be configured to receive processing unit status associated with the intermediate data from the second device.

Description

[[Technical Field]]

[0001] (Cross-Reference to Related Applications) [[Background Art]]

[0002] Video coding systems may be used to compress digital video signals, for example, to reduce the storage and / or transmission bandwidth of such signals. Video coding systems may include, for example, block-based, wavelet-based, and / or object-based systems. [[Summary of the Invention]]

[0003] Systems, methods, and means for processing processing unit information associated with a split point may be disclosed. A wireless transmit / receive unit (WTRU) (for example, a first device) may include a processor. The first device may be configured to determine processing unit information associated with a processing unit of a second device. The processing unit information may be associated with intermediate data. The processing unit information may include at least an indication of a processing unit type to be used by the second device to process the intermediate data, and / or a processing unit identifier associated with the processing unit of the second device. The device may be configured to transmit the processing unit information and / or the intermediate data to the second device. The device may be configured to receive, from the second device, a processing unit status associated with the intermediate data.

[0004] The first device may be configured to receive instructions from the second device regarding the hardware capabilities associated with the second device. Processing unit information may be determined based on the hardware capabilities associated with the second device. The first device may be configured to transmit processing unit information to the second device as part of metadata. The metadata may be used by the second device to decode or process intermediate data and / or to select or identify processing units for the second device. The first device may be configured to transmit processing unit recommendations to the second device as part of the metadata. Processing unit recommendations may be based on processing capabilities.

[0005] Metadata may be transmitted via a different channel than the intermediate data. The first device may be configured to receive inference results from the second device. The inference results may be associated with a score. The processing unit status associated with the intermediate data may include an indication that the processing unit of the second device is unavailable. Based on the indication that the processing unit of the second device is unavailable, the first device may be configured to receive a response message and / or inference results from the second device. The response message may indicate that the processing unit of the second device is unavailable and / or may indicate processing unit information. The processing unit status may include an indication of at least one of the following: processing unit load, processing unit type, and processing unit identifier.

[0006] The method may be performed by a first device. The method may include determining processing unit information associated with a processing unit of a second device. The processing unit information may be associated with intermediate data. The processing unit information may include at least an indication of the type of processing unit to be used by the second device to process the intermediate data, and / or a processing unit identifier associated with the processing unit of the second device. The method may include transmitting the processing unit information and / or the intermediate data to the second device. The method may include receiving the processing unit status associated with the intermediate data from the second device.

[0007] The method may include receiving instructions from a second device regarding the hardware capabilities associated with the second device. Processing unit information may be determined based on the hardware capabilities associated with the second device. The method may include transmitting processing unit information to the second device as part of metadata. The metadata may be used by the second device to decode or process intermediate data and / or to select or identify processing units of the second device. The method may include transmitting processing unit recommendations to the second device as part of metadata. Processing unit recommendations may be based on processing capabilities.

[0008] Metadata may be transmitted via a different channel than the intermediate data. The method may include receiving inference results from a second device. The inference results may be associated with a score. The processing unit status associated with the intermediate data may include an indication that the processing unit of the second device is unavailable. Based on the indication that the processing unit of the second device is unavailable, the method may include receiving a response message and / or inference results from the second device. The response message may indicate that the processing unit of the second device is unavailable and / or include processing unit information. The processing unit status may include an indication of at least one of the following: processing unit load, processing unit type, and processing unit identifier.

[0009] A wireless transceiver unit (WTRU) (e.g., a second device) may include a processor. The second device may be configured to receive processing unit information and / or intermediate data from the first device. The processing unit information may be associated with a processing unit of the second device. The processing unit information may also be associated with intermediate data. The processing unit information may include at least an indication of the type of processing unit to be used by the second device to process the intermediate data, and / or a processing unit identifier associated with a processing unit of the second device. The second device may be configured to determine the processing unit status in response to receiving the processing unit information and / or intermediate data. The second device may be configured to transmit the processing unit status to the first device.

[0010] The second device may be configured to send instructions for hardware capabilities associated with the second device to the first device. These instructions may be associated with the processing unit type of the second device. The second device may be configured to receive processing unit information from the first device as part of metadata. The second device may be configured to decode the intermediate data based on the received metadata. The second device may be configured to identify its own processing unit based on the metadata. The second device may be configured to receive processing unit recommendations from the first device as part of the metadata. These processing unit recommendations may be associated with the amount of processing power required to process the intermediate data. The metadata may be received via a different channel than the intermediate data.

[0011] The second device may be configured to send inference results to the first device. The inference results may be associated with a score. The processing unit status associated with the intermediate data may include an indication that the processing unit of the second device is unavailable. Based on the indication that the processing unit of the second device is unavailable, the second device may be configured to send a response message and / or inference results. The response message may indicate the unavailability of the processing unit of the second device and / or processing unit information. The processing unit status may include an indication of at least one of the following: processing unit load, processing unit type, and processing unit identifier.

[0012] A method performed by the second device may include receiving processing unit information and / or intermediate data from the first device. The processing unit information may be associated with a processing unit of the second device. The processing unit information may also be associated with intermediate data. The processing unit information may include at least an indication of the type of processing unit to be used by the second device to process the intermediate data, and / or a processing unit identifier associated with a processing unit of the second device. In response to receiving the processing unit information and / or intermediate data, the method may include determining the processing unit status. The method may include transmitting the processing unit status to the first device.

[0013] The method may include sending a hardware capability instruction to a first device, associated with a second device. The hardware capability instruction may be associated with a processing unit type of the second device. The method may include receiving processing unit information from the first device as part of metadata. The method may include decoding the intermediate data based on the received metadata. The method may include identifying the processing unit of the second device based on the metadata. The method may include receiving a processing unit recommendation from the first device as part of the metadata. The processing unit recommendation may be associated with the amount of processing power required to process the intermediate data. The metadata may be received via a different channel than the intermediate data.

[0014] The method may include sending inference results to a first device. The inference results may be associated with a score. The processing unit status associated with the intermediate data may include an indication that the processing unit of the second device is unavailable. Based on the indication that the processing unit of the second device is unavailable, the method may include sending a response message and / or inference results. The response message may indicate the unavailability of the processing unit of the second device and / or processing unit information. The processing unit status may include an indication of at least one of the following: processing unit load, processing unit type, and processing unit identifier.

[0015] A computer program product stored on a non-temporary computer-readable medium may, when executed by at least one processor, include program code instructions for implementing steps of the methods described herein. A computer program may, when executed by a processor, include program code instructions for implementing steps of the methods described herein. Video data may include information representing encoded output generated according to one of the methods described herein.

[0016] A device (e.g., a WTRU) may be configured with a processor to perform one or more actions. The device may receive a set of processing units and processing unit information associated with each processing unit in the set. The device may determine the amount of processing power required to process the intermediate data. Based on the processing unit information and the amount of processing power, the device may select processing units from the set. The device may transmit instructions indicating the selected processing units.

[0017] In the example, the processing unit information may include the processing unit type associated with each processing unit in the set of processing units.

[0018] In the example, the processing unit information may include a processing unit identifier associated with each processing unit in the set of processing units.

[0019] In the example, the instruction may include a processing unit identifier associated with the selected processing unit.

[0020] In the example, the device may receive status updates about the load associated with at least one processing unit in a set of processing units.

[0021] In this example, the device may send processing unit recommendations as metadata.

[0022] The systems, methods, and means described in the present specification may involve a decoder. In some examples, the systems, methods, and means described in the present specification may involve an encoder. In some examples, the systems, methods, and means described in the present specification may involve a signal (e.g., received from an encoder and / or by a decoder). A computer-readable medium may include instructions for causing one or more processors to perform the methods described herein. A computer program product may include instructions that, when the program is executed by one or more processors, can cause one or more processors to perform the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] [Figure 1A] It is a system diagram showing an exemplary communication system in which one or more disclosed embodiments may be implemented. [Figure 1B] It is a system diagram showing an exemplary wireless transmit / receive unit (WTRU) that may be used in the communication system shown in FIG. 1A according to an embodiment. [Figure 1C] It is a system diagram showing an exemplary radio access network (RAN) and an exemplary core network (CN) that may be used in the communication system shown in FIG. 1A according to an embodiment. [Figure 1D] It is a system diagram showing a further exemplary RAN and a further exemplary CN that may be used in the communication system shown in FIG. 1A according to an embodiment. [Figure 2] Shows an exemplary video encoder. [Figure 3] Shows an exemplary video decoder. [Figure 4] Shows an example of a system in which various aspects and examples may be implemented. [Figure 5A] Shows an exemplary architecture for split inference between a WTRU and a network, where the WTRU has a media data source. [Figure 5B] Shows an exemplary architecture for split inference between a WTRU and a network, where a media data source is located within the network. [Figure 5C] Shows an example of messages exchanged between a WTRU and a network device to manage dynamic splitting points. [Figure 6] It is a block diagram showing an example of processing unit information exchange. [Figure 7] Shows an exemplary message exchange between a first device and a second device, wherein each of the first device or the second device is a WTRU or a network device. [Figure 8] Shows an exemplary method for parsing and / or processing processing unit types, and / or processing unit IDs. [Figure 9] It is a block diagram showing an example of metadata and / or processing unit information. DETAILED DESCRIPTION OF THE INVENTION

[0024] A more detailed understanding may be obtained from the following description, which is given by way of example in conjunction with the accompanying drawings.

[0025] Figure 1A shows an exemplary communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content such as voice, data, video, messaging, and broadcast to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communication system 100 may employ one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique-word OFDM (UW-OFDM), resource block filtering OFDM, and filter bank multicarrier (FBMC).

[0026] As shown in Figure 1A, the communication system 100 may include radio transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106 / 115, public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, but it will be understood that the disclosed embodiments are intended to be any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, and 102d may be any type of device configured to operate and / or communicate in a radio environment. For example, WTRU102a, 102b, 102c, 102d (any of which may be referred to as “Station” and / or “STA”) may be configured to transmit and / or receive radio signals and may include user equipment (WTRU), mobile stations, fixed or mobile subscriber units, subscriber-based units, pagers, mobile phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearables, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of industrial and / or automated processing chains), consumer electronics devices, and devices operating on commercial and / or industrial wireless networks. WTRU102a, 102b, 102c, and 102d can all be referred to synonymously as WTRU.

[0027] The communication system 100 may also include base stations 114a and / or base stations 114b. Each of the base stations 114a and 114b may be any type of device configured to radio interface with at least one of the WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks, such as CN 106 / 115, the Internet 110, and / or other networks 112. For example, base stations 114a and 114b may be a base transceiver station (BTS), node B, enode B, home node B, home enode B, gNB, NR node B, site controller, access point (AP), wireless router, etc. Although base stations 114a and 114b are shown as single elements, it will be understood that base stations 114a and 114b may include any number of interconnected base stations and / or network elements.

[0028] Base station 114a may be part of RAN 104 / 113, which may also include other base station and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), and relay nodes. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. Cells may provide coverage of radio services to a particular geographic area, which may be relatively fixed or change over time. Cells may be further divided into cell sectors. For example, a cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver per sector of the cell. In one embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and utilize multiple transceivers per sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0029] Base stations 114a and 114b may communicate with one or more WTRUs 102a, 102b, 102c, and 102d via an air interface 116, which may be any suitable radio communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0030] More specifically, as described above, the communication system 100 may be a multiple access system and may employ one or more channel access schemes such as CDMA, TDMA, FDMA, OFDMA, and SC-FDMA. For example, base stations 114a and WTRUs 102a, 102b, and 102c in RAN 104 / 113 may implement radio technologies such as Universal Mobile Telecommunications System (UMTS) and Universal Terrestrial Radio Access (UTRA), which may establish air interfaces 115 / 116 / 117 using wideband CDMA (WCDMA®). WCDMA may include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or Advanced HSPA (HSPA+). HSPA may include High-Speed ​​Downlink Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0031] In one embodiment, base stations 114a and WTRUs 102a, 102b, 102c may implement radio technologies such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish an air interface 116 using Long Term Evolution (LTE), LTE-Advanced (LTE-A), and / or LTE-Advanced Pro (LTE-A Pro).

[0032] In one embodiment, base stations 114a and WTRUs 102a, 102b, and 102c may implement radio technologies such as NR radio access, which may establish an air interface 116 using New Radio (NR).

[0033] In one embodiment, base station 114a and WTRU 102a, 102b, 102c may implement multiple radio access technologies. For example, base station 114a and WTRU 102a, 102b, 102c may implement both LTE radio access and NR radio access using, for example, the dual connectivity (DC) principle. Thus, the air interface utilized by WTRU 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions to / from multiple types of base stations (e.g., eNB and gNB).

[0034] In other embodiments, base stations 114a and WTRUs 102a, 102b, and 102c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity, WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access, WiMAX), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), and GSM EDGE (GERAN).

[0035] The base station 114b in Figure 1A may be, for example, a wireless router, home node B, home e-node B, or access point, and may utilize any suitable RAT to facilitate wireless connectivity in local areas such as offices, homes, vehicles, campuses, industrial facilities, aerial corridors (for use by drones), roads, etc. In one embodiment, the base station 114b and WTRU 102c, 102d may implement wireless technologies such as IEEE 802.11 to establish a wireless local area network (WLAN). In another embodiment, the base station 114b and WTRU 102c, 102d may implement wireless technologies such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, base stations 114b and WTRUs 102c, 102d may establish picocells or femtocells using cellular-based RATs (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.). As shown in Figure 1A, base station 114b may be directly connected to the internet 110. Therefore, base station 114b may not need to access the internet 110 via CN 106 / 115.

[0036] RAN104 / 113 may communicate with CN106 / 115, which may be any type of network configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRU102a, 102b, 102c, and 102d. The data may have various quality of service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, and mobility requirements. CN106 / 115 may provide call control, billing services, mobile location-based services, prepaid calls, internet connectivity, video distribution, and / or perform high-level security functions such as user authentication. Although not shown in Figure 1A, it will be understood that RAN104 / 113 and / or CN106 / 115 can communicate directly or indirectly with other RANs employing the same or different RATs as RAN104 / 113. For example, in addition to connecting to RAN104 / 113 which may utilize NR radio technology, CN106 / 115 may also communicate with another RAN (not shown) employing GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0037] CN106 / 115 may also function as a gateway for WTRU102a, 102b, 102c, 102d to access PSTN108, the Internet 110, and / or other networks 112. PSTN108 may include a circuit-switched telephone network providing plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices using common communication protocols such as the transmission control protocol (TCP), the user datagram protocol (UDP), and / or the Internet protocol (IP) of the TCP / IP Internet Protocol suite. Network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs, one or more RANs may employ the same RAT as RAN104 / 113 or a different RAT.

[0038] Some or all of the WTRUs 102a, 102b, 102c, and 102d in the communication system 100 may include multimode functionality (for example, WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers for communicating with different radio networks via different radio links). For example, WTRU 102c shown in Figure 1A may be configured to communicate with base station 114a, which may employ cellular-based radio technology, and base station 114b, which may employ IEEE 802 radio technology.

[0039] Figure 1B is a system diagram showing an exemplary WTRU 102. As shown in Figure 1B, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It will be understood that the WTRU 102 may include any partial combination of the aforementioned elements while maintaining consistency with a particular embodiment.

[0040] The processor 118 may be a general-purpose processor, a dedicated processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functions that enable the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to a transceiver 120 which can be coupled to a transmit / receive element 122. Although Figure 1B shows the processor 118 and transceiver 120 as separate components, it will be understood that the processor 118 and transceiver 120 may be integrated together in an electronic package or chip.

[0041] The transmit / receive element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In one embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive, for example, IR signals, UV signals, or visible light signals. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF signals and optical signals. It will be understood that the transmit / receive element 122 may be configured to transmit and / or receive any combination of radio signals.

[0042] Although the transmit / receive element 122 is shown as a single element in Figure 1B, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for sending and receiving radio signals via the air interface 116.

[0043] The transceiver 120 may be configured to modulate the signal to be transmitted by the transmit / receive element 122 and to demodulate the signal to be received by the transmit / receive element 122. As described above, the WTRU 102 may have multimode capabilities. Therefore, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0044] The processor 118 of the WTRU102 may be coupled to a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light-emitting diode (OLED) display unit) and may receive user input from these. The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from any type of suitable memory, such as non-removable memory 130 and / or removable memory 132, and may store data in memory. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from memory not physically located on the WTRU 102, such as on a server or home computer (not shown), and store data in memory.

[0045] The processor 118 may be configured to receive power from the power supply 134 and distribute and / or control power to other components within the WTRU 102. The power supply 134 may be any suitable device for supplying power to the WTRU 102. For example, the power supply 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0046] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) about the current location of the WTRU 102. In addition to, or instead of, the information from the GPS chipset 136, the WTRU 102 may determine its location based on receiving location information from base stations (e.g., base stations 114a, 114b) via the air interface 116 and / or based on the timing of signals received from two or more nearby base stations. It will be understood that the WTRU 102 may acquire location information by any preferred location determination method while maintaining consistency with a particular embodiment.

[0047] The processor 118 may be further coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functions, and / or wired or wireless connectivity. For example, peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, and the like. The peripheral device 138 may include one or more sensors, one or more of which may be a gyroscope, accelerometer, Hall effect sensor, magnetometer, compass sensor, proximity sensor, temperature sensor, time sensor, geolocation sensor, altimeter, light sensor, touch sensor, magnetometer, barometer, gesture sensor, biometric sensor, and / or humidity sensor.

[0048] WTRU102 may include a full-duplex radio in which the transmission and reception of some or all of a signal (e.g., a signal associated with specific subframes of both UL (e.g., for transmission) and downlink (e.g., for reception)) may occur in parallel and / or simultaneously. The full-duplex radio may include an interference management unit for reducing and / or substantially eliminating self-interference via either hardware (e.g., chokes) or signal processing via a processor (e.g., via a separate processor (not shown) or processor 118). In one embodiment, WTRU102 may include a half-duplex radio for the transmission and reception of some or all of a signal (e.g., associated with specific subframes of either UL (e.g., for transmission) or downlink (e.g., for reception)).

[0049] Figure 1C is a system diagram showing RAN104 and CN106 according to one embodiment. As described above, RAN104 can communicate with WTRU102a, 102b, and 102c via the air interface 116 using E-UTRA wireless technology. RAN104 can also communicate with CN106.

[0050] RAN104 may include e-nodes B160a, 160b, and 160c, but it will be understood that RAN104 may include any number of e-nodes B while maintaining consistency with a particular embodiment. Each of e-nodes B160a, 160b, and 160c may include one or more transceivers for communicating with WTRU102a, 102b, and 102c via the air interface 116. In one embodiment, e-nodes B160a, 160b, and 160c may implement MIMO technology. Thus, e-node B160a may, for example, use multiple antennas to transmit radio signals to and / or receive radio signals from WTRU102a.

[0051] Each of the e-nodes B160a, 160b, and 160c may be associated with a specific cell (not shown) and may be configured to handle wireless resource management decisions, handover decisions, user scheduling in UL and / or DL, etc. As shown in Figure 1C, the e-nodes B160a, 160b, and 160c may communicate with each other via the X2 interface.

[0052] The CN106 shown in Figure 1C may include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (or packet gateway, PGW) 166. Although each of the aforementioned elements is shown as part of CN106, it should be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0053] The MME162 can be connected to each of the e-nodes B162a, 162b, and 162c within RAN104 via the S1 interface and can function as a control node. For example, the MME162 may be responsible for user authentication of WTRU102a, 102b, and 102c, activation / deactivation of bearers, and selection of a specific serving gateway during the initial connection of WTRU102a, 102b, and 102c. The MME162 may provide control plane functionality for switching between RAN104 and other RANs (not shown) employing other radio technologies such as GSM and / or WCDMA.

[0054] The SGW164 can connect to each of the e-nodes B160a, 160b, and 160c within RAN104 via the S1 interface. The SGW164 can generally route and forward user data packets to and from WTRU102a, 102b, and 102c. The SGW164 can perform other functions, such as anchoring the user plane during e-node B handovers, triggering paging when DL data is available to WTRU102a, 102b, and 102c, and managing and remembering the context of WTRU102a, 102b, and 102c.

[0055] SGW164 can connect to PGW166, which allows WTRU102a, 102b, and 102c to access packet-switched networks such as the Internet 110, thereby facilitating communication between WTRU102a, 102b, and 102c and IP-enabled devices.

[0056] CN106 can facilitate communication with other networks. For example, CN106 can provide WTRU102a, 102b, and 102c with access to a circuit-switched network such as PSTN108 to facilitate communication between WTRU102a, 102b, and 102c and conventional terrestrial line communication devices. For example, CN106 may include, or communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that functions as an interface between CN106 and PSTN108. In addition, CN106 may provide WTRU102a, 102b, and 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers.

[0057] While the WTRU is shown as a wireless terminal in Figures 1A to 1D, in certain representative embodiments, such a terminal is intended to be able to use a wired communication interface with a communication network (e.g., temporarily or permanently).

[0058] In a typical embodiment, the other network 112 may be a WLAN.

[0059] In Infrastructure Basic Service Set (BSS) mode, a WLAN may have access points (APs) for the BSS and one or more stations (STAs) associated with the APs. APs may have access to or interfaces with a Distribution System (DS) or another type of wired / wireless network that carries traffic into and / or out of the BSS. Traffic originating from outside the BSS to an STA may arrive via an AP and be delivered to the STA. Traffic originating from an STA to a destination outside the BSS may be sent to an AP and delivered to its respective destination. Traffic between STAs within the BSS may be transmitted, for example, via an AP, where a source STA sends traffic to an AP, and the AP delivers the traffic to the destination STA. Traffic between STAs within the BSS may be considered and / or referred to as peer-to-peer traffic. Peer-to-peer traffic may be transmitted between a source STA and a destination STA (for example, directly between them) using a direct link setup (DLS). In certain representative embodiments, the DLS may use 802.11e DLS or 802.11z tunneled DLS (TDLS). A WLAN using Independent BSS (IBSS) mode may not have APs, and STAs within or using IBSS (e.g., all STAs) may communicate directly with one another. The IBSS communication mode may be referred to herein as the “ad hoc” communication mode.

[0060] When using the 802.11ac infrastructure operating mode or a similar operating mode, an AP may transmit beacons on a fixed channel, such as the primary channel. The primary channel may be of a fixed width (e.g., a 20 MHz bandwidth) or a dynamically set width via signaling. The primary channel may be the operating channel of the BSS and may be used by the STA to establish a connection with the AP. In a particular representative embodiment, a Carrier Sense Multiple Access with Collision Avoidance (CSMA / CA) scheme may be implemented, for example, in an 802.11 system. In the case of CSMA / CA, the STA, including the AP (e.g., all STAs), may sense the primary channel. If the primary channel is sensed / detected and / or determined to be busy by a particular STA, that STA may backoff. One STA (e.g., only one station) may transmit on a given BSS at any given time.

[0061] A high-throughput (HT) STA can use a 40MHz wide channel for communication, for example, by forming a 40MHz wide channel through a combination of a primary 20MHz channel and adjacent or non-adjacent 20MHz channels.

[0062] Very High Throughput (VHT) STAs can support channels with widths of 20 MHz, 40 MHz, 80 MHz, and / or 160 MHz. 40 MHz and / or 80 MHz channels can be formed by combining consecutive 20 MHz channels. 160 MHz channels can be formed by combining eight consecutive 20 MHz channels or two discontinuous 80 MHz channels, which may be referred to as an 80+80 configuration. In the 80+80 configuration, data can be passed through a segment parser that, after channel coding, can split the data into two streams. Inverse Fast Fourier Transform (IFFT) processing and time-domain processing can be performed independently on each stream. The streams may be mapped to two 80 MHz channels, and the data can be transmitted by the transmitting STA. At the receiver of the receiving STA, the above operation for the 80+80 configuration may be reversed, and the combined data can be transmitted to Medium Access Control (MAC).

[0063] Sub-1 GHz operating modes are supported in 802.11af and 802.11ah. Channel operating bandwidth and carrier are reduced in 802.11af and 802.11ah compared to those used in 802.11n and 802.11ac. 802.11af supports bandwidths of 5 MHz, 10 MHz, and 20 MHz in the TV White Space (TVWS) spectrum, while 802.11ah supports bandwidths of 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz using the non-TVWS spectrum. According to a typical embodiment, 802.11ah may support meter-type control / machine-type communications, such as MTC devices in a macro coverage area. MTC devices may have limited functionality, including support for specific bandwidths and / or limited bandwidths (e.g., support only). MTC devices may include batteries with battery life exceeding a threshold (e.g., to maintain very long battery life).

[0064] A WLAN system that can support multiple channels and channel bandwidths such as 802.11n, 802.11ac, 802.11af, and 802.11ah includes a channel that can be designated as the primary channel. The primary channel may have a bandwidth equal to the maximum common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel may be set and / or limited by the STA that supports the minimum bandwidth operating mode among all STAs operating in the BSS. In the 802.11ah example, even if the AP and other STAs in the BSS support operating modes of 2MHz, 4MHz, 8MHz, 16MHz, and / or other channel bandwidths, the primary channel of an STA that supports 1MHz mode (e.g., only 1MHz mode) (e.g., an MTC type device) may be 1MHz wide. Carrier detection and / or Network Allocation Vector (NAV) settings may depend on the status of the primary channel. For example, if the primary channel is busy because an STA (which only supports 1MHz operating mode) is transmitting to the AP, the entire available frequency band may be considered busy, even though a large portion of the frequency band remains idle and could potentially be available.

[0065] In the United States, the available frequency band for 802.11ah is 902MHz to 928MHz. In South Korea, the available frequency band is 917.5MHz to 923.5MHz. In Japan, the available frequency band is 916.5MHz to 927.5MHz. The total available bandwidth for 802.11ah is 6MHz to 26MHz, depending on the country code.

[0066] Figure 1D is a system diagram showing RAN113 and CN115 according to one embodiment. As described above, RAN113 can communicate with WTRU102a, 102b, and 102c via air interface 116 using NR radio technology. RAN113 can also communicate with CN115.

[0067] RAN113 may include gNB180a, 180b, and 180c, but it will be understood that RAN113 may include any number of gNBs while maintaining consistency with a particular embodiment. Each of gNB180a, 180b, and 180c may include one or more transceivers for communicating with WTRU102a, 102b, and 102c via the air interface 116. In one embodiment, gNB180a, 180b, and 180c may implement MIMO technology. For example, gNB180a and 108b may use beamforming to transmit signals to and / or receive signals from gNB180a, 180b, and 180c. Thus, gNB180a may, for example, use multiple antennas to transmit radio signals to and / or receive radio signals from WTRU102a. In some embodiments, gNB180a, 180b, and 180c may implement carrier aggregation technology. For example, gNB180a may transmit multiple component carriers to WTRU102a (not shown). A subset of these component carriers may be on the unlicensed spectrum, while the remaining component carriers may be on the licensed spectrum. In some embodiments, gNB180a, 180b, and 180c may implement coordinated multi-point (CoMP) technology. For example, WTRU102a may receive coordinated transmissions from gNB180a and gNB180b (and / or gNB180c).

[0068] WTRU102a, 102b, and 102c may communicate with gNB180a, 180b, and 180c using transmissions associated with scalable numerology. For example, OFDM symbol intervals and / or OFDM subcarrier intervals may vary for different transmissions, different cells, and / or different parts of the radio transmission spectrum. WTRU102a, 102b, and 102c may communicate with gNB180a, 180b, and 180c using subframes or transmit time intervals (TTIs) of varying or scalable lengths (e.g., containing varying numbers of OFDM symbols and / or spanning varying lengths of absolute time).

[0069] gNB180a, 180b, and 180c can be configured to communicate with WTRU102a, 102b, and 102c in standalone and / or non-standalone configurations. In a standalone configuration, WTRU102a, 102b, and 102c can communicate with gNB180a, 180b, and 180c without accessing other RANs (e.g., e-nodes B160a, 160b, and 160c). In a standalone configuration, WTRU102a, 102b, and 102c can utilize one or more of gNB180a, 180b, and 180c as mobility anchor points. In a standalone configuration, WTRU102a, 102b, and 102c can communicate with gNB180a, 180b, and 180c using signals in the unlicensed band. In a non-standalone configuration, WTRU102a, 102b, and 102c may communicate with and connect to gNB180a, 180b, and 180c, while also communicating with and connecting to other RANs such as enodes B160a, 160b, and 160c. For example, WTRU102a, 102b, and 102c may implement a DC principle for substantially simultaneous communication with one or more gNB180a, 180b, and 180c and one or more enodes B160a, 160b, and 160c. In a non-standalone configuration, enodes B160a, 160b, and 160c may function as mobility anchors for WTRU102a, 102b, and 102c, and gNB180a, 180b, and 180c may provide additional coverage and / or throughput to service WTRU102a, 102b, and 102c.

[0070] Each of the gNB180a, 180b, and 180c may be associated with a specific cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, user scheduling in UL and / or DL, support for network slicing, dual connectivity, interworking between NR and E-UTRA, routing of user plane data to User Plane Functions (UPFs) 184a and 184b, access to control plane information, and routing to Mobility Management Functions (AMFs) 182a and 182b. As shown in Figure 1D, the gNB180a, 180b, and 180c may communicate with each other via the Xn interface.

[0071] The CN115 shown in Figure 1D may include at least one AMF182a, 182b, at least one UPF184a, 184b, at least one Session Management Function (SMF)183a, 183b, and optionally a Data Network (DN)185a, 185b. Although each of the aforementioned elements is shown as part of the CN115, it will be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0072] AMF182a and 182b may be connected to one or more of gNB180a, 180b, and 180c in RAN113 via the N2 interface and may function as control nodes. For example, AMF182a and 182b may be responsible for user authentication of WTRU102a, 102b, and 102c, support for network slicing (e.g., handling different PDU sessions with different requirements), selection of specific SMF183a and 183b, management of registration areas, termination of NAS signaling, mobility management, etc. Network slicing may be used by AMF182a and 182b to customize CN support for WTRU102a, 102b, and 102c based on the type of services utilized by WTRU102a, 102b, and 102c. For example, different network slices can be established for various use cases, such as services relying on ultra-reliable low latency (URLLC) access, services relying on enhanced massive mobile broadband (eMBB) access, and services using machine-type communication (MTC) access. The AMF182a and 182b can provide control plane functionality for switching between RAN113 and other RANs (not shown) using other radio technologies such as LTE, LTE-A, LTE-A Pro, and / or non-3GPP® access technologies like WiFi.

[0073] SMF183a and 183b may be connected to AMF182a and 182b in CN115 via the N11 interface. SMF183a and 183b may also be connected to UPF184a and 184b in CN115 via the N4 interface. SMF183a and 183b may select and control UPF184a and 184b and configure traffic routing through UPF184a and 184b. SMF183a and 183b may perform other functions such as managing and assigning WTRU IP addresses, managing PDU sessions, controlling policy enforcement and QoS, and providing downlink data notifications. The PDU session type may be IP-based, non-IP-based, Ethernet-based, etc.

[0074] UPF184a, 184b may be connected via the N3 interface to one or more of gNB180a, 180b, 180c in RAN113, thereby providing WTRU102a, 102b, 102c with access to a packet-switched network such as the Internet 110, facilitating communication between WTRU102a, 102b, 102c and IP-enabled devices. UPF184, 184b may perform other functions such as packet routing and forwarding, enforcement of user plane policies, support for multi-homed PDU sessions, processing of user plane QoS, buffering of downlink packets, and providing mobility anchors.

[0075] CN115 can facilitate communication with other networks. For example, CN115 may include, or communicate with, an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that functions as an interface between CN115 and PSTN108. In addition, CN115 may provide WTRU102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers. In one embodiment, WTRU102a, 102b, 102c may be connected to local data networks (DN) 185a, 185b via UPF184a, 184b through an N3 interface to UPF184a, 184b and an N6 interface between UPF184a, 184b and DN185a, 185b.

[0076] With regard to Figures 1A-1D and the corresponding descriptions in Figures 1A-1D, with respect to one or more of the WTRU102a-d, base stations 114a-b, e-nodes B160a-c, MME162, SGW164, PGW166, gNB180a-c, AMF182a-b, UPF184a-b, SMF183a-b, DN185a-b, and / or any other devices described herein, one or more of the functions described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more of the functions described herein. For example, an emulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0077] Emulation devices may be designed to implement one or more tests of other devices in a laboratory and / or operator network environment. For example, one or more emulation devices may perform one or more or all functions while fully or partially implemented and / or deployed as part of a wired and / or wireless network to test other devices in a communications network. One or more emulation devices may perform one or more or all functions while temporarily implemented / deployed as part of a wired and / or wireless network. Emulation devices may be directly coupled to another device for testing purposes and / or perform tests using over-the-air wireless communication.

[0078] One or more emulation devices may perform one or more functions, including all of the above, while not implemented / deployed as part of a wired and / or wireless communication network. For example, an emulation device may be used in a test laboratory test scenario, and / or in an undeployed (e.g., test) wired and / or wireless communication network, to implement testing of one or more components. One or more emulation devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (e.g., which may include one or more antennas) may be used by the emulation device to transmit and / or receive data.

[0079] This application describes various embodiments, including tools, features, examples, models, and approaches. Many of these embodiments are described in detail, often in a manner that sounds restrictive, at least to illustrate individual features. However, this is for clarity and not to limit the use or scope of these embodiments. In fact, all of the different embodiments may be combined and interchangeable to provide further embodiments. Furthermore, these embodiments may be combined and interchangeable with embodiments described in previous applications.

[0080] The embodiments described and contemplated herein may be implemented in many different forms. Figures 5 to 9 described herein may provide some examples, but other examples are also contemplated. The considerations in Figures 5 to 9 are not intended to limit the scope of implementation forms. At least one of these embodiments generally relates to video encoding and decoding, and at least one other embodiment generally relates to transmitting a generated or encoded bitstream. These and other embodiments may be implemented as a computer-readable storage medium storing instructions for encoding or decoding video data according to any of the methods, apparatus, or described methods, and / or a computer-readable storage medium storing a bitstream generated according to any of the described methods.

[0081] In this application, the terms “reconstructed” and “decoded” may be used interchangeably, the terms “pixel” and “sample” may be used interchangeably, and the terms “image,” “picture,” and “frame” may be used interchangeably.

[0082] Various methods are described herein, each of which includes one or more steps or actions to achieve the described method. Unless a particular order of steps or actions is required for the normal operation of the method, the order and / or use of any particular steps and / or actions may be modified or combined. In addition, terms such as “first,” “second,” etc., may be used in various examples to modify elements, components, steps, actions, etc., such as “first decryption” and “second decryption.” The use of such terms does not imply a modified order of actions unless specifically required. In this example, the first decryption does not need to be performed before the second decryption, and may occur, for example, before, during, or overlapping with the second decryption.

[0083] Various methods and other embodiments described herein may be used to modify modules, such as the decoding module of a video encoder 200 and decoder 300, as shown in Figures 2 and 3. Furthermore, the subject matter disclosed herein may be applied to any type, format, or version of video coding, whether, for example, it is described in a standard or recommendation, whether it is existing or to be developed in the future, and whether it is an extension of any such standard or recommendation. Unless otherwise specifically indicated or technically excluded, the embodiments described herein may be used individually or in combination.

[0084] Various numerical values ​​are used in the examples described in this application. These and other specific values ​​are for illustrative purposes only, and the embodiments described are not limited to these specific values.

[0085] Figure 2 shows an exemplary video encoder. Variations of the exemplary encoder 200 are intended, but encoder 200 is described below without describing all expected variations for clarity.

[0086] Before encoding, the video sequence may undergo pre-encoding processing (201), for example, by applying a color conversion to the input color picture (e.g., from RGB 4:4:4 to YCbCr 4:2:0), or by remapping the input picture components (e.g., using histogram equalization of one of the color components) to obtain a more resilient signal distribution against compression. Metadata may be associated with the pre-processing and attached to the bitstream.

[0087] In encoder 200, the picture is encoded by encoder elements as described below. The picture to be encoded is partitioned (202) and processed in units of coding units (CUs), for example. Each unit is encoded using either intra-mode or inter-mode, for example. If a unit is encoded in intra-mode, intra-prediction is performed (260). In inter-mode, motion estimation (275) and motion compensation (270) are performed. The encoder decides whether to use intra-mode or inter-mode for encoding a unit (205), and indicates the intra / inter decision, for example, by a prediction mode flag. The prediction residual is calculated, for example, by subtracting the predicted blocks from the original image blocks (210).

[0088] Next, the predicted residual is transformed (225) and quantized (230). The quantized transformation coefficients, as well as the motion vector and other syntactic elements, are entropy coded to output a bitstream (245). The encoder can skip the transformation and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transformation and quantization, i.e., the residual is coded directly without applying any transformation or quantization process.

[0089] The encoder decodes the encoded blocks to provide a reference for further prediction. The quantized transformation coefficients are inversely quantized (240), inversely transformed (250), and the predicted residuals are decoded. The decoded predicted residuals and the predicted blocks are combined (255) to reconstruct the image blocks. An in-loop filter (265) is applied to the reconstructed picture to reduce encoding artifacts, for example, by performing deblocking / SAO (Sample Adaptive Offset) filtering. The filtered image is stored in a reference picture buffer (280).

[0090] Figure 3 shows an example of a video decoder. In the exemplary decoder 300, the bitstream is decoded by the decoder elements as described below. The video decoder 300 generally performs a decoding path that is the reverse of the encoding path described in Figure 2. The encoder 200 also generally performs video decoding as part of encoding the video data.

[0091] In particular, the decoder input includes a video bitstream that may be generated by the video encoder 200. First, the bitstream is entropy-decoded to obtain transformation coefficients, motion vectors, and other coded information (330). Picture partitioning information indicates how the picture is partitioned. Thus, the decoder may partition the picture according to the decoded picture partitioning information (335). The transformation coefficients are inversely quantized (340) and inversely transformed (350) to decode the prediction residuals. The image blocks are reconstructed by combining the decoded prediction residuals and the predicted blocks (355). The predicted blocks may be obtained from intra-prediction (360) or motion-compensated prediction (i.e., inter-prediction) (375) (370). An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (380).

[0092] The decoded picture may undergo further post-decoded processing (385), such as reverse color transformation (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or reverse remapping, which is the reverse of the remapping process performed in pre-encoded processing (201). The post-decoded processing can use metadata derived in pre-encoded processing and signaled within the bitstream. For example, the decoded image (e.g., after applying the in-loop filter (365) and / or after post-decoded processing (385) if post-decoded processing is used) may be sent to a display device for rendering to the user.

[0093] Figure 4 shows an example of a system in which various embodiments and examples described herein may be implemented. System 400 may be embodied as a device comprising various components described below and configured to perform one or more of the embodiments described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of System 400 may be embodied individually or in combination as a single integrated circuit (IC), multiple ICs, and / or individual components. For example, in at least one example, the processing elements and encoder / decoder elements of System 400 are distributed across multiple ICs and / or individual components. In various examples, System 400 is communicably coupled to one or more other systems or other electronic devices, for example, via a communication bus or via dedicated input and / or output ports. In various examples, System 400 is configured to implement one or more of the embodiments described herein.

[0094] System 400 includes, for example, at least one processor 410 configured to execute loaded instructions in order to implement various embodiments described herein. The processor 410 may include embedded memory, input / output interfaces, and various other circuits known in the art. System 400 includes at least one memory 420 (e.g., a volatile memory device and / or a non-volatile memory device). System 400 includes a storage device 440 which may include non-volatile memory and / or volatile memory, including, but not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drives, and / or optical disk drives. The storage device 440 may, in non-limiting examples, include an internal storage device, an attached storage device (including removable and non-removable storage devices), and / or a network-accessible storage device.

[0095] System 400 includes, for example, an encoder / decoder module 430 configured to process data to provide encoded or decoded video, the encoder / decoder module 430 of which may include its own processor and memory. The encoder / decoder module 430 represents a module that may be included in a device for performing encoding and / or decoding functions. As is known, the device may include one or both of the encoding and decoding modules. Furthermore, the encoder / decoder module 430 may be implemented as a separate element of System 400, or it may be incorporated into the processor 410 as a combination of hardware and software, as is known to those skilled in the art.

[0096] Program code loaded into the processor 410 or encoder / decoder 430 to perform various embodiments described herein may be stored in the storage device 440 and subsequently loaded into memory 420 for execution by the processor 410. According to various examples, one or more of the processor 410, memory 420, storage device 440, and encoder / decoder module 430 may store one or more of various items during the execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video, or a portion of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0097] In some examples, internal memory of the processor 410 and / or encoder / decoder module 430 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other examples, external memory of the processing device (for example, the processing device may be either the processor 410 or the encoder / decoder module 430) is used for one or more of these functions. The external memory may be memory 420 and / or storage device 440, for example, dynamic volatile memory and / or non-volatile flash memory. In some examples, for example, external non-volatile flash memory is used to store the television's operating system. In at least one example, high-speed external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations.

[0098] Inputs to the elements of System 400 can be provided via various input devices, as shown in Block 445. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives RF signals transmitted over the air by a broadcasting station, for example, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI®) input terminal. Another example not shown in Figure 4 is composite video.

[0099] In various examples, the input device of block 445 has associated input processing elements, each of which is known in the art. For example, the RF portion may be associated with suitable elements for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a certain frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band in order to select a signal frequency band that may be referred to as a channel in a particular example, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and / or (vi) demultiplexing to select a desired data packet stream. The RF portion of various examples includes one or more elements for performing these functions, e.g., frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion may include tuners for performing various of these functions, including, for example, down-converting the received signal to a lower frequency (e.g., an intermediate frequency or a frequency close to the baseband) or to the baseband. In one example of a set-top box, the RF section and its associated input processing elements perform frequency selection by receiving, filtering, down-converting, and filtering again to a desired frequency band of RF signals transmitted via a wired (e.g., cable) medium. In various examples, the order of the above (and other) elements is rearranged, some of these elements are removed, and / or other elements performing similar or different functions are added. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various examples, the RF section includes an antenna.

[0100] The USB and / or HDMI terminals may include their respective interface processors for connecting the system 400 to other electronic devices via USB and / or HDMI connections. It should be understood that various forms of input processing, such as Reed-Solomon error correction, may be implemented as needed, for example, within a separate input processing IC or within the processor 410. Similarly, forms of USB or HDMI interface processing may be implemented as needed, within a separate interface IC or within the processor 410. The demodulated, error-corrected, and demultiplexed stream is provided to various processing elements, including, for example, the processor 410 and encoder / decoder 430 operating in combination with memory and storage elements, to process the data stream as needed for display on an output device.

[0101] Various elements of system 400 may be provided within an integrated housing, and these elements may be interconnected within the integrated housing and transmit data between them using an internal bus known in the art, such as an Inter-IC (Inter-IC, I2C) bus, wiring, and printed circuit board.

[0102] System 400 includes a communication interface 450 that enables communication with other devices via a communication channel 460. The communication interface 450 may, but is not limited to, include transceivers configured to transmit and receive data via the communication channel 460. The communication interface 450 may, but is not limited to, include a modem or a network card, and the communication channel 460 may be implemented, for example, in a wired and / or wireless medium.

[0103] In various examples, data is streamed to system 400 or otherwise provided using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE stands for Institute of Electrical and Electronics Engineers). In these examples, the Wi-Fi signal is received via a communication channel 460 and a communication interface 450 adapted for Wi-Fi communication. In these examples, communication channel 460 is typically connected to an access point or router that provides access to an external network, including the Internet, enabling streaming applications and other over-the-top communications. In other examples, the streamed data is provided to system 400 using a set-top box that distributes data via an HDMI connection in input block 445. Still other examples provide the streamed data to system 400 using an RF connection in input block 445. As described above, various examples provide data in a non-streaming manner. Additionally, various examples use wireless networks other than Wi-Fi, e.g., cellular networks or Bluetooth® networks.

[0104] System 400 can provide output signals to various output devices, including a display 475, a speaker 485, and other peripheral devices 495. Various examples of the display 475 include, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 475 may be for a television, tablet, laptop, mobile phone, or other device. The display 475 may also be integrated with other components (for example, in a smartphone) or separate (for example, an external monitor for a laptop). Other peripheral devices 495, in various examples, include one or more of a standalone digital video disc (or digital multipurpose disc) (both terms refer to DVDs), a disc player, a stereo system, and / or a lighting system. Various examples use one or more peripheral devices 495 that provide functions based on the output of System 400. For example, a disc player performs the function of playing back the output of System 400.

[0105] In various examples, control signals are communicated between the system 400 and the display 475, speaker 485, or other peripheral devices 495 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable inter-device control with or without user intervention. Output devices may be communicably coupled to the system 400 via dedicated connections through their respective interfaces 470, 480, and 490. Alternatively, output devices may be connected to the system 400 using a communication channel 460 via a communication interface 450. The display 475 and speaker 485 may be integrated into a single unit with other components of the system 400 in an electronic device such as a television. In various examples, the display interface 470 may include a display driver, such as a timing controller (TCon) chip.

[0106] The display 475 and speaker 485 may, alternatively, be separate from one or more of the other components, for example, if the RF portion of input 445 is part of a separate set-top box. In various examples where the display 475 and speaker 485 are external components, the output signal may be provided via a dedicated output connection, for example, including an HDMI port, a USB port, or a COMP output.

[0107] The example may be implemented by the processor 410, by hardware, or by computer software implemented by a combination of hardware and software. In a non-limiting example, the example may be implemented by one or more integrated circuits. The memory 420 may be of any type appropriate for the technical environment and, in a non-limiting example, may be implemented using any suitable data storage technology such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 410 may be of any type appropriate for the technical environment and, in a non-limiting example, may include one or more of microprocessors, general-purpose computers, dedicated computers, and processors based on multi-core architectures.

[0108] Various implementations involve decoding. As used in this application, “decoding” can encompass all or part of the processes performed on a received encoded sequence to produce, for example, a final output suitable for display. In various examples, such processes typically include one or more of the processes normally performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding.

[0109] For further examples, in one instance, “decoding” refers only to entropy decoding; in another instance, “decoding” refers only to differential decoding; and in yet another instance, “decoding” refers to a combination of entropy decoding and differential decoding. Whether the phrase “decoding process” is intended to refer specifically to a subset of operations or to a broader decoding process in general becomes clear from the context of the particular description and is expected to be well understood by those skilled in the art.

[0110] Various implementations involve encoding. Similar to the above considerations regarding "decoding," the term "encoding" as used in this application may encompass all or part of the processes performed on an input video sequence to generate an encoded bitstream. In various examples, such processes typically include one or more processes performed by the encoder, such as partitioning, differential encoding, transformation, quantization, and entropy encoding.

[0111] For further examples, in one instance, “encoding” refers only to entropy coding; in another instance, “encoding” refers only to differential coding; and in yet another instance, “encoding” refers to a combination of differential coding and entropy coding. Whether the phrase “encoding process” is intended to refer specifically to a subset of operations or to a broader encoding process in general becomes clear from the context of the particular description and is expected to be well understood by those skilled in the art.

[0112] It should be noted that the coding syntax used herein for syntactic elements, such as host sender, message type, and split point, is descriptive terminology. Therefore, it does not preclude the use of other syntactic element names.

[0113] When a diagram is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0114] The implementation forms and embodiments described herein may be implemented, for example, in methods or processes, apparatus, software programs, data streams, or signals. Even if considered only in the context of a single form of implementation (e.g., considered only as a method), the implementation forms of the considered features may also be implemented in other forms (e.g., apparatus or programs). Apparatus may be implemented, for example, with appropriate hardware, software, and firmware. The method may be implemented in a processor, for example, a computer, microprocessor, integrated circuit, or programmable logic device, generally referring to a processing device. Processors also include communication devices, such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the communication of information between end users.

[0115] References to “one example” or “example,” or “one implementation” or “a certain implementation,” and other variations thereof, mean that the specific features, structures, characteristics, etc. described in relation to the example are included in at least one example. Therefore, the appearance of phrases such as “one example” or “in one example” or “in one implementation” or “in a certain implementation,” and any other variations, found in various places throughout this application, do not necessarily all refer to the same example.

[0116] Additionally, this application may refer to “determining” various types of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory. Acquiring may include receiving, retrieving, structuring, generating, and / or determining.

[0117] Furthermore, this application may also refer to “accessing” various types of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0118] Additionally, this application may refer to “receiving” various types of information. Receiving is intended to be a broad term, similar to “accessing.” Receiving information may include, for example, accessing information or retrieving information (for example, from memory). Furthermore, “receiving” typically accompanies, in some way, operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0119] For example, in the cases of "A / B", "A and / or B", and "at least one of A and B", please understand that the use of any of the following " / ", "and / or", and "at least one of" is intended to cover the selection of only the first option (A), only the second option (B), or both options (A and B). As further examples, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such phrasing is intended to cover the selection of only the first option (A), only the second option (B), only the third option (C), only the first and second options (A and B), only the first and third options (A and C), only the second and third options (B and C), or all three options (A, B, and C). This may be extended to the number of items listed, as is obvious to anyone with ordinary knowledge of this and related technologies.

[0120] Furthermore, as used herein, the word “signaling” refers, in particular, to pointing something to a corresponding decoder. In this way, in one example, the same parameters are used on both the encoder and decoder sides. Thus, for example, the encoder can send a specific parameter to the decoder (explicit signaling), and as a result, the decoder can use the same specific parameter. Conversely, if the decoder already has a specific parameter as well as other parameters, signaling may be used without transmission (implicit signaling) to allow the decoder to easily recognize and select the specific parameter. Bit savings are achieved in various examples by avoiding the transmission of any actual function. It will be understood that signaling can be achieved in various ways. For example, one or more syntactic elements, flags, etc., are used to signal information to the corresponding decoder in various examples. While this concerns the verb form of the word “signaling,” the word “signaling” can also be used as a noun in this specification.

[0121] As will be apparent to those skilled in the art, implementations can generate a wide variety of signals, for example, that are formatted to carry information that can be stored or transmitted. This information may include, for example, instructions for performing a method, or data generated by one of the implementations described. For example, a signal may be formatted to carry the bitstream of the example described. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. The signal may be transmitted over a wide variety of different wired or wireless links, as is known. The signal may be stored in a processor-readable medium and may be accessed or received from a processor-readable medium.

[0122] Many examples are provided herein. The features of the examples may be provided individually or in any combination across various categories and types of claims. Furthermore, the examples may include, individually or in any combination across various categories and types of claims, one or more of the features, devices, or embodiments described herein. For example, the features described herein may be implemented in a bitstream or signal containing information generated as described herein. The information enables a decoder to decode the bitstream, and the encoder, bitstream, and / or decoder follow any of the embodiments described. For example, the features described herein may be implemented by creating, transmitting, receiving, and / or decoding a bitstream or signal. For example, the features described herein may be implemented by a method, process, apparatus, medium for storing instructions, medium for storing data, or signal. For example, the features described herein may be implemented by a TV, set-top box, mobile phone, tablet, or other electronic device that performs decoding. A TV, set-top box, mobile phone, tablet, or other electronic device may display the resulting image (e.g., an image from residual reconstruction of a video bitstream) (e.g., using a monitor, screen, or other type of display). A TV, set-top box, mobile phone, tablet, or other electronic device may receive and decode a signal containing an encoded image.

[0123] As used herein, WTRU may refer to a user equipment (UE).

[0124] Figures 5A and 5B show an exemplary architecture (e.g., 3GPP SA4 AI4media) for partitioned inference of an artificial intelligence and / or machine learning (AI / ML) model (e.g., an AI / ML model) consisting of n layers (1..n) between the network and the WTRU. A first inference may process a first part (e.g., the first part) of the model (e.g., layers 1..k), and / or a second inference may process a second part (e.g., the second part) of the model (e.g., layers k+1..n).

[0125] The exemplary architectures shown in Figures 5A and 5B may include intermediate data distribution and access functions between the WTRU and the network (for example, for the two different scenarios shown in Figures 5A and 5B). Figure 5A shows exemplary partitioned inference between the WTRU and the network, with a media data source within the WTRU. Figure 5B shows exemplary partitioned inference between the WTRU and the network, with a media data source within the network.

[0126] In one example, as shown in Figure 5A, the media data source may originate from a WTRU associated with the network. When the media data source originates from a WTRU, the first part of the inference may be performed by the WTRU, and the second part of the inference may be performed by the network. The resulting output data may be sent back to the WTRU (e.g., from the network).

[0127] In one example, as shown in Figure 5B, the media data source may originate from a network. The media data source may be received from the network by a WTRU. When the media data source is received from the network or via a WTRU, the first part of the AI / ML model may be executed by the network (e.g., on the network side), and the second part may be executed by the WTRU (e.g., on the WTRU).

[0128] The WTRU and / or network can select the type of processing unit on which the model (e.g., an AI / ML model) will be executed (e.g., when selecting a model to run). The type of processing unit can be at least one of the following: CPU, TPU, GPU, FPGA, etc.

[0129] The first part of the AI / ML model can be compiled and / or loaded into a machine learning framework. The machine learning framework is configured and / or capable of inferring from input media data. The resulting intermediate data can be sent to a second part of the AI / ML model (for example, to confirm the inference).

[0130] Messages can be defined to wrap data exchange within an AI / ML partitioning pipeline. Messages may contain information that follows the intermediate data (e.g., information that may consist of...). This information can assist remote nodes (e.g., networks) in processing the intermediate data (e.g., by indicating one or more of the following: the model to use, the partition points to set, the number of intermediate data to copy, or how the intermediate data is encoded).

[0131] Information and intermediate data can be transported via data structures. An exemplary data structure (for example, for transporting information and / or intermediate data) may include at least one of the following: a model identifier (e.g., Model_RefID), an indication of whether the split point has changed since the previous message (e.g., ChangeOfSplitPoint), the split point used by the WTRU (e.g., SplitPointID), the next split point used by the WTRU (e.g., NextSplitPointID), the length of the intermediate data (e.g., IntermediateDataLength), the array dimensions of the intermediate data (e.g., IntermediateDataDim), an indication of whether the data has been compressed and / or the compression algorithm used (e.g., EncodingMethod), a sequential identifier of the input data (e.g., SequenceNumber), a timestamp of the intermediate data (e.g., TimeStamp), or the intermediate data (e.g., IntermediateData). An exemplary data structure (for example, for transporting information and intermediate data) may be represented as follows: Current Intermediate Data Wrapper:

[0132] [Table 1]

[0133] In the example of in-band intermediate data information transport, transport information can be transported in-band together with the intermediate data (for example, together) (for example, on the same channel).

[0134] In this example, information can be transported out of band (e.g., separately) from intermediate data (e.g., over another channel).

[0135] Figure 5C shows an example of dynamic partition point management. Messages may be exchanged between the WTRU and network devices (e.g., operators, AI / ML application providers, etc.) to manage dynamic partition points. A local decision module (e.g., within the WTRU) may be used to set the best partition point based on requirements (e.g., bandwidth, latency, energy, etc.), local information, and / or partition point range.

[0136] A local decision module, included as part of a first device (e.g., a WTRU, a network device, etc.) (e.g., located on top of it), or a decision module included as part of a second device (e.g., a WTRU, a network device, etc.) (e.g., a remote decision module), can determine where to partition (e.g., determine the optimal partition point) based on requirements (e.g., latency or prediction score) and / or current conditions (e.g., available bandwidth, energy level, available processing units, etc.).

[0137] Conditions can change. In order to maintain the initially defined requirements (for example, if conditions change), the local decision module and / or decision modules (for example, the remote decision module) may decide to change the processing unit (for example, to a processing unit on another device, e.g., from a processing unit on the first device to a processing unit on the second device). Instructions containing information ordering the change of processing unit may be sent (for example, from the first device to the second device).

[0138] The first device may determine (e.g., compute) the processing power required to process intermediate data to be sent to the second device. Based on the determined processing power, the first device may send information (e.g., processing unit recommendation) to the second device. The transmitted information may include (e.g., may include) processing unit information such as type (e.g., processing unit type such as TPU, GPU, CPU, FPGA, etc.) and / or one or more processing unit identifiers.

[0139] The first device may request and / or receive (from, for example, the second device) a list of processing units and their respective capabilities. The first device may determine and / or select one or more processing units (from, for example, the list of processing units) based on requirements (e.g., bandwidth, latency, energy, etc.) (e.g., satisfying the requirements). The first device may transmit (e.g., to the second device) the identifiers of the selected processing units.

[0140] The second device may provide the first device with the real-time status of one or more processing units (e.g., processing units in a list of processing units). For example, the second device may transmit the load of each processing unit (e.g., to the first device).

[0141] Processing unit information may be transmitted from the first device to the second device.

[0142] The first device may provide processing unit information (e.g., as metadata) for transmission to the second device. The second device (e.g., receiving the processing unit information) may process (e.g., compute) the processing unit information to determine and / or select processing units for processing the data (e.g., an AI / ML model or a subset of an AI / ML model).

[0143] Processing unit information may include one or more of the following: processing unit capacity, processing unit type, processing unit identifier, processing capacity identifier, number of parameters (e.g., number of parameters), parameter precision (e.g., parameter precision), multiply-accumulate (MAC) operation (e.g., kilo), temporary memory (e.g., MB), and processing unit real-time status.

[0144] Processing unit information may include processing unit capabilities. Processing unit capabilities may describe the capabilities required of the processing unit to process intermediate data transmitted from a first device to a second device. Processing unit capabilities may include processing unit speed, random access memory (RAM), cache memory, and / or energy consumption.

[0145] The processing unit speed (e.g., flops per clock cycle) can define the number of operations per clock cycle required by the processing unit to compute the intermediate data. RAM is the amount of memory that can be used to process the intermediate data (e.g., the minimum memory required to process the intermediate data). Cache memory may be the amount of memory used to perform inference on the intermediate data (e.g., the minimum cache (e.g., L1 / L2) memory required to perform inference on the intermediate data). Energy consumption may be the estimated energy consumption for processing the intermediate data.

[0146] Processing unit information may include the processing unit type. The processing unit type may be one or more of the following: CPU, GPU, TPU, FPGA vision processing unit (VPU), and quantum processing unit (QPU).

[0147] Processing unit information may include a processing unit identifier. The processing unit identifier may identify a processing unit associated with a processing unit capability. The first device may request and / or receive a list of processing units having each capability (e.g., from the second device). The first device may determine the processing unit capability (e.g., for each processing unit), select a processing unit (e.g., from the list of processing units), and / or transmit the processing unit identifier (e.g., associated with the selected processing unit). The processing unit identifier may be transmitted to the second device along with intermediate data. In the example, "GPU ID0", "GPU ID1", "GPU ID3", "GPU ID4", "CPU ID0", or processing unit list numbers [1,2,3,4] may be transmitted and / or received.

[0148] Processing unit information may include processing unit real-time status. Processing unit real-time status may describe the live status of each processing unit (e.g., processing unit load and / or occupied memory).

[0149] Processing unit information may include processing unit status (for example, received by the first device from the second device).

[0150] Processing unit information may include processing unit type. Processing unit information may define processing unit type (e.g., CPU, GPU, TPU, FPGA vision processing unit (VPU), or quantum processing unit (QPU)).

[0151] Processing unit information may include a processing capacity identifier. The processing capacity identifier may be used to indicate the assigned and / or recommended processing capacity (e.g., of a processing unit). Processing capacity may be negotiated during the configuration phase. Processing capacity and / or settings may be stored (e.g., during the configuration phase). Processing capacity may be identified by a processing capacity identifier.

[0152] Processing unit information may include the number of parameters (e.g., the total number of parameters). The number of parameters may indicate the (e.g., total) number of parameters in the neural network.

[0153] Processing unit information may include parameter precision. Parameter precision may be indicated as bits (e.g., number of bits) for storing a parameter (e.g., one parameter). In the example, the first indicator (e.g., "I") may be used to indicate an integer parameter. In the example, the second indicator (e.g., "F") may be used to indicate a floating-point number. In the example, if the proposed method uses a 16-bit integer to represent a parameter, the 16-bit integer may be reported as 16(I), 16I, or 16Int.

[0154] Processing unit information may include multiply-accumulate (MAC) operations (e.g., kilo). MAC operations may represent the worst-case number of MAC operations (e.g., per pixel, or general) during the inference phase. A MAC operation can add the product of two numbers to an accumulator. Processing unit information may also include temporary memory (e.g., MB). Temporary memory may indicate memory used to store output feature maps for all intermediate layers (e.g., forward passes).

[0155] The second device may provide processing unit real-time status (for example, to the first device). The processing unit real-time status may include one or more of the following: processing unit load, overall processing unit load, selected processing unit type, or selected processing unit identifier.

[0156] The processing unit load may identify the load of a processing unit (e.g., as a percentage from 0% to 100%). The overall processing unit load may identify the load of all processing units (e.g., GPUs 1 through 4). In the example, the overall processing unit load may be expressed as a percentage (e.g., from 0% to 100%). The selected processing unit type may define the current processing unit type used to process the model portion (e.g., the AI / ML model portion). The selected processing unit identifier may identify the current processing unit used to process the model portion (e.g., the AI / ML model portion). The selected processing unit identifier may belong to a list of candidate processing units shared across devices (e.g., "GPU ID0", "GPU ID1", "GPU ID3", "GPU ID4", "CPU ID0", or processing unit list number [1,2,3,4]).

[0157] Figure 6 is a block diagram showing an example of processing unit information exchange.

[0158] Processing unit messages and / or metadata may contain processing unit information. Processing unit messages may be used by a local decision module (e.g., hosted by the first device) and an AI / ML model manager (e.g., hosted by the second device) to exchange information (e.g., about the capabilities of the first and / or second devices, such as processing capacity). The exchanged information may be compiled by the local decision module of the first device. The exchanged information may be used by the local decision module to determine the appropriate processing unit (e.g., the processing unit that can process the most intermediate data).

[0159] The metadata associated with the intermediate data may include instructions for an AI / ML processing module (e.g., on a second device) to decode and process the intermediate data.

[0160] Features associated with control data for providing processing unit information (e.g., before and during partitioned inference) are provided herein.

[0161] The first device may send a message to the second device to receive and / or obtain (e.g., obtain) status information regarding the processing usage and / or requirements of the second device. An AI / ML model manager on the second device (e.g., as shown in Figure 6) may process (e.g., handle) the message. The AI / ML model manager may respond with information regarding the usage and / or availability of processing units (e.g., CPU, GPU, TPU, FPGA, VPU, etc.) on the second device. The first device may receive information regarding the availability of processing units and / or on the device. A local decision module may process (e.g., handle) the response. The local decision module may compile the information contained in the response. The local decision module may determine and / or select the best split point and / or best processing unit for use (e.g., on the first device and / or on the second device). In the example, a decision module (e.g., a remote decision module) may determine and / or select the best set of processing units and / or split points for use (e.g., on a first device and / or a second device). In the example, the second device may send instructions to the first device for the selected best set of processing units and / or split points.

[0162] A processing unit on the second device (e.g., an identified processing unit) may be communicated to the first device via metadata, or it may be associated with intermediate data (e.g., in a 2-byte format that can describe the type and ID of the processing unit used) (e.g., it may be directly associated).

[0163] The messages exchanged between the first device and the second device may include the following exemplary structure: ProcessingUnit message:

[0164] [Table 2]

[0165] Here, for example, <Host_name> : This could be a "network" or a "WTRU". <Message_Type> : This could be a "request" or a "response." <Model_RefID> : This can be a unique model identifier. SplitPoints: If the message type is "Request," it may be a list of split points that have the requested processing unit; if the message type is "Response," it may be a list of split points that have the updated configuration. <splitpointid>This may also be a unique identifier for the division point. <processingunittype>This can also be "CPU", "GPU", "TPU", or "FPGA". <processingunitid>This may be "GPU ID0", "GPU ID1", "GPU ID3", "GPU ID4", "CPU ID0", "TPU", or "FPGA". <architecturetype>These could be the main types of partitioning architectures (e.g., CNN, FC, ...) that can influence the selection of processing units. <processingunitload>This could also be the load on the processing unit (for example, a percentage from 0% to 100%).

[0166] Figure 7 shows an exemplary message exchange between a first device and a second device (for example, a second device containing an AI / ML model manager running on it). In the example, messages may be exchanged by performing one or more of the following, as described herein:

[0167] As shown in Figure 7, in 1, the first device using a processing unit request may request information from the second device. A processing unit request may include one or more requests for information associated with a split point, a request containing information associated with a processing unit, a request containing information associated with the processing unit capability, or a request containing information associated with the processing unit real-time status.

[0168] In the example, a request for information associated with a split point may involve the first device communicating a list of split points for which information can be expected. An exemplary request for information associated with a split point may include one or more of the following: Host_name (e.g., "WTRU"), Message_type (e.g., "Request"), or SplitPoints (e.g., [{SplitPointID1},...,{SplitPointIDn}]).

[0169] In the example, a request for information associated with a processing unit (e.g., associated with a split point) may involve the first device communicating a list of proposed processing units and split points. A request for information associated with a processing unit may include one or more of the following: Host_name (e.g., "WTRU"), Message_type (e.g., "Request"), or SplitPoints (e.g., [{SplitPointID1,ProcessingUnitType1},...,{SplitPointIDn,ProcessingUnitTypen}]).

[0170] In the example, a request containing information associated with processing unit capabilities may include one or more of the following: Host_name (e.g., "WTRU"), Message_type (e.g., "Request"), or processing unit capabilities (e.g., as described herein).

[0171] In the example, a request containing information associated with the processing unit real-time status may include one or more of the following: Host_name (e.g., "WTRU"), Message_type (e.g., "Request"), or processing unit real-time status (e.g., as described herein).

[0172] The AI / ML model manager (for example, of a second device) may receive (e.g., collect) a request. The request may include instructions for the requested information regarding the split points and / or processing units. As shown in Figure 7, in 2, the AI / ML model manager may determine, retrieve, or calculate a response (e.g., a response based on the requested information) that includes the requested information.

[0173] In step 3, the AI / ML model manager may respond to the request (e.g., provide the requested information). The response may include one or more of the following: information about the split point, information about the processing unit (e.g., associated with the split point), information about the processing unit's capabilities, or information about the processing unit's real-time status.

[0174] In the example, a response containing information associated with a split point may include one or more of the following: Host_name (e.g., "Network"), Message_type (e.g., "Response"), or SplitPoints (e.g., [{SplitPointID1,ProcessingUnitType1,ArchitectureType1},...,{SplitPointIDn,ProcessingUnitTypen,ArchitectureTypen}]).

[0175] In the example, a response containing information associated with a processing unit (e.g., associated with a split point) may include one or more of the following: Host_name (e.g., "Network"), Message_type (e.g., "Response"), or SplitPoints (e.g., [{SplitPointID1,ProcessingUnitType1,ProcessingUnitID1,ArchitectureType1},...,{SplitPointIDn,ProcessingUnitTypen,ProcessingUnitIDn,ArchitectureTypen}]).

[0176] In the example, the response, which includes information associated with the processing unit capabilities, is Host_name (e.g., "Network"), Message_type (e.g., "Response"), or ProcessingUnitInfos:[{ProcessingUnitType:ProcessingUnitType1,ProcessingUnitID:ProcessingUnitIDi>,ProcessingUnitSpeed: <processingunitspeed>,ProcessingUnitRAM: <processingunitram>,ProcessingUnitCache:<ProcessingUnitCacheL1,ProcessingUnitCacheL2>,ProcessingUnitEnergyConsumption: <processingunitenergyconsumption>It may include one or more of the following:},...}]).

[0177] In the example, a response containing information associated with the real-time status of a processing unit may include one or more of the following: Host_name (e.g., "Network"), Message_type (e.g., "Response"), or processing unit real-time status (e.g., [{ProcessingUnitType:ProcessingUnitType1,ProcessingUnitID:ProcessingUnitID1,ProcessingUnitLoad:ProcessingUnitLoad1}...]).

[0178] Features associated with metadata for providing processing unit information may be provided herein.

[0179] The first device may provide processing unit information as metadata (for example, to send to a second device). The receiving device (e.g., the second device) may retrieve or compute processing information to select which processing unit will process the data (e.g., an AI / ML model or a subset of AI / ML models).

[0180] Processing information may include the processing unit type (e.g., ProcessingUnitType) and / or the processing unit ID (e.g., ProcessingUnitID).

[0181] The processing unit type (e.g., ProcessingUnitType) is a parameter that can be transmitted (e.g., as 8 bits / 1 byte). In the example, the processing unit type value may include one or more of 0x00 (e.g., CPU), 0x01 (e.g., GPU), 0x02 (e.g., TPU), or 0x03 (e.g., FPGA). Please note that the exemplary processing unit type values ​​described herein may not be exhaustive, as one or more processing units (e.g., VPU, QPU, etc.) may be provided.

[0182] The Processing Unit ID (e.g., ProcessingUnitID) is a parameter that can be transmitted (e.g., as 8 bits / 1 byte). In the example, the Processing Unit ID (e.g., ProcessingUnitID) may be used to transmit information in addition to the Processing Unit Type (e.g., ProcessingUnitType) parameter. For example, there could be 256 entities for the selected (e.g., chosen) Processing Unit Type (e.g., ProcessingUnitType). For example, ProcessingUnitType=0x01 and ProcessingUnitID=0x02 (e.g., the selected processing unit for the second device may compile and / or load a partition model for "GPU:2").

[0183] The processing unit type and / or processing unit ID (e.g., ProcessingUnitType and / or ProcessingUnitID) may be parameters (e.g., they may not be required). The second device may use the processing unit type and / or processing unit ID if they exist. The second device may not set the recommended processing unit type and / or processing unit ID if the recommended processing unit is not available (e.g., if the second device has received the processing unit type and / or processing unit ID from the first device).

[0184] An intermediate data wrapper containing processing unit information may be used. The intermediate data wrapper may include one or more of the following: a model identifier, whether the split point has changed since the previous message, the next split point (for example, used by the first device), the length of the intermediate data, the array dimensions of the intermediate data, an indication of whether the data has been compressed, an indication of whether a compression algorithm was used, a sequential identifier of the input data, a timestamp of the intermediate data, the processing unit type, the processing unit ID, or the intermediate data. In the example, an intermediate data wrapper containing processing unit information may be represented as follows:

[0185] [Table 3]

[0186] Features associated with processing unit type metadata (for example, for selecting the required processing unit) may be provided herein.

[0187] Figure 8 shows an exemplary method for analyzing and / or processing the processing unit type and processing unit ID.

[0188] The first partitioned model may be executed on a processing unit (e.g., on a first device). The first partitioned model may generate intermediate data resulting from the inference. The intermediate data may be transmitted to a second device via a network. The second device may complete the inference process (e.g., terminate). The second device may transmit the inference results (e.g., score) to the first device.

[0189] Intermediate data may be transmitted (for example, to a second device) along with information indicating how to process the intermediate data. Processing unit type and / or processing unit ID may be transmitted (for example, in addition to the intermediate data and / or in addition to any existing information). The second device may parse the intermediate data and / or the existing information.

[0190] As shown in Figure 8, the processing unit type value can be read at point 1.

[0191] In step 2, the second device may determine whether a processing unit type exists. If a processing unit type exists, the exemplary method proceeds to step 3. If a processing unit type does not exist, the exemplary method proceeds to step 5. In step 5, the second device may remain on the current processing unit (e.g., the current processing unit type and / or the current processing unit ID).

[0192] In step 3, the second device can read the processing unit ID.

[0193] In step 4, the second device may determine whether a processing unit ID exists. If a processing unit ID does not exist, the exemplary method proceeds to step 6, and the second device may use a default processing unit (e.g., ID=0). If a processing unit ID exists, the exemplary method proceeds to step 7.

[0194] In step 7, the second device may apply (e.g., a new) processing unit type and processing unit ID for the partitioned model. The AI / ML model may be loaded into (e.g., the new) processing unit. The second device may send a message (e.g., a response to the first device). The message may indicate whether (e.g., the new) processing unit is in use. The message may be associated with (e.g., include) the (e.g., final) inference result, or it may be sent to the first device as a separate message.

[0195] Figure 9 is a block diagram illustrating an example of metadata and / or processing unit information (for example, how intermediate data and related information may be transmitted from one device to another, such as from a first device to a second device).

[0196] In this example, the application may run on a first device. The application may depend on an AI / ML service. The AI / ML service may implement a distributed inference method. The distributed inference method may include an AI / ML model. The AI / ML model may include one or more layers (e.g., n layers (1..n)) between the second device and the first device. The first AI / ML inference engine may process the first part of the AI / ML model (e.g., partitioning M1 (e.g., layers 1..k)), and the second AI / ML inference engine may process the second part of the AI / ML model (e.g., partitioning M2 (e.g., layers k+1..n)). M1 may run on the first device, and M2 may run on the second device.

[0197] A decision module (e.g., located on a first device) may be designated as a local decision module. A decision model (e.g., a local decision model) may take in (e.g., receive) input data such as private information, division point ranges, and / or requirements (e.g., bandwidth, latency, energy, etc.).

[0198] A decision module (e.g., a local decision model) may send messages to an AI / ML model manager (e.g., on a second device). These messages may request information about the hardware capabilities of the second device. Messages may be sent periodically (e.g., periodically) or at any given time.

[0199] The second device may send a response (e.g., a reply message) that includes hardware capabilities.

[0200] A decision module (for example, a local decision module on a first device) may compile the received information (e.g., private and / or local information, division point ranges, requirements, and / or second device hardware capabilities).

[0201] A decision module (for example, a local decision module) can distribute the division points to a division function module.

[0202] The partitioning module can manage the partitioning of AI / ML models. The partitioning module may also update the model (for example, in an AI / ML inference engine).

[0203] In addition to the division points, the decision module (e.g., the local decision module) may distribute processing unit information to the AI / ML processing module (e.g., the first device). Processing unit information (e.g., dedicated to the second device) may be sent to the intermediate data distribution module.

[0204] The processing unit information may include information about the type of processing unit to be used on a first device (e.g., locally) and / or the type of processing unit to be used on a second device.

[0205] A decision module (e.g., a local decision module) can execute (e.g., execute sequentially) and evaluate the input. The decision module can deliver (e.g., if necessary) (e.g., new) split point and / or (e.g., new) processing unit information.

[0206] The AI / ML inference engine (for example, in the first device) can process M1 from the input data and deliver intermediate data (for example, directed to the second device) to process M2.

[0207] Intermediate data may be sent to an intermediate data distribution function that can prepare it for transmission (for example, to a second device). Preparation may include serializing, encoding, packaging, and / or encapsulating the data.

[0208] The intermediate data distribution function may associate the intermediate data with information for a second device to decode and process. The information for the second device to decode and process may be enriched with bytes (e.g., 2 bytes), namely the processing unit type (e.g., ProcessingUnitType) and / or the processing unit ID (e.g., ProcessingUnitID).

[0209] The processing unit type (e.g., ProcessingUnitType) may define the type of processing unit that a second device may (or should) use to process intermediate data.

[0210] The processing unit ID (e.g., ProcessingUnitID) can define an identifier for a processing unit (for example, if there are multiple processing units of the same type (e.g., devices), e.g., GPU:0, GPU:2, etc.).

[0211] As shown in Figure 9, metadata may include informational data used by a second device to decode and / or process the intermediate data. Metadata may be transmitted in-band (e.g., together with the intermediate data) or out-of-band (e.g., on a channel separate from the channel on which the intermediate data is transmitted).

[0212] Intermediate data and metadata can be read by an intermediate data access function (for example, on a second device).

[0213] The partitioning point information can be sent to the AI / ML partitioning function module. The AI / ML partitioning function module can manage the partitioning of the AI / ML model (for example, if necessary).

[0214] Processing unit information (e.g., processing unit type and / or processing unit ID) may be sent to the AI / ML inference engine. This processing unit information may be used by the AI / ML framework to compile and / or load the partitioned model. In the example, the AI / ML framework may be Tensorflow and / or PyTorch. If the specified processing unit is not available, the processing unit information may be sent back to the first device (e.g., along with the inference results).

[0215] Intermediate data may be sent to the AI / ML inference engine (for example, to determine the inference process when processing the second part of the model, M2). Inference results may be sent to the results delivery module.

[0216] The results delivery module may prepare the inference results. This preparation may include, for example, packaging or encapsulating the inference results.

[0217] The processing unit status may be transmitted (for example, along with the inference results). If the processing unit status is transmitted (for example, along with the inference results), the processing unit status may indicate (for example, to the first device and its local decision module) whether the selection of the processing unit was relevant or relevant.

[0218] While features and elements are described herein in specific combinations, those skilled in the art will understand that each feature or element may be used alone or in any combination with other features and elements. In addition, the methods described herein may be implemented in computer programs, software, or firmware embedded in computer-readable media for execution by a computer or processor. Examples of computer-readable media include electronic signals (transmitted via wired or wireless connections) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital multi-purpose disks (DVDs). A processor associated with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.< / processingunitenergyconsumption> < / processingunitram> < / processingunitspeed> < / processingunitload> < / architecturetype> < / processingunitid> < / processingunittype> < / splitpointid>

Claims

1. The first device, A processor is provided, and the processor is A method for determining processing unit information associated with a processing unit of a second device, wherein the processing unit information is associated with intermediate data, and the processing unit information comprises at least an indication of the type of processing unit to be used by the second device to process the intermediate data, and a processing unit identifier associated with the processing unit of the second device. The processing unit information and the intermediate data are transmitted to the second device. A first device configured to receive processing unit status associated with the intermediate data from the second device.

2. The first device according to claim 1, wherein the processor is further configured to receive instructions from the second device regarding the hardware capabilities associated with the second device, and the processing unit information is determined based on the hardware capabilities associated with the second device.

3. The first device according to claim 1 or 2, wherein the processor is further configured to transmit the processing unit information to the second device as metadata.

4. The first device according to claim 3, wherein the metadata is used by the second device to decode or process the intermediate data.

5. The first device according to claim 3 or 4, wherein the metadata is used by the second device to select or identify the processing unit of the second device.

6. The processor is further configured to transmit a processing unit recommendation to the second device as part of the metadata, wherein the processing unit recommendation is based on processing capacity, the first device according to any one of claims 3 to 5.

7. The first device according to any one of claims 3 to 6, wherein the metadata is transmitted via a channel different from the intermediate data.

8. The first device according to any one of claims 1 to 7, wherein the processor is further configured to receive inference results from the second device, and the inference results are associated with a score.

9. The first device according to any one of claims 1 to 7, wherein the processing unit status associated with the intermediate data includes an indication that the processing unit of the second device is unavailable.

10. The first device according to claim 9, wherein the processor is further configured to receive a response message and an inference result from the second device based on the instruction that the processing unit of the second device is unavailable, the response message indicating that the processing unit of the second device is unavailable and the processing unit information.

11. The first device according to any one of claims 1 to 10, wherein the processing unit status includes at least one instruction among the processing unit load, the processing unit type, and the processing unit identifier.

12. A method performed by a first device, Determining processing unit information associated with a processing unit of a second device, wherein the processing unit information is associated with intermediate data, and the processing unit information comprises at least an indication of the type of processing unit to be used by the second device to process the intermediate data, and a processing unit identifier associated with the processing unit of the second device. The processing unit information and the intermediate data are transmitted to the second device. A method comprising receiving a processing unit status associated with the intermediate data from the second device.

13. The method according to claim 12, further comprising receiving instructions from the second device regarding hardware capabilities associated with the second device, wherein the processing unit information is determined based on the hardware capabilities associated with the second device.

14. The method according to claim 12 or 13, further comprising transmitting the processing unit information to the second device as metadata.

15. The method according to claim 14, wherein the metadata is used by the second device to decode or process the intermediate data.

16. The method according to claim 14 or 15, wherein the metadata is used by the second device to select or identify the processing unit of the second device.

17. The method according to any one of claims 14 to 16, further comprising transmitting a processing unit recommendation to the second device as part of the metadata, wherein the processing unit recommendation is based on processing capacity.

18. The method according to any one of claims 14 to 17, wherein the metadata is transmitted via a channel different from the intermediate data.

19. The method according to any one of claims 12 to 18, further comprising receiving an inference result from the second device, wherein the inference result is associated with a score.

20. The method according to any one of claims 12 to 18, wherein the processing unit status associated with the intermediate data includes an indication that the processing unit of the second device is unavailable.

21. The method according to claim 20, further comprising receiving a response message and an inference result from the second device based on the instruction that the processing unit of the second device is unavailable, wherein the response message indicates that the processing unit of the second device is unavailable and the processing unit information.

22. The method according to any one of claims 12 to 21, wherein the processing unit status includes at least one instruction among the processing unit load, the processing unit type, and the processing unit identifier.

23. The second device is A processor is provided, and the processor is The first device receives processing unit information and intermediate data, the processing unit information is associated with a processing unit of the second device, the processing unit information is associated with the intermediate data, and the processing unit information includes at least an indication of the type of processing unit to be used by the second device to process the intermediate data, and a processing unit identifier associated with the processing unit of the second device. In response to receiving the processing unit information and the intermediate data, the processing unit status is determined. A second device configured to transmit the processing unit status to the first device.

24. The second device according to claim 23, wherein the processor is further configured to transmit instructions for hardware capabilities associated with the second device to the first device, and the instructions for hardware capabilities are associated with a processing unit type of the second device.

25. The second device according to claim 23 or 24, wherein the processor is further configured to receive the processing unit information from the first device as part of metadata.

26. The second device according to claim 25, wherein the processor is further configured to decode the intermediate data based on the received metadata.

27. The second device according to any one of claims 25 to 26, wherein the processor is further configured to identify the processing unit of the second device based on the metadata.

28. The second device according to any one of claims 25 to 27, wherein the processor is further configured to receive processing unit recommendations from the first device as part of the metadata, the processing unit recommendations being related to the amount of processing power required to process the intermediate data.

29. The second device according to any one of claims 25 to 28, wherein the metadata is received via a channel different from the intermediate data.

30. The processor is further configured to transmit inference results to the first device, the inference results being associated with a score, the second device according to any one of claims 23 to 29.

31. The second device according to any one of claims 23 to 29, wherein the processing unit status associated with the intermediate data includes an indication that the processing unit of the second device is unavailable.

32. The processor is further configured to transmit a response message and an inference result based on the instruction that the processing unit of the second device is unavailable, the response message indicating the unavailability of the processing unit of the second device and the processing unit information, the second device according to claim 31.

33. The second device according to any one of claims 23 to 32, wherein the processing unit status includes at least one instruction among the processing unit load, the processing unit type, and the processing unit identifier.

34. A method performed by a second device, Receiving processing unit information and intermediate data from a first device, wherein the processing unit information is associated with a processing unit of the second device, the processing unit information is associated with the intermediate data, and the processing unit information comprises at least an indication of the type of processing unit to be used by the second device to process the intermediate data, and a processing unit identifier associated with the processing unit of the second device. In response to receiving the processing unit information and the intermediate data, the processing unit status is determined. A method comprising transmitting the processing unit status to the first device.

35. The method according to claim 34, further comprising transmitting a hardware capability instruction associated with the second device to the first device, wherein the hardware capability instruction is associated with a processing unit type of the second device.

36. The method according to claim 34 or 35, further comprising receiving the processing unit information from the first device as metadata.

37. The method according to claim 36, further comprising decoding the intermediate data based on the received metadata.

38. The method according to claim 36 or 37, further comprising identifying the processing unit of the second device based on the metadata.

39. The method according to any one of claims 36 to 38, further comprising receiving a processing unit recommendation from the first device as part of the metadata, wherein the processing unit recommendation is related to the amount of processing capacity required to process the intermediate data.

40. The method according to any one of claims 36 to 39, wherein the metadata is received via a channel different from the intermediate data.

41. The method according to any one of claims 34 to 40, further comprising transmitting an inference result to the first device, wherein the inference result is associated with a score.

42. The method according to any one of claims 34 to 40, wherein the processing unit status associated with the intermediate data includes an indication that the processing unit of the second device is unavailable.

43. The method according to claim 42, further comprising sending a response message and an inference result based on the instruction that the processing unit of the second device is unavailable, wherein the response message indicates the unavailability of the processing unit of the second device and the processing unit information.

44. The method according to any one of claims 34 to 43, wherein the processing unit status includes at least one instruction among the processing unit load, the processing unit type, and the processing unit identifier.

45. A computer program product comprising program code instructions for implementing steps of the method according to at least one of claims 12-22 or 34-44, which are stored in a non-temporary computer-readable medium and executed by at least one processor.

46. A computer program, when executed by a processor, comprising program code instructions for implementing a step of the method according to at least one of claims 12 to 22 or 34 to 44.

47. Video data including information representing an encoded output generated according to one of the methods described in any one of claims 12 to 22 or claims 34 to 44.

Images