METHODS FOR SPLITTING GRAPHS OF NEURAL NETWORK MODELS

Abstract

The invention relates to a method for splitting a neural network model graph, where the first endpoint receives information describing the graph structure of the machine learning model, ML, where the ML has multiple nodes. The first endpoint receives split point information indicating the splitting of the ML into at least a first and a second submodel, each node belonging to one of the submodels, where the split point information identifies a node of the ML model as the split point node. The first endpoint receives at least the output of the first submodel by executing the first submodel on the model input and sending the first submodel output from the first endpoint to the second endpoint. The second endpoint can execute the second submodel on the outputs of the first submodel to obtain the model output.

Description

METHOD TO SPLIT A NEURAL NETWORK MODEL GRAPHCROSS-REFERENCE

[0001] This application claims the priority of European Patent Application No. 23306913.7, filed 6 November 2023, entitled “Method to Split a Neural Network Model Graph,” and of European Patent Application No. 24305115.8, filed 19 January 2024, entitled “Method to Split a Neural Network Model Graph,” both of which are incorporated herein by reference in their entirety.BACKGROUND

[0002] The present disclosure relates to machine learning (ML) models. New uses for machine learning models are constantly being developed. However, as the usefulness of such models increases, the computational complexity of those models continues to grow. It may be difficult to implement a complete machine learning model entirely on one computing device, particularly where the computing device is consumer level user equipment.SUMMARY

[0003] Briefly stated, a method according to some embodiments comprises: obtaining information describing a graph structure of a machine learning, ML, model, wherein the ML model has a plurality of nodes; obtaining split point information indicating a split of the ML model into at least a first sub-model and a second sub-model, each of the nodes being in one of the sub-models, wherein the split point information identifies a node of the ML model as a split point node; at a first endpoint, obtaining at least a first sub-model output by executing the first sub-model on a model input; and sending the first sub-model output from the first endpoint to a second endpoint.

[0004] An apparatus according to some embodiments comprises one or more processors configured to perform at least: obtaining information describing a graph structure of a machine learning, ML, model, wherein the ML model has a plurality of nodes; obtaining split point information indicating a split of the ML model into at least a first sub-model and a second submodel, each of the nodes being in one of the sub-models, wherein the split point information identifies a node of the ML model as a split point node; at a first endpoint, obtaining at least a first sub-model output by executing the first sub-model on a model input; and sending the first sub-model output from the first endpoint to a second endpoint.

[0005] In some embodiments, the first sub-model output is an output of a first node in the first sub-model, and the embodiment further comprises: obtaining a second sub-model output by the executing of the first sub-model on the model input, the second sub-model output being an output of a second node in the first sub-model; and sending the second sub-model output to the second endpoint.

[0006] Some embodiments further comprise, in communication with the first endpoint, selecting a naming or numbering convention for outputs of the first sub-model, wherein obtaining the first sub-model output comprises obtaining information identifying each submodel output according to the naming or numbering convention.

[0007] In some embodiments, each sub-model output is a tensor, and the naming or numbering convention comprises assigning a name to each tensor.

[0008] An apparatus according to some embodiments comprises: a processor; and a non- transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any of the methods described herein.

[0009] An apparatus according to some embodiments comprises at least one processor configured to perform any of the methods described herein.

[0010] Some embodiments include a computer-readable medium storing instructions for causing one or more processors to perform any of the methods described herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The following detailed description will be better understood when read in conjunction with the appended drawings, in which there are shown examples of one or more of the multiple embodiments of the present disclosure. It should be understood, however, that the embodiments described herein are not limited to the precise arrangements and instrumentalities shown in the drawings.

[0012] FIG. 1 A is a system diagram illustrating an example communications system according to some embodiments.

[0013] FIG. 1 B is a system diagram illustrating an example wireless transmit / receive unit (WTRU) that may be used within the communications system illustrated in FIG. 1 A according to some embodiments.

[0014] FIG. 1C is a system diagram illustrating an example apparatus on which some embodiments may be implemented.

[0015] FIG. 2A illustrates split inference between the UE and network, with media data source in the UE.

[0016] FIG. 2B illustrates split inference between the UE and network, with media data source in the network.

[0017] FIGs. 3A and 3B are two representations of the same model. A list of nodes or layers is provided in FIG. 3A. A directed acyclic graph is provided in FIG. 3B.

[0018] FIGs. 4A-4C provide some examples of different model topologies. FIG. 4A illustrates a model with one branch. FIG. 4B illustrates a model with one or two branches. FIG. 4C illustrates a model with multiple interconnected branches.

[0019] FIG. 5 is a schematic illustration of two devices implementing a split operation on an Al model using dynamic graph based Al model splitting according to an embodiment described herein.

[0020] FIG. 6 is a call flow diagram illustrating a method performed in some embodiments.

[0021] FIGs. 7A-7B illustrate examples of node numbering that may be used in some embodiments.

[0022] FIG. 8 is a flow diagram illustrating a method performed in some embodiments.

[0023] FIGs. 9A and 9B illustrate examples of node numbering.

[0024] FIG. 10 illustrates an ML model represented as an acyclic directed graph and divided into a first part and a second part.

[0025] FIG. 11 A illustrates an example ML model.

[0026] FIG. 11 B illustrates a first part of the model of FIG. 11A.

[0027] FIG. 11C illustrates a second part of the model of FIG. 11A.

[0028] FIG. 12 is a functional block diagram illustrating operations performed in an example embodiment.

[0029] FIG. 13A illustrates the output properties of an example first model part.

[0030] FIG. 13B illustrates the input properties of an example second model part.

[0031] FIGs. 14A-14C illustrate an example of a split machine learning model, with FIG. 14A representing the first sub-model and FIGs. 14B-14C representing the second sub-model.DETAILED DESCRIPTIONOverview of example communications systems

[0032] FIG. 1A is a diagram illustrating an example communications system 100 in which one or more disclosed embodiments may be implemented. The communications system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communications system 100 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communications systems 100 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block-filtered OFDM, filter bank multicarrier (FBMC), and the like.

[0033] As shown in FIG. 1A, the communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN 104, a ON 106, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a “station” and / or a “STA”, may be configured to transmit and / or receive wireless signals and may include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (loT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot and / or other wireless devices operating in an industrial and / or an automated processing chain contexts), a consumer electronics device, a device operating on commercial and / or industrial wireless networks, and the like. Any of the WTRUs 102a, 102b, 102c and 102d may be interchangeably referred to as a UE.

[0034] The communications systems 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106, the Internet 110, and / or the other networks 112. By way of example, the base stations 114a, 114b may be a basetransceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a gNB, a NR NodeB, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0035] The base station 114a may be part of the RAN 104, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and / or the base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage for a wireless service to a specific geographical area that may be relatively fixed or that may change over time. The cell may further be divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one for each sector of the cell. In an embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in desired spatial directions.

[0036] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0037] More specifically, as noted above, the communications system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a in the RAN 104 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 116 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High-Speed Downlink (DL) Packet Access (HSDPA) and / or High-Speed UL Packet Access (HSUPA).

[0038] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA),which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE- Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0039] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR Radio Access , which may establish the air interface 116 using New Radio (NR).

[0040] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may implement LTE radio access and NR radio access together, for instance using dual connectivity (DC) principles. Thus, the air interface utilized by WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., a eNB and a gNB).

[0041] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

[0042] The base station 114b in FIG. 1A may be a wireless router, Home Node B, Home eNode B, or access point, for example, and may utilize any suitable RAT for facilitating wireless connectivity in a localized area, such as a place of business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a roadway, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR etc.) to establish a picocell or femtocell. As shown in FIG. 1A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not be required to access the Internet 110 via the CN 106.

[0043] The RAN 104 may be in communication with the CN 106, which may be any type of network configured to provide voice, data, applications, and / or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have varying quality of service (QoS) requirements, such as differing throughput requirements,latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like. The CN 106 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions, such as user authentication. Although not shown in FIG. 1A, it will be appreciated that the RAN 104 and / or the CN 106 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 104 or a different RAT. For example, in addition to being connected to the RAN 104, which may be utilizing a NR radio technology, the CN 106 may also be in communication with another RAN (not shown) employing a GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0044] The CN 106 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or the other networks 112. The PSTN 108 may include circuit-switched telephone networks that provide plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP) and / or the internet protocol (IP) in the TCP / IP internet protocol suite. The networks 112 may include wired and / or wireless communications networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, which may employ the same RAT as the RAN 104 or a different RAT.

[0045] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links). For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0046] FIG. 1 B is a system diagram illustrating an example WTRU 102. As shown in FIG. 1 B, the WTRU 102 may include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138, among others. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0047] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, oneor more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. While FIG. 1 B depicts the processor 118 and the transceiver 120 as separate components, it will be appreciated that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0048] The transmit / receive element 122 may be configured to transmit signals to, or receive signals from, a base station (e.g., the base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In an embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF and light signals. It will be appreciated that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0049] Although the transmit / receive element 122 is depicted in FIG. 1 B as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0050] The transceiver 120 may be configured to modulate the signals that are to be transmitted by the transmit / receive element 122 and to demodulate the signals that are received by the transmit / receive element 122. As noted above, the WTRU 102 may have multimode capabilities. Thus, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11 , for example.

[0051] The processor 118 of the WTRU 102 may be coupled to, and may receive user input data from, the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and / orthe removable memory 132. The non-removable memory 130 may includerandom-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0052] The processor 118 may receive power from the power source 134, and may be configured to distribute and / or control the power to the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0053] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 116 from a base station (e.g., base stations 114a, 114b) and / or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[0054] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a Virtual Reality and / or Augmented Reality (VR / AR) device, an activity tracker, and the like. The peripherals 138 may include one or more sensors, the sensors may be one or more of a gyroscope, an accelerometer, a hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0055] The WTRU 102 may include a full duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for both the UL (e.g.,for transmission) and downlink (e.g., for reception) may be concurrent and / or simultaneous. The full duplex radio may include an interference management unit to reduce and or substantially eliminate self-interference via either hardware (e.g., a choke) or signal processing via a processor (e.g., a separate processor (not shown) or via processor 118). In an embodiment, the WRTU 102 may include a half-duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for either the UL (e.g., for transmission) or the downlink (e.g., for reception)).

[0056] Although the WTRU is described in FIGs. 1A-1B as a wireless terminal, it is contemplated that in certain representative embodiments that such a terminal may use (e.g., temporarily or permanently) wired communication interfaces with the communication network.

[0057] In representative embodiments, the other network 112 may be a WLAN.

[0058] In view of FIGs. 1 A-1 B, and the corresponding description , one or more, or all, of the functions described herein may be performed by one or more emulation devices (not shown). The emulation devices may be one or more devices configured to emulate one or more, or all, of the functions described herein. For example, the emulation devices may be used to test other devices and / or to simulate network and / or WTRU functions.

[0059] The emulation devices may be designed to implement one or more tests of other devices in a lab environment and / or in an operator network environment. For example, the one or more emulation devices may perform the one or more, or all, functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network in order to test other devices within the communication network. The one or more emulation devices may perform the one or more, or all, functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation device may be directly coupled to another device for purposes of testing and / or may performing testing using over-the-air wireless communications.

[0060] The one or more emulation devices may perform the one or more, including all, functions while not being implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation devices may be utilized in a testing scenario in a testing laboratory and / or a non-deployed (e.g., testing) wired and / or wireless communication network in order to implement testing of one or more components. The one or more emulation devices may be test equipment. Direct RF coupling and / or wireless communications via RF circuitry (e.g., which may include one or more antennas) may be used by the emulation devices to transmit and / or receive data.

[0061] The embodiments described herein are not limited to being implemented on a WTRU. Such embodiments may be implemented using other systems, such as the system of FIG. 1C. FIG. 1C is a system diagram illustrating an example set of interfaces for a system according to some embodiments. An extended reality display device, together with its control electronics, may be implemented using a system such as the system of FIG. 1C. System 150 can be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this document. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 150, singly or in combination, can be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 150 are distributed across multiple ICs and / or discrete components. In various embodiments, the system 150 is communicatively coupled to one or more other systems, or other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various embodiments, the system 1000 is configured to implement one or more of the aspects described in this document.

[0062] The system 150 includes at least one processor 152 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this document. Processor 152 may include embedded memory, input output interface, and various other circuitries as known in the art. The system 150 includes at least one memory 154 (e.g., a volatile memory device, and / or a non-volatile memory device). System 150 may include a storage device 158, which can include non-volatile memory and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read- Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drive, and / or optical disk drive. The storage device 158 can include an internal storage device, an attached storage device (including detachable and non-detachable storage devices), and / or a network accessible storage device, as non-limiting examples.

[0063] System 150 includes an encoder / decoder module 156 configured, for example, to process data to provide an encoded video or decoded video, and the encoder / decoder module 156 can include its own processor and memory. The encoder / decoder module 156 represents module(s) that can be included in a device to perform the encoding and / or decoding functions. As is known, a device can include one or both of the encoding and decoding modules. Additionally, encoder / decoder module 156 can be implemented as a separate element ofsystem 150 or can be incorporated within processor 152 as a combination of hardware and software as known to those skilled in the art.

[0064] Program code to be loaded onto processor 152 or encoder / decoder 156 to perform the various aspects described in this document can be stored in storage device 158 and subsequently loaded onto memory 154 for execution by processor 152. In accordance with various embodiments, one or more of processor 152, memory 154, storage device 158, and encoder / decoder module 156 can store one or more of various items during the performance of the processes described in this document. Such stored items can include, but are not limited to, the input video, the decoded video or portions of the decoded video, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0065] In some embodiments, memory inside of the processor 152 and / or the encoder / decoder module 156 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other embodiments, however, a memory external to the processing device (for example, the processing device can be either the processor 152 or the encoder / decoder module 152) is used for one or more of these functions. The external memory can be the memory 154 and / or the storage device 158, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of, for example, a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for video coding and decoding operations, such as for MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also referred to as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or WC (Versatile Video Coding, a new standard being developed by JVET, the Joint Video Experts Team).

[0066] The input to the elements of system 150 can be provided through various input devices as indicated in block 172. Such input devices include, but are not limited to, (i) a radio frequency (RF) portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Component (COMP) input terminal (or a set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples include composite video.

[0067] In various embodiments, the input devices of block 172 have associated respective input processing elements as known in the art. For example, the RF portion can be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting asignal, or band-limiting a signal to a band of frequencies), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select (for example) a signal frequency band which can be referred to as a channel in certain embodiments, (iv) demodulating the downconverted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner that performs various of these functions, including, for example, downconverting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, downconverting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements can include inserting elements in between existing elements, such as, for example, inserting amplifiers and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna.

[0068] Additionally, the USB and / or HDMI terminals can include respective interface processors for connecting system 150 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed-Solomon error correction, can be implemented, for example, within a separate input processing IC or within processor 152 as necessary. Similarly, aspects of USB or HDMI interface processing can be implemented within separate interface ICs or within processor 152 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 152, and encoder / decoder 156 operating in combination with the memory and storage elements to process the datastream as necessary for presentation on an output device.

[0069] Various elements of system 150 can be provided within an integrated housing, Within the integrated housing, the various elements can be interconnected and transmit data therebetween using suitable connection arrangement 174, for example, an internal bus as known in the art, including the I nter-IC (I2C) bus, wiring, and printed circuit boards.

[0070] The system 150 includes communication interface 160 that enables communication with other devices via communication channel 162. The communication interface 160 can include, but is not limited to, a transceiver configured to transmit and to receive data overcommunication channel 162. The communication interface 160 can include, but is not limited to, a modem or network card and the communication channel 162 can be implemented, for example, within a wired and / or a wireless medium.

[0071] Data is streamed, or otherwise provided, to the system 150, in various embodiments, using a wireless network such as a Wi-Fi network, for example IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal of these embodiments is received over the communications channel 162 and the communications interface 160 which are adapted for Wi-Fi communications. The communications channel 162 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet for allowing streaming applications and other over- the-top communications. Other embodiments provide streamed data to the system 150 using a set-top box that delivers the data over the HDMI connection of the input block 172. Still other embodiments provide streamed data to the system 150 using the RF connection of the input block 172. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, for example a cellular network or a Bluetooth network.

[0072] The system 150 can provide an output signal to various output devices, including a display 176, speakers 178, and other peripheral devices 180. The display 176 of various embodiments includes one or more of, for example, a touchscreen display, an organic lightemitting diode (OLED) display, a curved display, and / or a foldable display. The display 176 can be for a television, a tablet, a laptop, a cell phone (mobile phone), or other device. The display 176 can also be integrated with other components (for example, as in a smart phone), or separate (for example, an external monitor for a laptop). The other peripheral devices 180 include, in various examples of embodiments, one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 180 that provide a function based on the output of the system 150. For example, a disk player performs the function of playing the output of the system 150.

[0073] In various embodiments, control signals are communicated between the system 150 and the display 176, speakers 178, or other peripheral devices 180 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communications protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to system 1000 via dedicated connections through respective interfaces 164, 166, and 168. Alternatively, the output devices can be connected to system 150 using the communications channel 162 via the communications interface 160. The display176 and speakers 178 can be integrated in a single unit with the other components of system 150 in an electronic device such as, for example, a television. In various embodiments, the display interface 164 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0074] The display 176 and speaker 178 can alternatively be separate from one or more of the other components, for example, if the RF portion of input 172 is part of a separate set-top box. In various embodiments in which the display 176 and speakers 178 are external components, the output signal can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0075] The system 150 may include one or more sensor devices 168. Examples of sensor devices that may be used include one or more GPS sensors, gyroscopic sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors may be used to determine information such as user’s position and orientation. Where the system 150 is used as the control module for an extended reality display (such as control modules 124, 132), the user’s position and orientation may be used in determining how to render image data such that the user perceives the correct portion of a virtual object or virtual scene from the correct point of view. In the case of head-mounted display devices, the position and orientation of the device itself may be used to determine the position and orientation of the user for the purpose of rendering virtual content. In the case of other display devices, such as a phone, a tablet, a computer monitor, or a television, other inputs may be used to determine the position and orientation of the user for the purpose of rendering content. For example, a user may select and / or adjust a desired viewpoint and / or viewing direction with the use of a touch screen, keypad or keyboard, trackball, joystick, or other input. Where the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation used for the purpose of rendering content may be selected and / or adjusted based on motion of the display device.

[0076] The embodiments can be carried out by computer software implemented by the processor 152 or by hardware, or by a combination of hardware and software. As a nonlimiting example, the embodiments can be implemented by one or more integrated circuits. The memory 154 can be of any type appropriate to the technical environment and can be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 152 can be of any type appropriate to the technical environment, and can encompass one or more ofmicroprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples.Overview of split inferencing of a trained model

[0077] FIG. 2A and FIG. 2B show the basic architectures (3GPP SA4 AI4media) for split inferences of a model composed of n layers (1 ..n) between the network and a user equipment (UE), where a first inference processes first part of the model, namely layers 1..k, and the second inference processes the second part of the model, layers k+1..n. The architecture shows the delivery and access functions of intermediate data between the UE and the network for two different scenarios.

[0078] In a first scenario, when the media data source originates from the UE, the first part of the inference is performed in the UE, the second part in the network. The resulting output data is finally sent back to the UE.

[0079] In a second scenario, when the media data source comes from the network, or from the network via the UE, the first part of the Al model is performed on the network side and the second part on the UE.

[0080] FIG. 2A illustrates split inference between the UE and network, with media data source in the UE.

[0081] FIG. 2B illustrates split inference between the UE and network, with media data source in the network.Overview of trained model, graph and directed acyclic graph

[0082] This section describes general examples of a trained model, a graph, and a directed acyclic graph. A graph may be described as a data structure that defines the flow of data and computations. An AI / ML (artificial intelligence I machine learning) model may be described as a set of nodes and edges (links between nodes) that perform a given task, for instance classification or regression. In that sense, a model is a specific implementation of a graph that has been trained on a set of data.

[0083] A graph may be used to describe the structure of the computations that will be performed, and a model is a specific instantiation of that structure that has been trained to perform a specific task. The directed acyclic graph is a type of a graph.

[0084] Thus, an Al trained model such as DNN / CNN (deep neural network I convolutional neural network) and other models can be seen as a collection of layers / nodes interconnected with edge that can be represented in a directed acyclic graph. A directed acyclic draph (DAG) is a directed graph, with no directed cycles (or loops) where the edge (or links between the nodes) are directed from one node to another in such a way that there is no closed loop. Thetopology of the DAG is linearly ordered. These specific features (no loop and nodes ordering) are represented in FIG. 3B.

[0085] FIGs. 3A and 3B are two representations of the same model. A list of nodes or layers is provided in FIG. 3A. A directed acyclic graph is provided in FIG. 3B.Overview of the ONNX format

[0086] One format that may be used to represent machine learning models is ONNX (open neural network exchange). ONNX uses a common set of operators and a common file format to enable Al developers to use models with a variety of frameworks, tools, runtimes, and compilers. The operators are the building blocks of machine learning and deep learning models.

[0087] ONNX describes a computational graph. Each machine learning operation is represented by a “node”. Each node has following properties: node. name : a string that identifies this node node. input: a list of string that represent tensor name expected by the node node. output: a list of string that represent tensor name generated by the node node.op_type: a string that identifies the operation of the node

[0088] All nodes are available through a list (e.g. onnx_model. graph. node). The model itself is organized in a graph, in a tree-like way. FIGs. 4A-4C provide some examples of different model topologies. FIG. 4A illustrates a model with one branch. FIG. 4B illustrates a model with one or two branches. FIG. 4C illustrates a model with multiple interconnected branches.Issues addressed in some embodiments.

[0089] When an Al model is split in two parts for dynamic split point reselection, the first endpoint splits a model, executes the first part, and sends an indication to the second endpoint on the split point to start. The first endpoint also sends the output intermediate data to the second endpoint. The second endpoint computes the received split point indication to obtain the second part of the model and executes this second part by using the received intermediate data to generate the final output.

[0090] One issue is providing the specific split operation for the appropriate Al model. A given trained model for a particular framework (PyTorch, tensor flow) can be split in several pieces from modifying the software producing the Al model or parsing the trained model using another software. The split function is therefore specific to the model, and it can be difficult to generalize the split to another model.

[0091] Another issue is the use of static model parts to enable a dynamic split. One way to provide a dynamic split point reselection is to prepare the set of static trained model parts, e.g. part1_splitpoint10, part2_Splitpoint10 , part1_splitpoint27, part2_Splitpoint27. This technique calls for preparing and storing a lot of split models parts for each endpoint and for all the different models to be used, which is not optimal.

[0092] Another issue is finding an unambiguous split. A split based on a single indication as a split point number or split point name calls for having on both sides a deterministic algorithm able to split a model without ambiguity. It also calls for having the same understanding on both sides of the meaning of the split point, in which part the split point itself is included.

[0093] A further issue is providing unambiguous transmission of intermediate data. In case of a split occurring in a multiple branches part of the model, thus resulting in multiple intermediate data, the receiver side must be able to uniquely map the intermediate data received (model Part I outputs) with the different expected (model Part II) inputs.

[0094] Some example embodiments may provide one of more of the following benefits:• Providing the ability to split a model at any node regardless of the number or complexity of the links between the nodes.• Enabling a strategy of cutting any model at any node and measuring the effects in term of sub-model size, latency, energy cost, intermediate data size, transmission cost helping to determine the most effective split point.• Providing solutions to communicate several tensors at each inference phase.• Providing solutions to avoid sending of tensor names at each inference phase.Overview of example embodiments

[0095] Example embodiments provide and share a parsing algorithm and rules for enabling a split operation on a model that can be represented as directed acyclic graphs. Some embodiments include a setup process, a split process, and an inference process.

[0096] An example setup process describes the set of information to be used to process an Al model graph on each endpoint. Such information may include, for example, the graph information, model parsing algorithm, exchange split point metadata format, and exchange input / output intermediate data format.

[0097] An example split process applies the split of the model on each endpoint according to the rules defined in the setup process.

[0098] An example inference process describes actions performed on each endpoint including transmitting split information from one endpoint to second endpoint according to intermediate data format, metadata format.

[0099] Example embodiments provide and share information regarding a parsing algorithm and rules for enabling a split operation on an Al model that can be represented as one or more directed acyclic graphs.

[0100] FIG. 5 is a schematic illustration of two devices implementing a split operation on an Al model using dynamic graph based Al model splitting according to an embodiment described herein.

[0101] In some embodiments, in a setup process, configuration is performed on both endpoints. The configuration process may include the determination and exchange of some or all of the following information: model graph information, model parsing algorithm information, split point metadata format information, and input / output intermediate data format information.

[0102] The model graph information may include a model graph identifier to uniquely identify a model graph. The identifier may be chosen by the application configuration. The model graph information may further include a model graph framework, which may provide an identifier that uniquely identifies the AI / ML framework in which the model was developed (onnx, keras, Pytorch).

[0103] The model parsing algorithm information may include some or all of the following information: a unique identifier of a parsing algorithm, and a parsing algorithm and rules for each node and connected node edges.

[0104] The unique parsing algorithm identifier may be an identifier that uniquely identify a model parsing algorithm.

[0105] The parsing algorithm and rules for each node and connected node edges, may include node numbering rules and / or edge node identification information. The node numbering rules may include numbering the node with the rank of the node in the list, or starting numbering from parsing the nodes from a top / down and left to right directions of the graph’s node, as examples, The edge node identification may be, for example, a string ‘node. source’, ‘node. destination.’

[0106] The split point metadata format information may include information used by the second endpoint to split the second part of the trained model regarding the first one. Split point information may be based on one or more of the following: name, number, inclusion / exclusion flag (in which part is included the split point).

[0107] The input / output intermediate data format information may provide the structure and identification of the output tensors regarding node or edge information. Such information may include some or all of the following: format type identifier, output tensor parsing rules, tensor identification, and / or tensor structure format.

[0108] The format type identifier may be an identifier that uniquely identifies the input / output intermediate data format (e.g. numpy_vx, torch_tensor, tensorflow_vx.y).

[0109] The output tensor parsing rules may define which rule is used for parsing tensors from received intermediate data. In some embodiments, the rule may be one of the following rules, which are described in further detail below.• Rule 1 : Tensor names are sent at the setup phase, and the tensor value is sent at the inference phase.• Rule 2: Both tensor names and tensors are sent at the inference phase.• Rule 3: Only tensors are sent at the inference phase; the association tensor name and tensor is done with the dimension of the tensor.• Rule 4: Only tensors are sent at the inference phase; the association tensor name and tensor is done based on the alphabetical order of the tensor name.

[0110] The tensor identification information may define the names of tensors, such as through an ordered list of tensor names.

[0111] The tensor structure format information may be used to define the tensor format of the intermediate data transmitted in inference stage. The following are examples:• List [tensor_name, Tensor]• List [Tensor, tensor_name,]• List[tensor]• List[tensor dimension, tensor]

[0112] An example method is illustrated in the call flow diagram of FIG . 6, with details provided for the setup process. At 601 , each endpoint parses the model information required by the application: model’s name, Al framework. At 602, each endpoint parses the model and proceeds to node numbering, edge node identification. At 603, endpoint 1 selects a split point and communicates split point metadata, which may include some or all of the following: the node’s name, node’s number and a flag indicating if the split occurs before or after the split point, i.e. if the split point is included in the part I or in the part II.

[0113] At its reception the endpoint 2 checks if whether is able to find this split point in its model and stores the convention on the split point inclusion / exclusion. At 604, the split results in pending data (tensors) at the output of the part I. This step defines the type of the intermediate data exchanged (e.g. NumPy array, string array, serialized data), the algorithm used for the split, the tensor’s names and dimensions and the output tensor parsing rule. This information is then communicated to the endpoint 2. At its reception, the endpoint 2 checks its compatibility with the requested format.

[0114] An example of node numbering is shown in FIGs. 7A-7B. The FIGs. 7A-7B illustrate the following node numbering rule: a number is associated to each node corresponding to its rank in the list. Other rules may alternatively be used. One potential benefit of this numbering scheme is the ability to be able to cut anywhere in the model without any knowledge of the node’s name or any knowledge of the links between nodes. This solution enables a strategy of trying to cut any model at various different nodes and measuring the effects in term of submodel size, latency, energy cost, intermediate data size, transmission cost.

[0115] Details of example intermediate data parsing rules that may be used in some embodiments are provided as follows. Intermediate data parsing rules 1 and 2 are not described in greater detail because they are straightforward, as the tensors and names of the tensors are explicitly given. Rule 3 and rule 4 are explained further. These rules provide examples of techniques that may be used to discriminate tensors when they are received on endpoint 2 but without any additional information. For the sake of illustration, consider tensors with the following names and dimensions of intermediate data:• Name: 7additional_blocks.2 / additional_blocks.2.0 / Conv_output_0', Dimension: float32[1 , 128, 10, 10]Dimension: float32 [1 ,1024,38,38]• Name: 7additional_blocks.0 / additional_blocks.0.5 / Relu_output_0', Dimension: float32 [1 ,512,19,19]• Name: 7additional_blocks.1 / additional_blocks.1 ,5 / Relu_output_0', Dimension: float32 [1 ,512,10,10]

[0116] In the example of Rule 3, an association between the tensor name and the tensor is done with the dimension of the tensor. At the inference stage tensors may be sent following any order. For instance, the order above. At the reception, the receiver receives tensors having different dimensions: [1 ,128,10,10], [1 ,1024,38,38], [1 ,512,19,19], [1 ,512,10,10], Having the knowledge of the dimensions of its expected inputs, the receiver is able to associate data received and input tensors. This technique is applicable to tensors having different dimensions.

[0117] In the example of Rule 4, an association between tensor name and tensor is done based on the alphabetical order of the tensor name. In this example, the alphabetical order of the tensors is:• 7additional_blocks.0 / additional_blocks.0.5 / Relu_output_0'• 7additional_blocks.1 / additional_blocks.1 ,5 / Relu_output_0'• 7additional_blocks.2 / additional_blocks.2.0 / Conv_output_0'• 7feature_extractor / feature_extractor / feature_extractor.6 / feature_extractor.6.5 / relu_2 / Relu_output_0

[0118] At the inference stage tensors are sent following this order. At the reception, the receiver has the knowledge to associate data received and input tensors.

[0119] In some embodiments, details of a split stage may be described as follows.

[0120] In a split stage, a sender may perform one or more of the following actions: selecting a split point, applying model parsing rules, and sending the split point metadata.

[0121] The selection of the split point may be a selection from among a list of the possible split points of the model.

[0122] The sender applies the model parsing rules algorithm to split a model from a selected split point to obtain the first part of the model and output tensor names of the first part.

[0123] The sender sends split point metadata information according to split point metadata Format defined in the setup phase. The split point metadata may include split point information identifying a split point, such as a node number (e.g. k=75, where k is the node’s rank in the node’s list), and / or a node name (e.g. “conv2”). The split point metadata may further include an inclusion / exclusion flag indicating whether the split occurs before or after the identified point.

[0124] In the split point stage, a receiver may obtain the split point information from split point metadata, e.g k=75. The receiver may apply the model parsing rules algorithm to split a model from a selected split point to obtain the second part of the model and the input tensor names of the second part.

[0125] In the inference stage a sender executes the inference of the first part of the model and delivers intermediate data (possibly with appropriate metadata) to the receiver. The intermediate date is output according to an output intermediate data format rule defined in the setup phase.

[0126] In an example inference stage, a receiver may perform one or more of the following actions. The receiver receives intermediate data and / or tensor name information according to the setup phase rule. The receiver receives a tensor payload according to tensor structureformat. For example, the payload may be in the form [[tensorl , tensor2]] or, with the use of tensor names, in the form [[‘Conv_output_0’, tensorl], [‘Relu_output_0’, tensor2]]. The receiver assigns a dictionary tensor list to the input names according to the setup phase rule, as described above. For example, {‘Conv_output_0’: [tensorl], ‘Relu_output_0’: [tensor2]}. The receiver feeds the intermediate data to the inference stage and executes the inference of the second part of the model.

[0127] Examples of graph parsing algorithms that may be used in some embodiments are described in the following sections.Graph parsing of a linear graph

[0128] For some Al models (e.g. VGG16), the structure is made of only one branch. FIG. 4A schematically illustrates one such model. Each node has only one input and one output. The output of node “i” is the input of node “i+1”.

[0129] In this case, example embodiments allow the model to be split anywhere on the node’s list and to create (at least) two split models, as shown in the flow diagram of FIG. 8.

[0130] For example, use “n” to represent the number of nodes, and use “k” as the index used to identify the split. The model may be split at index “k,” thus keeping the first “k” nodes for the part I, and nodes from “k” to “n” for the part II. As a result, the node list {0..n} of the model is split into Model Part I with nodes list {0..k-1} and Model Part II with node list {k..n}.

[0131] In an example model where part I is run on a first device and model part II on a second device, both the index “k” and the “intermediate data” representing intermediate output of the model Part I may be transmitted to the second device.

[0132] The following is a description of an algorithm that may be implemented with linear model using the Keras interface for the Tensor Flow library.Python code: def cut_model (model , cut_layer_name ) : clone = clone_model (model ) clone . set_weights (model . get_weights ( ) ) cut_layer = clone . get_layer ( name=cut_layer_name ) model_a = Model ( clone . inputs , cut_layer . output ) model_b = Model ( Input ( tensor=cut_layer . output ) , clone . outputs ) return model_a , model_bCall:"split_points"= [ "pool l_pool" , "conv2_blockl_add" , "conv3_blockl_add" , "con v4_blockl_add" , "conv5_blockl_add" , "conv5_block2_add" , "conv5_block3_add" , "avg_pool" , "predictions" ] for split_layer in split_points : model_a , model_b = cut_model (base_model , split_layer )# do the inference with model_a on first device# transmit intermediate data as a result of inference model_a with the input to the second device# do the inference with model_b with previous intermediate data on the second device# transmit the result back to first deviceGraph parsing of a complex graph with several branches

[0133] Some embodiments may be used with Al models that are complex and can contain a lot of branches, such as yolov4, ssd_resnet, and retinanet, among others. In such embodiments it is desirable to identify all the inputs required by the model Part II. The input of a node can be the output of another one, or it can be constant data contained in the model file (e.g. weights and bias for a convolution node; these data are called “initializer”). In addition, the output of a node may serve as the input of several nodes.

[0134] FIGs. 9A and 9B illustrate examples of node numbering. In the example of FIGs. 9A- 9B, the output of the node “MaxPool” (node 2) is used by the first node “Conv” (node 3) and “Add” (node 6).

[0135] In a case where it is desirable to be able to split a graph at any position, the pending data resulting from this split should be considered.

[0136] In the example of FIGs. 9A-9B, if the model is cut just after the first “Conv” function (node 3, 7backbone / backnone.4 / Conv1 / Conv”) pending data will be:• the output of MaxPool node (“ / backbone / backnone.3 / MaxPool_output_0”)• and the output of the Conv node (“ / backbone / backnone.4 / Conv1 / Conv_output_0”)

[0137] Various different algorithms may be used to identify the pending data. Specific embodiments are described below as Example 1 and Example 2.Example 1

[0138] In an example, an Al computational directed acyclic graph, such as the example shown in FIG. 10, is composed of a set of nodes and edges.

[0139] Each node represents an individual AI / ML operation. Each node may be described using some or all of the following information. node. id (‘n’lnteger): where the numbering starts from parsing the nodes from a top / down and left to right directions e.g.; n3 node 3) node. name: a string that identifies this node node.op_type: a string that identifies the Al operation of the nodenode. input: a list of string that represent tensor name expected by the node node. output: a list of string that represent tensor name generated by the node

[0140] Each edge represents a tensor that is output from a source node and input to a destination node. Each tensor may be described with some or all of the following information: edge. id (‘n”source. node’. ’destination. node. id’): The edge number representing the relation between a source node and a destination node, (e.g., edge e1.2 means edge from node 1 to node 2) edge.tensor: a string that represent tensor name, either output by the source node and input destination node.

[0141] The first process in this example is the identification of nodes and edges. This may be done using a node numbering process that produces a unique number for each node of the graph. An example numbering rule is starting numbering from parsing the nodes from a top / down and left to right direction. The node identification process may produce a unique node name, such as ‘conv_3.1 ’. An edge identification may be performed to produce a name that binds and carries a relationship between two existing node connections. For example, e1 .5 may be used to identify a relation between node 1 and node 5 that bypasses nodes 2,3, and 4. The unique number for each node is associated with and bound to the unique node name, “e.g Conv_3,” and the Al operation is associated to the node number.

[0142] The second process in this example is the splitting operation performed based on the graph as numbered in the first process above. For a split performed at node n3, the set of edges belonging to path part 1 and part 2 can be enumerated in two lists as follows.• A first list is a list of immediate n3 predecessor edges. In the example of FIG. 10, that includes only edge e1 .3.• A second list is a list of edges where, in the example of FIG. 10, a node ni is connected to a node nj where i belongs to [1 ..2] and j [4..6], In the example of FIG. 10, that includes edges e1.5 and e2.4.

[0143] Based on these lists,• parti comprises the subset of the graph before n3 (n1 and n2) with the output tensors of the identified edges above (edge 1.3, edges e1.5 and e2.4).• part2 comprise the subset of the graph including n3 and the rest of the nodes (n3, n4, n5, n6) with Inputs tensors of the identified edges above (edge 1.3, edges e1.5 and e2.4).

[0144] The method of Example 1 may be generalized as follows. The first process involves parsing and numbering nodes and edges between nodes as follows:• nodes numbering: parsing the graph and numbering nodes from a top / down and left to right directions.• edge numbering: parsing the graph and numbering the node to carry the relationship between the two nodes e.g., e 1.5 represent a relationship between node 1 and node 5.

[0145] For a split performed at node k the set of edges belonging to path parti and part 2 can be enumerated as follows:• List of the immediate predecessor edges reaching node k. This comprises at most the set of edges eO.k .. k-1 .k• List of edges where a node nk 1is directly connected to a node nk+i.

[0146] Based on these lists:• parti comprises the subset of the graph before node k and the output tensors of the identified edges above e.g. 'Conv_output_0'• part2 comprises the subset of the graph including node k and Inputs tensors of the identified edges above. 'Conv_input_0'Example 2

[0147] A second example of an algorithm for splitting a model is described here with reference to the model of FIG. 11 A. The model may proceed as follows.• Select a model M• Select a split point k (e.g. n3)• Split model M at split point k into M1 (n1 ,n2) (FIG. 11 B) and M2 (n3,n4,n5,n6,n7) (FIG. 11C)• Build the list “all_outputs” of all outputs of all nodes of full model M (e;g. e1.2,e1.3,e1.5,e2.4,e3.4,e4.5,e4.6,e5.7,e6.7)• Build the list “all_inputs_needed” of all inputs of the model M2 that are coming from output of other nodes (i.e. that are also presents on the list “all_outputs” ; we do not keep inputs that are coming from the model itself as weights and bias) (e.g. e1 ,3,e1 ,5,e2.4,e3.4,e4.5,e4.6,e5.7,e6.7)• Remove from the list “all_inputs_needed” all inputs that are generated by nodes of model M2 (e.g. remove e3.4, e4.5, e4.6, e5.7, e6.7 => all_inputs_needed= {e1.3,e1.5,e2.4}

[0148] In a Python implementation, Example 2 may be expressed as follows:def FindlOAtSplit ( k_split , onnxjnodel ) : nodes=onnx_model . graph . node# we first build the list of all ouputs of all nodes all_outputs= [ ] for k in range ( 0 , len (nodes ) ) : for outp in nodes [ k] . output : if outp not in all_outputs : all_outputs . append ( outp )# we look for all inputs needed by each node after the split all_inputs_needed= [ ] for k in range ( k_split, len ( nodes ) ) : for inp in nodes [ k] . input : if inp not in all_inputs_needed :# we check i f this input is generated by a previous node or is constant data if inp in all_outputs : all_inputs_needed . append ( inp )# then remove all entries that are generated by nodes after the split for k in range ( k_split, len ( nodes ) ) : for inp in nodes [ k] . output : if inp in all_inputs_needed : all_inputs_needed . remove ( inp ) return all_inputs_needed

[0149] The output of the process is a list of nodes outputs that represents the list of all outputs of the first part of the model and / or the list of all inputs of the second part of the model.Intermediate metadata communicationOne branch split (split at bottleneck)

[0150] Example embodiments operate to transmit both the split point identifier (index number “k”, or node name) and the “intermediate data” representing intermediate output of the model. FIG. 12 illustrates an example embodiment.

[0151] In some embodiments, the split number (k) is communicated synchronously (inline). In other embodiments, the split number is communicated asynchronously. This may be indicated in the setup-phase.

[0152] In a case where the split number is communicated asynchronously, the sender may communicate metadata including the split point identifier (number or name) and the convention on the split point flag (before, after). The sender performs the first part of the inference,encodes intermediate data, and communicates the intermediate data to the receiver. The receiver receives metadata including the split point identifier and makes the split accordingly. The receiver receives the intermediate data and makes the second part of the inference.

[0153] In a case where the split number is communicated synchronously, the sender makes the first part of the inference and encodes intermediate data. The sender then communicates metadata including the split point identifier (number or name) and the convention on the split point flag (before, after) and intermediate data. The receiver receives metadata and split point identifier (number or name) and the convention on the split point flag (before, after) and intermediate data. The receiver then makes the second part of the inference.Multi branches split

[0154] As an example, the ssd_resnet model may be split just before the node numbered one hundred and six in that model’s numbering scheme. At this level of split, four branches are cut. The names of the pending data are:• 7additional_blocks.2 / additional_blocks.2.0 / Conv_output_0',• 7feature_extractor / feature_extractor / feature_extractor.6 / feature_extractor.6.5 / relu_2 / Relu_output_0',• 7additional_blocks.0 / additional_blocks.0.5 / Relu_output_0',• 7additional_blocks.1 / additional_blocks.1 ,5 / Relu_output_0'

[0155] FIG. 13A illustrates the output properties of the first part of the model. FIG. 13B illustrates the input properties of the second part of the model. At the end of the inference of the Part I, the result is four tensors. Thus, in this case there are four tensors to be communicated, and it is desirable for the receiver to be able to discern which tensor is associated to which input. Different embodiments use different techniques to provide this information to the receiver.

[0156] In some embodiments, both the tensor itself and the tensor_names are communicated during the inference phase. In such embodiments, the sender in the setup phase communicates metadata including the split point identifier (number or name) and the convention on the split point flag (before, after). The sender in the inference phase makes the first part of the inference, encodes intermediate data, and communicates metadata including the tensor names and the intermediate data of each tensor. As an example, the communicated information may be expressed as follows:[tensorl ,tensor2,tensor3,tensor4],7additional_blocks.2 / additional_blocks.2.0 / Conv_output_0',7feature_extractor / feature_extractor / feature_extractor.6 / feature_extractor.6.5 / relu_2 / Relu_output_0',7additional_blocks.0 / additional_blocks.0.5 / Relu_output_0', 7additional_blocks.1 / additional_blocks.1.5 / Relu_output_0'

[0157] The receiver in the setup phase receives metadata including the split point identifier and makes the split accordingly. The receiver in the inference phase receives the metadata including both the tensor names and the intermediate data of each tensor. Based on the metadata, the receiver reconstructs the dictionary {tensor_name: tensor_value}. For example, the dictionary may be reconstructed as follows.{7additional_blocks.2 / additional_blocks.2.0 / Conv_output_0': tensorl ,7feature_extractor / feature_extractor / feature_extractor.6 / feature_extractor.6.5 / relu_2 / Relu_output_0':tensor2, 7additional_blocks.0 / additional_blocks.0.5 / Relu_output_0':tensor3, 7additional_blocks.1 / additional_blocks.1.5 / Relu_output_0':tensor4}

[0158] The receiver then makes the second part of the inference.

[0159] In other embodiments, only the tensors themselves are communicated assuming that the order of tensor names is well known by the receiver side, either by a convention rule or explicitly during the setup phase. In one such embodiment, the sender during the setup phase communicates metadata including the split point identifier (number or name) and the convention on the split point flag (before, after), and the ordered list of tensor names, such as the following.7additional_blocks.2 / additional_blocks.2.0 / Conv_output_0',7feature_extractor / feature_extractor / feature_extractor.6 / feature_extractor.6.5 / relu_2 / Relu_output_0',7additional_blocks.0 / additional_blocks.0.5 / Relu_output_0', 7additional_blocks.1 / additional_blocks.1.5 / Relu_output_0'

[0160] During the inference phase, the sender makes the first part of the inference, encodes intermediate data and communicates intermediate data of each tensor, for example as follows.[[tensorl ,tensor2,tensor3,tensor4]]

[0161] During the setup phase, the receiver receives metadata including the split point identifier and ordered list of tensor names, makes the split accordingly, and takes note of the order list of tensor names. During the inference phase, the receiver receives the intermediatedata of each tensor and reconstructs the dictionary {tensor_name: tensor_value}. For example, the dictionary may be reconstructed as follows.{7additional_blocks.2 / additional_blocks.2.0 / Conv_output_0': tensorl , 7feature_extractor / feature_extractor / feature_extractor.6 / feature_extractor.6.5 / relu_2 / Relu_output_0':tensor2,7additional_blocks.0 / additional_blocks.0.5 / Relu_output_0':tensor3,7additional_blocks.1 / additional_blocks.1.5 / Relu_output_0':tensor4}

[0162] The receiver then makes the second part of the inference.

[0163] Examples of information that may be used in some embodiments is summarized in the following table.

[0164] FIGs. 14A-14C illustrate an example of a split Al model, specifically an SSD_resnet model. FIG. 14A illustrates the first part of the model. The first part of the model may be implemented on a first device. The first part of the model receives an input and provides two outputs Conv_output_0 and MaxPool_output_0. The first device may provide these outputs to a second device. The second device may implement the second part of the model, as shown in FIGs. 14B-14C. The second part of the model receives Conv_output_0 and MaxPool_output_0 as inputs and processes them accordingly.Further Embodiments.

[0165] Briefly stated, a method according to some embodiments comprises: obtaining information describing a graph structure of a machine learning, ML, model, wherein the ML model has a plurality of nodes; obtaining split point information indicating a split of the ML model into at least a first sub-model and a second sub-model, each of the nodes being in one of the sub-models, wherein the split point information identifies a node of the ML model as a split point node; at a first endpoint, obtaining at least a first sub-model output by executing the first sub-model on a model input; and sending the first sub-model output from the first endpoint to a second endpoint.

[0166] An apparatus according to some embodiments comprises one or more processors configured to perform at least: obtaining information describing a graph structure of a machine learning, ML, model, wherein the ML model has a plurality of nodes; obtaining split point information indicating a split of the ML model into at least a first sub-model and a second submodel, each of the nodes being in one of the sub-models, wherein the split point information identifies a node of the ML model as a split point node; at a first endpoint, obtaining at least a first sub-model output by executing the first sub-model on a model input; and sending the first sub-model output from the first endpoint to a second endpoint.

[0167] In some embodiments, the split point information identifies only one node.

[0168] In some embodiments, the first sub-model output is an output of a first node in the first sub-model, and the embodiment further comprises: obtaining a second sub-model output by the executing of the first sub-model on the model input, the second sub-model output being an output of a second node in the first sub-model; and sending the second sub-model output to the second endpoint.

[0169] In some embodiments, the split point information indicates whether the split point node is in the first sub-model or the second sub-model.

[0170] Some embodiments further comprise, in communication with the second endpoint, selecting a naming or numbering convention for outputs of the first sub-model, wherein sending the first sub-model output comprises sending information identifying each sub-model output according to the naming or numbering convention.

[0171] In some embodiments, each sub-model output is a tensor, and the naming or numbering convention comprises assigning a name to each tensor.

[0172] Some embodiments further comprise assigning numbers to a plurality of the nodes in the ML model, wherein the information identifying the split point node comprises a number assigned to the split point node.

[0173] In some embodiments, the split point information includes model graph parsing information.

[0174] In some embodiments, the model graph parsing information includes a model parsing algorithm identifier.

[0175] In some embodiments, the model graph parsing information includes node numbering rules.

[0176] In some embodiments, the graph parsing information includes node identification rules.

[0177] In some embodiments, the graph parsing information includes edge node identification information.

[0178] A method according to some embodiments comprises: obtaining information describing a graph structure of a machine learning, ML, model, wherein the ML model has a plurality of nodes, and wherein the ML model is configured to generate at least a model output based on a model input; obtaining split point information indicating a split of the ML model into at least a first sub-model and a second sub-model, each of the nodes being in one of the submodels, wherein the split point information identifies a node of the ML model as a split point node; obtaining at least a first sub-model output from a first endpoint; and at a second endpoint, obtaining the model output by executing the second sub-model on the first submodel output.

[0179] An apparatus according to some embodiments comprises one or more processors configured to perform at least: obtaining information describing a graph structure of a machine learning, ML, model, wherein the ML model has a plurality of nodes, and wherein the ML model is configured to generate at least a model output based on a model input; obtaining split point information indicating a split of the ML model into at least a first sub-model and a second sub-model, each of the nodes being in one of the sub-models, wherein the split point information identifies a node of the ML model as a split point node; obtaining at least a firstsub-model output from a first endpoint; and at a second endpoint, obtaining the model output by executing the second sub-model on the first sub-model output.

[0180] In some embodiments, the split point information identifies only one node.

[0181] Some embodiments further comprise: obtaining a second sub-model output from the first endpoint; wherein executing the second sub-model comprises using the first sub-model output as an input of a first node of the second sub-model and using the second sub-model output as an input of a second node of the second sub-model.

[0182] In some embodiments, the split point information indicates whether the split point node is in the first sub-model or the second sub-model.

[0183] Some embodiments further comprise, in communication with the first endpoint, selecting a naming or numbering convention for outputs of the first sub-model, wherein obtaining the first sub-model output comprises obtaining information identifying each submodel output according to the naming or numbering convention.

[0184] In some embodiments, each sub-model output is a tensor, and the naming or numbering convention comprises assigning a name to each tensor.

[0185] A method according to some embodiments comprises: obtaining a machine learning, ML, model; obtaining initial split information indicating an initial split of the ML model into at least an initial first part and an initial second part of the ML model; obtaining initial intermediate data representing an input to the initial second part of the ML model; and executing the initial second part of the ML model using the initial intermediate data.

[0186] Some embodiments further comprise obtaining updated split information indicating an updated split of the ML model into at least an updated first part and an updated second part of the ML model; obtaining updated intermediate data representing an input to the updated second part of the ML model; and executing the updated second part of the ML model using the updated intermediate data.

[0187] In some embodiments, the ML model includes a plurality of nodes and a plurality of edges, each edge connecting two nodes. In some embodiments, the initial split information includes information identifying a node in the ML model. In some embodiments, the initial split information includes information identifying only one node in the ML model.

[0188] In some embodiments, the initial split information includes information indicating whether the split occurs before or after the identified node.

[0189] In some embodiments, the initial intermediate data comprises at least one tensor. In some embodiments, the initial intermediate data comprises a plurality of tensors.

[0190] In some embodiments, the initial split information includes information indicating whether a split point is identified using a numbering convention or a naming convention.

[0191] In some embodiments, the initial split information and the intermediate data are received from a first endpoint, and the initial second part of the ML model is executed by a second endpoint different from the first endpoint.

[0192] A method according to some embodiments comprises: obtaining a machine learning, ML, model; providing initial split information indicating an initial split of the ML model into at least an initial first part and an initial second part of the ML model; obtaining initial input information representing an initial input to the initial first part of the ML model; executing the initial first part of the ML model using the input information; and providing initial intermediate data indicating an output of the initial first part of the ML model.

[0193] Some embodiments further comprise providing updated split information indicating an updated split of the ML model into at least an updated first part and an updated second part of the ML model; obtaining updated input information representing an updated input to the updated first part of the ML model; executing the updated first part of the ML model using the updated input information; and providing updated intermediate data indicating an output of the updated first part of the ML model.

[0194] In some embodiments, the ML model includes a plurality of nodes and a plurality of edges, each edge connecting two nodes.

[0195] In some embodiments, the initial split information indicating an initial split of the ML model includes information identifying a node in the ML model.

[0196] In some embodiments, the information indicating a split of the ML model includes information identifying only one node in the ML model.

[0197] In some embodiments, the initial split information includes information indicating whether the split occurs before or after the identified node.

[0198] In some embodiments, the initial intermediate data comprises at least one tensor.

[0199] In some embodiments, the initial intermediate data comprises a plurality of tensors.

[0200] In some embodiments, the initial split information includes information indicating whether a split point is identified using a numbering convention or a naming convention.

[0201] In some embodiments, the initial split point information and the intermediate data are provided to a second endpoint, and wherein the initial first part of the ML model is executed by a first endpoint different from the second endpoint.

[0202] An apparatus according to some embodiments comprises: a processor; and a non- transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any of the methods described herein.

[0203] An apparatus according to some embodiments comprises at least one processor configured to perform any of the methods described herein.

[0204] An apparatus according to some embodiments comprises a computer-readable medium storing instructions for causing one or more processors to perform any of the methods described herein.

[0205] An apparatus according to some embodiments comprising at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform any of the methods described herein.

[0206] This disclosure describes a variety of aspects, including tools, features, embodiments, models, approaches, etc. Many of these aspects are described with specificity and, at least to show the individual characteristics, are often described in a manner that may sound limiting. However, this is for purposes of clarity in description, and does not limit the disclosure or scope of those aspects. Indeed, all of the different aspects can be combined and interchanged to provide further aspects. Moreover, the aspects can be combined and interchanged with aspects described in earlier filings as well.

[0207] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are illustrated specifically, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of the implementations. At least one of the aspects generally relates to encoding and decoding, and at least one other aspect generally relates to transmitting a bitstream generated or encoded. These and other aspects can be implemented as a method, an apparatus, a computer readable storage medium having stored thereon instructions for encoding or decoding data according to any of the methods described, and / or a computer readable storage medium having stored thereon a bitstream generated according to any of the methods described.

[0208] Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as “first”, “second”, etc. may be used in various embodiments to modify an element, component, step, operation, etc., such as, for example, a “first decoding” and a “second decoding”. Use of such terms does notimply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or in an overlapping time period with the second decoding.

[0209] Various numeric values may be used in the present disclosure, for example. The specific values are for example purposes and the aspects described are not limited to these specific values.

[0210] Embodiments described herein may be carried out by computer software implemented by a processor or other hardware, or by a combination of hardware and software. As a nonlimiting example, the embodiments can be implemented by one or more integrated circuits. The processor can be of any type appropriate to the technical environment and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples.

[0211] Various implementations involve decoding. “Decoding”, as used in this disclosure, can encompass all or part of the processes performed, for example, on a received encoded sequence in order to produce a final output suitable for presentation. In various embodiments, such processes include one or more of the processes typically performed by a decoder, for example, entropy decoding, inverse quantization, inverse transformation, and differential decoding. In various embodiments, such processes also, or alternatively, include processes performed by a decoder of various implementations described in this disclosure.

[0212] Various implementations involve encoding. In an analogous way to the above discussion about “decoding”, “encoding” as used in this disclosure can encompass all or part of the processes performed, for example, on an input in order to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, for example, partitioning, differential encoding, transformation, quantization, and entropy encoding. In various embodiments, such processes also, or alternatively, include processes performed by an encoder of various implementations described in this disclosure.

[0213] When a figure is presented as a flow diagram, it should be understood that it also provides a block diagram of a corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow diagram of a corresponding method / process.

[0214] The implementations and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal.Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed can also be implemented in other forms (for example, an apparatus or program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, ora programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.

[0215] Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout this disclosure are not necessarily all referring to the same embodiment.

[0216] Additionally, this disclosure may refer to “determining” various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0217] Further, this disclosure may refer to “accessing” various pieces of information. Accessing the information can include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0218] Additionally, this disclosure may refer to “receiving” various pieces of information. Receiving is, as with “accessing”, intended to be a broad term. Receiving the information can include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0219] It is to be appreciated that the use of any of the following “ / ”, “and / or”, and “at least one of”, for example, in the cases of “A / B”, “A and / or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended for as many items as are listed.

[0220] Also, as used herein, the word “signal” refers to, among other things, indicating something to a corresponding decoder. For example, in certain embodiments the encoder signals a particular one of a plurality of parameters for region-based filter parameter selection for de-artifact filtering. In this way, in an embodiment the same parameter is used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as well as others, then signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual functions, a bit savings is realized in various embodiments. It is to be appreciated that signaling can be accomplished in a variety of ways. For example, one or more syntax elements, flags, and so forth are used to signal information to a corresponding decoder in various embodiments. While the preceding relates to the verb form of the word “signal”, the word “signal” can also be used herein as a noun.

[0221] Implementations can produce a variety of signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal can be formatted to carry the bitstream of a described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0222] We describe a number of embodiments. Features of these embodiments can be provided alone or in any combination, across various claim categories and types.

[0223] Note that various hardware elements of one or more of the described embodiments are referred to as “modules” that carry out (i.e., perform, execute, and the like) various functions that are described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable for a given implementation. Each described module may also include instructions executable for carrying out the one or more functions described as being carried out by the respective module, and it is noted that those instructions could take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or the like, and may be stored in any suitable non-transitory computer- readable medium or media, such as commonly referred to as RAM, ROM, etc.

[0224] Although features and elements are described above in particular combinations, each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

CLAIMS1. A method comprising: obtaining information describing a graph structure of a machine learning, ML, model, wherein the ML model has a plurality of nodes; obtaining split point information indicating a split of the ML model into at least a first sub-model and a second sub-model, each of the nodes being in one of the sub-models, wherein the split point information identifies a node of the ML model as a split point node; at a first endpoint, obtaining at least a first sub-model output by executing the first submodel on a model input; and sending the first sub-model output from the first endpoint to a second endpoint.

2. An apparatus comprising one or more processors configured to perform at least: obtaining information describing a graph structure of a machine learning, ML, model, wherein the ML model has a plurality of nodes; obtaining split point information indicating a split of the ML model into at least a first sub-model and a second sub-model, each of the nodes being in one of the sub-models, wherein the split point information identifies a node of the ML model as a split point node; at a first endpoint, obtaining at least a first sub-model output by executing the first submodel on a model input; and sending the first sub-model output from the first endpoint to a second endpoint.

3. The method of claim 1 or the apparatus of claim 2, wherein the split point information identifies only one node.

4. The method of claim 1 , or claim 3 as it depends from claim 1 , or the apparatus of claim 2, or claim 3 as it depends from claim 2, wherein the first sub-model output is an output of a first node in the first sub-model, further comprising: obtaining a second sub-model output by the executing of the first sub-model on the model input, the second sub-model output being an output of a second node in the first submodel; and sending the second sub-model output to the second endpoint.

5. The method of claim 1 , or claims 3-4 as they depend from claim 1 , or the apparatus of claim 2, or claims 3-4 as they depend from claim 2, wherein the split point information indicates whether the split point node is in the first sub-model or the second sub-model.

6. The method of claim 1 , or claims 3-5 as they depend from claim 1 , or the apparatus of claim 2, or claims 3-5 as they depend from claim 2, further comprising, in communication with the second endpoint, selecting a naming or numbering convention for outputs of the first submodel, wherein sending the first sub-model output comprises sending information identifying each sub-model output according to the naming or numbering convention.

7. The method of claim 6 as it depends from claim 1 , or the apparatus of claim 6 as it depends from claim 2, wherein each sub-model output is a tensor, and the naming or numbering convention comprises assigning a name to each tensor.

8. The method of claim 1 , or claims 3-7 as they depend from claim 1 , or the apparatus of claim 2, or claims 3-7 as they depend from claim 2, further comprising assigning numbers to a plurality of the nodes in the ML model, wherein the information identifying the split point node comprises a number assigned to the split point node.

9. The method of claim 1 , or claims 3-8 as they depend from claim 1 , or the apparatus of claim 2, or claims 3-8 as they depend from claim 2, wherein the split point information includes model graph parsing information.

10. The method of claim 9 as it depends from claim 1 , or the apparatus of claim 9 as it depends from claim 2, wherein the model graph parsing information includes a model parsing algorithm identifier.

11. The method of claim 9 as it depends from claim 1 , or the apparatus of claim 9 as it depends from claim 2, wherein the model graph parsing information includes node numbering rules.

12. The method of claim 9 as it depends from claim 1 , or the apparatus of claim 9 as it depends from claim 2, wherein the graph parsing information includes node identification rules.

13. The method of claim 9 as it depends from claim 1 , or the apparatus of claim 9 as it depends from claim 2, wherein the graph parsing information includes edge node identification information.

14. A method comprising: obtaining information describing a graph structure of a machine learning, ML, model, wherein the ML model has a plurality of nodes, and wherein the ML model is configured to generate at least a model output based on a model input;obtaining split point information indicating a split of the ML model into at least a first sub-model and a second sub-model, each of the nodes being in one of the sub-models, wherein the split point information identifies a node of the ML model as a split point node; obtaining at least a first sub-model output from a first endpoint; and at a second endpoint, obtaining the model output by executing the second sub-model on the first sub-model output.

15. An apparatus comprising one or more processors configured to perform at least: obtaining information describing a graph structure of a machine learning, ML, model, wherein the ML model has a plurality of nodes, and wherein the ML model is configured to generate at least a model output based on a model input; obtaining split point information indicating a split of the ML model into at least a first sub-model and a second sub-model, each of the nodes being in one of the sub-models, wherein the split point information identifies a node of the ML model as a split point node; obtaining at least a first sub-model output from a first endpoint; and at a second endpoint, obtaining the model output by executing the second sub-model on the first sub-model output.

16. The method of claim 14 or the apparatus of claim 15 wherein the split point information identifies only one node.

17. The method of claim 14, or claim 16 as it depends from claim 14, or the apparatus of claim15, or claim 16 as it depends from claim 15, further comprising: obtaining a second sub-model output from the first endpoint; wherein executing the second sub-model comprises using the first sub-model output as an input of a first node of the second sub-model and using the second sub-model output as an input of a second node of the second sub-model.

18. The method of claim 14, or claims 16-17 as they depend from claim 14, or the apparatus of claim 15, or claims 16-17 as they depend from claim 15, wherein the split point information indicates whether the split point node is in the first sub-model or the second sub-model.

19. The method of claim 14, or claims 16-18 as they depend from claim 14, or the apparatus of claim 15, or claims 16-17 as they depend from claim 15, further comprising, in communication with the first endpoint, selecting a naming or numbering convention for outputs of the first sub-model, wherein obtaining the first sub-model output comprises obtaininginformation identifying each sub-model output according to the naming or numbering convention.

20. The method of claim 19 as it depends from claim 14, or the apparatus of claim 19 as it depends from claim 15, wherein each sub-model output is a tensor, and the naming or numbering convention comprises assigning a name to each tensor.21 . The method of claim 14, or claims 16-20 as they depend from claim 14, or the apparatus of claim 15, or claims 16-20 as they depend from claim 15, wherein the split point information includes model graph parsing information.

22. The method of claim 21 as it depends from claim 14, or the apparatus of claim 21 as it depends from claim 15, wherein the model graph parsing information includes a model parsing algorithm identifier.

23. The method of claim 21 as it depends from claim 14, or the apparatus of claim 21 as it depends from claim 15, wherein the model graph parsing information includes node numbering rules.

24. The method of claim 21 as it depends from claim 14, or the apparatus of claim 21 as it depends from claim 15, wherein the graph parsing information includes node identification rules.

25. The method of claim 21 as it depends from claim 14, or the apparatus of claim 21 as it depends from claim 15, wherein the graph parsing information includes edge node identification information.

Images