Method for splitting a neural network model graph

Abstract

In an example method, a first endpoint obtains information describing a graph structure of a machine learning, ML, model, where the ML model has a plurality of nodes. The first endpoint obtains information indicating a split of the ML model into at least a first sub-model and a second sub-model, each of the nodes being in one of the sub-models. The first endpoint obtains at least a first sub-model output by performing the first sub-model on a model input. The first endpoint then sends the first sub-model output from the first endpoint to a second endpoint. In response to determining that the model input is an input to at least one of the nodes in the second sub-model, the first endpoint also sends the model input to the second endpoint.

Description

[0001] Cross-referencing This application claims priority to European Patent Application No. 23306913.7 entitled "Method to Split a Neural Network Model Graph" filed on 6 November 2023 and European Patent Application No. 24305115.8 entitled "Method to Split a Neural Network Model Graph" filed on 19 January 2024, both of which are incorporated herein by reference in their entirety. Background Technology

[0002] This disclosure relates to machine learning (ML) models. New uses for machine learning models are constantly being developed. However, as the availability of such models increases, their computational complexity continues to grow. It may be difficult to fully implement a complete machine learning model on a single computing device, especially when the computing device is a consumer-grade user device. Summary of the Invention

[0003] In short, one example method includes: obtaining information describing a graph structure of a machine learning (ML) model, wherein the ML model has multiple nodes; obtaining information indicating a split of the ML model into at least a first sub-model and a second sub-model, each of the nodes being in one of the sub-models; at a first endpoint, obtaining at least a first sub-model output by executing the first sub-model on a model input; sending the first sub-model output from the first endpoint to a second endpoint; and in response to determining that the model input is an input to at least one of the nodes in the second sub-model, sending the model input to the second endpoint. An example apparatus includes one or more processors configured to perform such a method.

[0004] Some embodiments further include obtaining at least a second sub-model output by performing the first sub-model on the model input, wherein the ML model has multiple model outputs, and wherein the second sub-model output is one of the model outputs.

[0005] A method according to some embodiments includes: obtaining information describing a graph structure of a machine learning (ML) model, wherein the ML model has a plurality of nodes, and wherein the ML model is configured to generate at least a model output based on a model input; obtaining information indicating a split of the ML model into at least a first sub-model and a second sub-model, each of the nodes being in one of the sub-models; obtaining at least a first sub-model output from a first endpoint; obtaining the model input from the first endpoint in response to determining that the model input is an input to at least one of the nodes in the second sub-model; and obtaining the model output at a second endpoint by executing the second sub-model on the model input and the first sub-model output. An example apparatus includes one or more processors configured to perform such a method.

[0006] Some embodiments further include obtaining at least a second sub-model output from the first endpoint, wherein the ML model has multiple model outputs, and wherein the second sub-model output is one of the model outputs.

[0007] In some embodiments, each of the nodes has a node input and a node output, and the method further includes determining a submodel output set by identifying node inputs in a second submodel that do not correspond to node outputs in a second submodel, wherein the first submodel output is in the submodel output set.

[0008] In some embodiments, sending the first sub-model output from the first endpoint to the second endpoint includes sending the determined set of sub-model outputs to the second endpoint.

[0009] In some embodiments, the ML model has an acyclic directed graph structure.

[0010] In some embodiments, the output of the first sub-model is a tensor.

[0011] In some embodiments, the information indicating a split includes information identifying the split point node.

[0012] An apparatus according to some embodiments includes: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods described herein.

[0013] An apparatus according to some embodiments includes at least one processor configured to perform any of the methods described herein.

[0014] Some embodiments include a computer-readable medium storing instructions for causing one or more processors to perform any of the methods described herein. Attached Figure Description

[0015] The following detailed description will be better understood when read in conjunction with the accompanying drawings, which illustrate examples of one or more of the various embodiments of this disclosure. However, it should be understood that the embodiments described herein are not limited to the precise arrangements and means shown in the drawings.

[0016] Figure 1A This is a system diagram illustrating an example communication system according to some embodiments.

[0017] Figure 1B The illustration is based on some embodiments and can be used Figure 1A The diagram shows a system diagram of an example wireless transmit / receive unit (WTRU) used in a communication system.

[0018] Figure 1C This is a system diagram illustrating a set of example interfaces of a system according to some embodiments.

[0019] Figure 2A The diagram illustrates the split reasoning between the UE and the network, where the media data source is in the UE.

[0020] Figure 2B The diagram illustrates the split reasoning between the UE and the network, where the media data source is in the network.

[0021] Figure 3A and 3B These are two representations of the same model. Figure 3A The list of nodes or layers is provided. Figure 3B It provides directed acyclic graphs.

[0022] Figures 4A-4C Examples of different model topologies are provided. Figure 4A The diagram shows a model with one branch. Figure 4B The illustration shows a model with one or two branches. Figure 4C The diagram shows a model with multiple interconnected branches.

[0023] Figure 5 This is a schematic diagram of two devices implementing a splitting operation on an AI model using a dynamic graph-based AI model splitting according to the embodiments described herein.

[0024] Figure 6 This is a flowchart illustrating the method call process executed in some embodiments.

[0025] Figures 7A-7B The illustration shows an example of node numbering that may be used in some embodiments.

[0026] Figure 8 This is a flowchart illustrating a method performed in some embodiments.

[0027] Figure 9A and 9B An example of node numbering is shown in the diagram.

[0028] Figure 10 The diagram illustrates an ML model that is an acyclic directed graph divided into a first part and a second part.

[0029] Figure 11A The illustration shows an example ML model.

[0030] Figure 11B Illustration Figure 11A The first part of the model.

[0031] Figure 11C Illustration Figure 11A The second part of the model.

[0032] Figure 12 This is a functional block diagram illustrating the operations performed in the example embodiment.

[0033] Figure 13A The diagram illustrates the output properties of the first model section.

[0034] Figure 13B The illustration shows the input attributes of the second model section.

[0035] Figure 14A and Figure 14B The diagram illustrates an example of splitting a machine learning model, where... Figure 14A This represents the first sub-model, and Figure 14B-14C This represents the second sub-model.

[0036] Figures 15A-15C This illustration shows another example of splitting a machine learning model, where Figure 15A Representing the complete model, Figure 15B This represents the first sub-model, and Figure 15C This represents the second sub-model.

[0037] Figures 16A-16B The illustration shows the attributes of the sub-models resulting from splitting the RetinaNet model at node 5. Figure 16A The diagram shows the first part (M1) of the split model. Figure 16B The diagram shows the second part (M2) of the split model.

[0038] Figures 17A-17C This illustration shows another example of splitting a machine learning model, where Figure 17A Representing the complete model, Figure 17B This represents the first sub-model, and Figure 17C This represents the second sub-model.

[0039] Figure 18AThe illustration shows the properties of the first sub-model obtained by splitting the RetinaNet model at node number 2248, and... Figure 18B The illustration shows the properties of the second sub-model obtained by splitting the RetinaNet model at node number 2248. Detailed Implementation

[0040] Overview of the example communication system Figure 1A This is a schematic diagram illustrating an example communication system 100 in which one or more of the disclosed embodiments may be implemented. The communication system 100 may be a multi-access system that provides content such as voice, data, video, messages, and broadcasts to multiple wireless users. The communication system 100 enables multiple wireless users to access such content by sharing system resources, including wireless bandwidth. For example, the communication system 100 may employ one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero-Tail Unique Word DFT Spread Spectrum OFDM (ZT UW DTS-s OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtered OFDM, Filter Bank Multicarrier (FBMC), etc.

[0041] like Figure 1AAs shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104, CN 106, Public Switched Telephone Network (PSTN) 108, Internet 110, and other networks 112. However, it should be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, and 102d may be any type of device configured to operate and / or communicate in a wireless environment. For example, WTRUs 102a, 102b, 102c, and 102d (any of which may be referred to as a “station” and / or “STA”) may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain environments), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, etc. Any of WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0042] The communication system 100 may also include base station 114a and / or base station 114b. Each of base stations 114a and 114b may be any type of device configured to wirelessly interface with at least one of WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks, such as CN 106, the Internet 110, and / or other networks 112. For example, base stations 114a and 114b may be base transceiver stations (BTS), node B, eNode B, home node B, home eNode B, gNB, NR node B, site controller, access point (AP), wireless router, etc. Although base stations 114a and 114b are each depicted as a single element, it should be understood that base stations 114a and 114b may include any number of interconnected base stations and / or network elements.

[0043] Base station 114a may be part of RAN 104, which may also include other base stations and / or network elements (not shown), such as base station controllers (BSCs), radio network controllers (RNCs), relay nodes, etc. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage of a specific geographic area, which may be relatively fixed or may change over time. The cell may be further divided into cell sectors. For example, the cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver for each sector of the cell. In one embodiment, base station 114a may employ multiple-input multiple-output (MIMO) technology and may use multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0044] Base stations 114a and 114b can communicate with one or more of WTRUs 102a, 102b, 102c, and 102d via air interface 116. Air interface 116 can be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). Any suitable radio access technology (RAT) can be used to establish air interface 116.

[0045] More specifically, as described above, the communication system 100 can be a multi-access system and can employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, base stations 114a and WTRUs 102a, 102b, and 102c in RAN 104 can implement radio technologies such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which can establish an air interface 116 using Wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or evolved HSPA (HSPA+). HSPA may include High-Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0046] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c may implement radio technologies such as evolved UMTS terrestrial radio access (E-UTRA), which may use Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-A Pro) to establish air interface 116.

[0047] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as NR radio access, which can establish an air interface 116 using a new radio (NR).

[0048] In one embodiment, base station 114a and WTRUs 102a, 102b, and 102c can implement multiple radio access technologies. For example, base station 114a and WTRUs 102a, 102b, and 102c can jointly implement LTE radio access and NR radio access, for example, using the dual connectivity (DC) principle. Therefore, the air interface used by WTRUs 102a, 102b, and 102c can be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).

[0049] In other embodiments, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as IEEE 802.11 (i.e., Wi-Fi), IEEE 802.16 (i.e., WiMAX), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rate GSM Evolution (EDGE), GSM EDGE (GERAN), etc.

[0050] For example, Figure 1ABase station 114b can be a wireless router, home node B, home eNodeB, or access point, and can utilize any suitable RAT to facilitate wireless connectivity in a local area, such as commercial locations, homes, vehicles, campuses, industrial facilities, air corridors (e.g., for drone use), roads, etc. In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.11 to establish a wireless local area network (WLAN). In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, base station 114b and WTRUs 102c, 102d can utilize cellular-based RATs (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish picocells or femtocells. Figure 1A As shown, base station 114b can be directly connected to Internet 110. Therefore, base station 114b may not need to access Internet 110 via CN 106.

[0051] RAN 104 can communicate with CN 106, which can be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRUs 102a, 102b, 102c, and 102d. Data can have different Quality of Service (QoS) requirements, such as different throughput requirements, latency requirements, fault tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. CN 106 can provide call control, billing services, location-based services, prepaid calling, internet connectivity, video distribution, and / or perform advanced security functions such as user authentication. Although in Figure 1A As not shown, but it should be understood that RAN 104 and / or CN 106 can communicate directly or indirectly with other RANs that use the same RAT as RAN 104 or a different RAT. For example, in addition to connecting to RAN 104, which may utilize NR radio technology, CN 106 can also communicate with another RAN (not shown) that uses GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0052] CN 106 can also serve as a gateway for WTRUs 102a, 102b, 102c, and 102d to access PSTN 108, the Internet 110, and / or other networks 112. PSTN 108 may include a circuit-switched telephone network providing Common Old-Style Telephone Service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices using common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) from the TCP / IP Internet Protocol suite. Network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs, which may use the same RAT as RAN 104 or a different RAT.

[0053] Some or all of the WTRUs 102a, 102b, 102c, and 102d in the communication system 100 may include multi-mode capabilities (e.g., WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). For example... Figure 1A The WTRU 102c shown can be configured to communicate with base station 114a, which may employ cellular-based radio technology, and with base station 114b, which may employ IEEE 802 radio technology.

[0054] Figure 1B This is a system diagram illustrating the example WTRU 102. (Example:) Figure 1B As shown, among other things, WTRU 102 may include, in particular, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a global positioning system (GPS) chipset 136, and / or other peripheral devices 138, etc. It should be understood that WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with the embodiments.

[0055] Processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. Processor 118 may perform signal encoding, data processing, power control, input / output processing, and / or any other functions that enable WTRU 102 to operate in a wireless environment. Processor 118 may be coupled to transceiver 120, which may be coupled to transmitting / receiving element 122. Although Figure 1B The processor 118 and transceiver 120 are depicted as separate components, but it should be understood that the processor 118 and transceiver 120 may be integrated together in an electronic package or chip.

[0056] Transmitting / receiving element 122 can be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) over air interface 116. For example, in one embodiment, transmitting / receiving element 122 can be an antenna configured to transmit and / or receive RF signals. In one embodiment, transmitting / receiving element 122 can be, for example, a transmitter / detector configured to transmit and / or receive IR, UV, or visible light signals. In yet another embodiment, transmitting / receiving element 122 can be configured to transmit and / or receive both RF and optical signals. It should be understood that transmitting / receiving element 122 can be configured to transmit and / or receive any combination of wireless signals.

[0057] Although the transmitting / receiving element 122 is in Figure 1B While depicted as a single element, WTRU 102 may include any number of transmit / receive elements 122. More specifically, WTRU 102 may employ MIMO technology. Thus, in one embodiment, WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals on air interface 116.

[0058] Transceiver 120 can be configured to modulate signals transmitted by transmitting / receiving element 122 and demodulate signals received by transmitting / receiving element 122. As described above, WTRU 102 can have multi-mode capability. Therefore, for example, transceiver 120 may include multiple transceivers to enable WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0059] The processor 118 of WTRU 102 can be coupled to a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) unit or an organic light-emitting diode (OLED) display unit) and can receive user input data therefrom. The processor 118 can also output user data to the speaker / microphone 124, keypad 126, and / or display / touchpad 128. Furthermore, the processor 118 can access and store information from any type of suitable memory, such as non-removable memory 130 and / or removable memory 132. Non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. Removable memory 132 may include a user identification module (SIM) card, memory stick, secure digital storage (SD) card, etc. In other embodiments, the processor 118 can access and store information from memory that is not physically located on WTRU 102 (e.g., a server or home computer (not shown)).

[0060] The processor 118 can receive power from the power supply 134 and can be configured to distribute and / or control power to other components in the WTRU 102. The power supply 134 can be any suitable device that powers the WTRU 102. For example, the power supply 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0061] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) about the current location of the WTRU 102. In addition to, or instead of, information from the GPS chipset 136, the WTRU 102 may receive location information on the air interface 116 from base stations (e.g., base stations 114a, 114b) and / or determine its location based on the timing of signals received from two or more nearby base stations. It should be understood that the WTRU 102 may acquire location information using any suitable location determination method while remaining consistent with the embodiments.

[0062] The processor 118 may be further coupled to other peripheral devices 138, which may include one or more software and / or hardware modules providing additional features, functions, and / or wired or wireless connectivity. For example, peripheral devices 138 may include accelerometers, electronic compasses, satellite transceivers, digital cameras (for photos and / or videos), Universal Serial Bus (USB) ports, vibration devices, television transceivers, hands-free headsets, Bluetooth® modules, FM radio units, digital music players, media players, video game player modules, internet browsers, virtual reality and / or augmented reality (VR / AR) devices, activity trackers, etc. Peripheral devices 138 may include one or more sensors, which may be one or more of the following: gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors; geolocation sensors, altimeters, light sensors, touch sensors, magnetometers, barometers, attitude sensors, biosensors, and / or humidity sensors.

[0063] WTRU 102 may include a full-duplex radio for which the transmission and reception of some or all signals (e.g., signals associated with specific subframes for UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit to reduce and / or substantially eliminate self-interference via hardware (e.g., a choke) or via signal processing by a processor (e.g., a separate processor (not shown) or via processor 118). In one embodiment, WTRU 102 may include a half-duplex radio for which the transmission and reception of some or all signals (e.g., signals associated with specific subframes for UL (e.g., for transmission) or downlink (e.g., for reception)) may be concurrent and / or simultaneous.

[0064] Despite WTRU in Figure 1A-1B While described as a wireless terminal, it is conceivable that, in some representative embodiments, such a terminal may use (e.g., temporarily or permanently) a wired communication interface with a communication network.

[0065] In a representative embodiment, another network 112 may be a WLAN.

[0066] Given Figure 1A-1B As described herein, one or more, or all, of the functions described herein can be performed by one or more emulation devices (not shown). An emulation device can be one or more devices configured to emulate one or more, or all, of the functions described herein. For example, an emulation device can be used to test other devices and / or simulate network and / or WTRU functions.

[0067] Simulation devices can be designed to perform one or more tests on other devices in laboratory and / or carrier network environments. For example, one or more simulation devices can perform one or more functions, or all functions, while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices within the communication network. One or more simulation devices can perform one or more functions, or all functions, while being temporarily implemented / deployed as part of a wired and / or wireless communication network. Simulation devices can be directly coupled to another device for testing purposes and / or can perform tests using over-the-air wireless communication.

[0068] One or more simulation devices may perform one or more functions, including all functions, rather than being implemented / deployed as part of a wired and / or wireless communication network. For example, simulation devices may be used to test test scenarios in laboratory and / or non-deployment (e.g., testing) wired and / or wireless communication networks to implement the testing of one or more components. One or more simulation devices may be test devices. Simulation devices may transmit and / or receive data using direct RF coupling and / or wireless communication via RF circuitry (e.g., which may include one or more antennas).

[0069] The embodiments described herein are not limited to implementation on WTRU. Such embodiments can be implemented using, for example... Figure 1C This can be achieved through other systems such as [system name missing]. Figure 1C This is a system diagram illustrating a set of example interfaces for a system according to some embodiments. Interfaces such as... Figure 1C Systems such as 1000 are used to implement extended reality display devices together with their control electronics. System 150 can be embodied as a device including the various components described below and configured to perform one or more of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 150 can be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 150 are distributed across multiple ICs and / or discrete components. In various embodiments, system 150 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the aspects described in this document.

[0070] System 150 includes at least one processor 152 configured to execute instructions loaded thereon for implementing, for example, the various aspects described herein. Processor 152 may include embedded memory, input / output interfaces, and various other circuitry as known in the art. System 150 includes at least one memory 154 (e.g., a volatile memory device and / or a non-volatile memory device). System 150 may include a storage device 158, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 158 may include internal storage devices, additional storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0071] System 150 includes an encoder / decoder module 156 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 156 may include its own processor and memory. The encoder / decoder module 156 represents one or more modules that can be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Additionally, as is known to those skilled in the art, the encoder / decoder module 156 may be implemented as a separate element of system 150, or it may be incorporated within processor 152 as a combination of hardware and software.

[0072] Program code to be loaded onto processor 152 or encoder / decoder 156 to execute the various aspects described herein may be stored in storage device 158 and subsequently loaded onto memory 154 for execution by processor 152. According to various embodiments, one or more of processor 152, memory 154, storage device 158, and encoder / decoder module 156 may store one or more items of various kinds during the execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0073] In some embodiments, the memory within processor 152 and / or encoder / decoder module 156 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be processor 152 or encoder / decoder module 156) is used for one or more of these functions. External memory may be memory 154 and / or storage device 158, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding; also known as H.265 and MPEG H Part 2), or VVC (Multi-Functional Video Coding, a new standard being developed by the Joint Video Experts Group JVET).

[0074] Inputs to the components of system 150 can be provided through various input devices, as indicated in block 172. Such input devices include, but are not limited to: (i) an RF section that receives radio frequency (RF) signals, for example, transmitted over the air by a broadcasting device; (ii) a component input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Other examples include composite video.

[0075] In various embodiments, as known in the art, the input device of block 172 has associated respective input processing elements. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) further limiting the signal to a narrower frequency band to select, for example, a signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements to perform these functions, such as frequency selectors, signal selectors, frequency limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0076] Additionally, the USB and / or HDMI terminals may include their respective interface processors for connecting system 150 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) can be implemented as needed, for example, within a separate input processing IC or within processor 152. Similarly, aspects of USB or HDMI interface processing can be implemented as needed, either within a separate interface IC or within processor 152. Demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 152 and encoder / decoder 156, to operate in conjunction with memory and storage elements to process the data streams as needed for presentation on the output device.

[0077] Various components of system 150 can be housed within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data between them using a suitable connection arrangement 174 (e.g., internal buses as known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0078] System 150 includes a communication interface 160 that enables communication with other devices via a communication channel 162. The communication interface 160 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 162. The communication interface 160 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 162 may be implemented, for example, in a wired and / or wireless medium.

[0079] In various embodiments, data is streamed to or otherwise provided to system 150 using a wireless network such as a Wi-Fi network, for example, IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). In these embodiments, the Wi-Fi signal is received via a communication channel 162 and a communication interface 160 suitable for Wi-Fi communication. The communication channel 162 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that delivers data via an HDMI connection of input block 172 to provide streaming data to system 150. Still other embodiments use an RF connection of input block 172 to provide streaming data to system 150. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks instead of Wi-Fi, such as cellular networks or Bluetooth networks.

[0080] System 150 can provide output signals to various output devices, including display 176, speaker 178, and other peripheral devices 180. Display 176 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 176 can be used in televisions, tablets, laptops, cellular phones (mobile phones), or other devices. Display 176 can also be integrated with other components (e.g., as in a smartphone) or separate (e.g., an external monitor for a laptop). In various examples of embodiments, other peripheral devices 180 include one or more of a standalone digital video disc (or digital multifunction disc) (DVR for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 180 that provide functionality based on the output of system 150. For example, a disc player performs the function of playing the output of system 150.

[0081] In various embodiments, signaling such as AV links, Consumer Electronics Control (CEC), or other communication protocols enabling inter-device control with or without user intervention is used to communicate control signals between system 150 and display 176, speaker 178, or other peripheral devices 180. Output devices can be communicatively coupled to system 1000 via dedicated connections through their respective interfaces 164, 166, and 168. Alternatively, output devices can be connected to system 150 via communication interface 160 using communication channel 162. Display 176 and speaker 178 can be integrated into a single unit with other components of system 150 in electronic devices such as, for example, a television set. In various embodiments, display interface 164 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0082] For example, if the RF portion of input 172 is part of a standalone set-top box, then display 176 and speaker 178 may alternatively be separate from one or more other components. In various embodiments where display 176 and speaker 178 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0083] System 150 may include one or more sensor devices 168. Examples of sensor devices that may be used include one or more GPS sensors, gyroscope sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors can be used to determine information such as the user's position and orientation. Where system 150 is used as a control module (such as control modules 124, 132) for an extended reality display, the user's position and orientation can be used to determine how image data is rendered so that the user perceives the correct portion of a virtual object or scene from the correct viewpoint. In the case of a head-mounted display device, the device's own position and orientation can be used to determine the user's position and orientation for the purpose of rendering virtual content. In the case of other display devices such as telephones, tablets, computer monitors, or televisions, other inputs can be used to determine the user's position and orientation for the purpose of rendering content. For example, the user can use a touchscreen, keypad or keyboard, trackball, joystick, or other inputs to select and / or adjust the desired viewpoint and / or viewing direction. When the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation used for rendering content can be selected and / or adjusted based on the movement of the display device.

[0084] The embodiments can be implemented by computer software implemented by processor 152, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, memory 154 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 152 can be of any type suitable for the technical environment and can encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0085] Overview of Split Inference Using Trained Models Figure 2A and Figure 2B This illustrates the n layers used between the network and the user equipment (UE). The basic architecture of the split inference model (3GPP SA4 AI4media) is described, in which the first inference process handles the first part of the model, i.e., layers 1 to k, and the second inference process handles the second part of the model, layers k+1 to n. This architecture illustrates the function of transferring and accessing intermediate data between the UE and the network for two different scenarios.

[0086] In the first scenario, when the media data source originates from the UE, the first part of the inference is executed in the UE, and the second part is executed over the network. The resulting output data is ultimately sent back to the UE.

[0087] In the second scenario, when the media data source comes from the network or from the network via the UE, the first part of the AI ​​model is executed on the network side, and the second part is executed on the UE.

[0088] Figure 2A The diagram illustrates the split reasoning between the UE and the network, where the media data source is in the UE.

[0089] Figure 2B The diagram illustrates the split reasoning between the UE and the network, where the media data source is in the network.

[0090] Overview of trained models, graphs, and directed acyclic graphs This section describes general examples of trained models, graphs, and directed acyclic graphs. A graph can be described as a data structure that defines the flow of data and computation. An AI / ML (Artificial Intelligence / Machine Learning) model can be described as a set of nodes and edges (links between nodes) that perform a given task, such as classification or regression. In this sense, a model is a concrete implementation of a graph that has been trained on a set of data.

[0091] Graphs can be used to describe the structure of computations to be performed, while models are specific representations of structures that have been trained to perform a particular task. Directed acyclic graphs are a type of graph.

[0092] Therefore, AI training models such as DNNs / CNNs (Deep Neural Networks / Convolutional Neural Networks) and other models can be viewed as a collection of layers / nodes interconnected by edges, which can be represented by a Directed Acyclic Graph (DAG). A DAG is a directed graph without directed cycles (or loops), where edges (or links between nodes) point from one node to another in a manner that avoids closed loops. The topology of a DAG is linearly ordered. Figure 3B These specific features (acyclicity and node ordering) are represented in the text.

[0093] Figure 3A and 3B These are two representations of the same model. Figure 3A The list of nodes or layers is provided. Figure 3B It provides directed acyclic graphs.

[0094] Overview of ONNX format One format that can be used to represent machine learning models is ONNX (Open Neural Network Exchange). ONNX uses a common set of operators and a common file format to enable AI developers to use models with a wide variety of frameworks, tools, runtimes, and compilers. Operators are the building blocks of machine learning and deep learning models.

[0095] ONNX describes a computation graph. Each machine learning operation is represented by a "node". Each node has the following properties: • node.name: A string that identifies this node. • node.input: A list of strings representing the tensor names expected by the node. • node.output: A list of strings representing the names of the tensors generated by the node. • node.op_type: A string that identifies the operation performed on the node.

[0096] All nodes are accessible via a list (e.g., onnx_model.graph.node). The model itself is organized in a tree-like structure in the graph. Figures 4A-4C Examples of different model topologies are provided. Figure 4A The diagram shows a model with one branch. Figure 4B The illustration shows a model with one or two branches. Figure 4C The diagram shows a model with multiple interconnected branches.

[0097] Problems solved in some embodiments When the AI ​​model is split into two parts for dynamic split point reselection, the first endpoint splits the model, executes the first part, and sends an instruction to the second endpoint at the split point to begin. The first endpoint also sends intermediate output data to the second endpoint. The second endpoint computes the received split point instruction to obtain the second part of the model and executes this second part using the received intermediate data to generate the final output.

[0098] One problem is providing specific splitting operations for appropriate AI models. A given trained model for a particular framework (PyTorch, TensorFlow) can be split into chunks by modifying the software that generated the AI ​​model or by using another software to parse the trained model. Therefore, the splitting function is model-specific and may be difficult to generalize to another model.

[0099] Another issue is using static model parts to achieve dynamic splitting. One way to provide dynamic split point reselection is to prepare a set of statically trained model parts, such as part1_splitpoint10, part2_Splitpoint10, part1_splitpoint27, and part2_Splitpoint27. This technique requires preparing and storing a large number of split model parts for each endpoint and for all the different models to be used (which is not optimal).

[0100] Another problem is finding an explicit split. A split based on a single indicator, either a split point number or a split point name, requires deterministic algorithms on both sides capable of unambiguously splitting the model. It also requires the same understanding on both sides of the meaning of the split point (in which part the split point itself is included).

[0101] Another issue is providing explicit transmission of intermediate data. In cases where the model splits into multiple branches, resulting in multiple sets of intermediate data, the receiving side must be able to uniquely map the received intermediate data (model part I output) to the different expected inputs (model part II).

[0102] Some example embodiments may provide one or more of the following benefits: • Provides the ability to split the model at any node, regardless of the number or complexity of links between nodes.

[0103] • Enables a strategy to split any model at any node and measures its effects on sub-model size, latency, energy cost, intermediate data size, and transmission cost to help determine the most effective split point.

[0104] • Provide solutions to communicate several tensors at each inference stage.

[0105] • Provide a solution to avoid sending tensor names at each inference stage.

[0106] Overview of Example Implementations The example embodiments provide and share parsing algorithms and rules for implementing splitting operations on models that can be represented as directed acyclic graphs. Some embodiments include setup procedures, splitting procedures, and inference procedures.

[0107] The example setup procedure describes the set of information to be used to process the AI ​​model graph at each endpoint. Such information may include, for example, graph information, model parsing algorithms, exchange split point metadata formats, and exchange input / output intermediate data formats.

[0108] The example splitting process applies the model splitting to each endpoint according to the rules defined during setup.

[0109] The example inference process describes the actions performed at each endpoint, including transmitting split information from one endpoint to a second endpoint based on the intermediate data format and metadata format.

[0110] The example implementation provides and shares information about parsing algorithms and rules for implementing splitting operations on AI models that can be represented as one or more directed acyclic graphs.

[0111] Figure 5 This is a schematic diagram of two devices implementing a splitting operation on an AI model using a dynamic graph-based AI model splitting according to the embodiments described herein.

[0112] In some embodiments, configuration is performed at both endpoints during the setup process. The configuration process may include determining and exchanging some or all of the following information: model graph information, model parsing algorithm information, split point metadata format information, and input / output intermediate data format information.

[0113] Model graph information may include a model graph identifier to uniquely identify the model graph. This identifier can be selected by application configuration. Model graph information may further include a model graph framework, which can provide an identifier that uniquely identifies the AI / ML framework (ONNX, Keras, PyTorch) on which the model was developed.

[0114] The model parsing algorithm information may include some or all of the following: a unique identifier for the parsing algorithm, and the parsing algorithm and rules used for each node and the node edges connected.

[0115] A unique resolution algorithm identifier can be an identifier that uniquely identifies the model resolution algorithm.

[0116] The parsing algorithm and rules used for each node and connected node edges may include node numbering rules and / or edge node identification information. Node numbering rules may include numbering nodes by their rank in a list, or starting the numbering from, for example, the top / bottom direction and left-to-right direction of the nodes in the graph. Edge node identification may be, for example, the strings 'node.source' or 'node.destination'.

[0117] The split point metadata format information may include information used by the second endpoint to split the second part of the trained model about the first part. The split point information may be based on one or more of the following: name, number, include / exclude flag (in which part the split point is included).

[0118] Input / output intermediate data format information can provide the structure and identifier of the output tensor regarding node or edge information. Such information may include some or all of the following: format type identifier, output tensor parsing rules, tensor identifier, and / or tensor structure format.

[0119] The format type identifier can be an identifier that uniquely identifies the format of the input / output intermediate data (e.g., numpy_vx, torch_tensor, tensorflow_vx.y).

[0120] The output tensor parsing rule defines which rule is used to parse tensors from the received intermediate data. In some embodiments, this rule can be one of the following rules, which are described in further detail below.

[0121] • Rule 1: Send the tensor name during the setup phase and the tensor value during the inference phase.

[0122] • Rule 2: Send both the tensor name and the tensor during the inference phase.

[0123] • Rule 3: Only send tensors during the inference phase; use the tensor's dimensions to associate the tensor name with the tensor.

[0124] • Rule 4: Only tensors are sent during the inference phase; the association between tensor names and tensors is completed based on the alphabetical order of tensor names.

[0125] Tensor identification information can define the name of a tensor, such as an ordered list of tensor names.

[0126] Tensor structure format information can be used to define the tensor format of intermediate data transmitted during the inference phase. Here is an example: • List[Tensor_Name, Tensor] • List[Tensor, Tensor_Name] • List [Tensor] • List[tensor dimension, tensor].

[0127] exist Figure 6 The call flow diagram illustrates an example method, providing details for setting up the process. At 601, each endpoint resolves the model information required by the application: the model name and the AI ​​framework. At 602, each endpoint resolves the model, down to the node number and edge node identifier. At 603, endpoint 1 selects a split point and communicates the split point metadata, which may include some or all of the following: the node name, the node number, and a flag indicating whether the split occurs before or after the split point (i.e., whether the split point is included in part I or part II).

[0128] Upon receiving the data, endpoint 2 checks if the split point can be found in its model and stores conventions regarding the inclusion / exclusion of the split point. At 604, the split results in pending data (tensors) at the output of part I. This step defines the type of intermediate data exchanged (e.g., NumPy arrays, string arrays, serialized data), the algorithm used for splitting, the name and dimensions of the tensors, and the rules for parsing the output tensors. This information is then passed to endpoint 2. Upon receiving this data, endpoint 2 checks its compatibility with the requested format.

[0129] Figures 7A-7B The example shown is a node number. Figures 7A-7B The node numbering rule illustrated below: a number is associated with each node corresponding to its rank in the list. Alternatively, other rules can be used. A potential benefit of this numbering scheme is the ability to cut anywhere in the model without any knowledge of the node names or the links between nodes. This solution implements a strategy to attempt to cut any model at a variety of different nodes and measure the effects on sub-model size, latency, energy cost, intermediate data size, and transfer cost.

[0130] The following provides details of example intermediate data parsing rules that can be used in some embodiments. Intermediate data parsing rules 1 and 2 are not described in more detail because they are straightforward when the tensor and its name are explicitly given. Rules 3 and 4 are further explained. These rules provide examples of techniques that can be used to distinguish tensors when they are received at endpoint 2 without any additional information. For illustration, consider a tensor with the following intermediate data name and dimensions: • Name: ' / additional_blocks.2 / additional_blocks.2.0 / Conv_output_0', Dimensions: float32[1,128,10,10] • Name: / feature_extractor / feature_extractor / feature_extractor.6 / feature_extractor.6 .5 / relu_2 / Relu_output_0, Dimensions: float32 [1,1024,38,38] • Name: ' / additional_blocks.0 / additional_blocks.0.5 / Relu_output_0', Dimensions: float32 [1,512,19,19] • Name: ' / additional_blocks.1 / additional_blocks.1.5 / Relu_output_0', Dimension: float32 [1,512,10,10].

[0131] In the example of rule 3, the association between tensor names and tensors is done using the tensor's dimensions. During the inference phase, tensors can be sent in any order, such as the order above. Upon reception, the receiver receives tensors with different dimensions: [1,128,10,10], [1,1024,38,38], [1,512,19,19], [1,512,10,10]. Knowing the expected dimensions of the input, the receiver can associate the received data with the input tensors. This technique works for tensors with different dimensions.

[0132] In the example of rule 4, the association between tensor names and tensors is based on the alphabetical order of the tensor names. In this example, the alphabetical order of the tensors is: .

[0133] During the inference phase, tensors are sent in this order. Upon reception, the receiver has knowledge of associating the received data with the input tensors.

[0134] In some embodiments, the details of the splitting phase can be described as follows.

[0135] During the split phase, the sender can perform one or more of the following actions: select a split point, apply model resolution rules, and send split point metadata.

[0136] The split point can be selected from a list of possible split points of the model.

[0137] The sender applies a model parsing rule algorithm to split the model from the selected split point to obtain the first part of the model and outputs the tensor name of the first part.

[0138] The sender transmits split-point metadata information according to the split-point metadata format defined in the setup phase. Split-point metadata may include split-point information identifying the split point, such as a node number (e.g., k=75, where k is the node's rank in the node list) and / or a node name (e.g., "conv2"). Split-point metadata may further include include / exclude flags indicating whether the split occurs before or after the identified point.

[0139] During the split-point phase, the receiver can obtain split-point information from the split-point metadata, such as k=75. The receiver can then apply a model parsing rule algorithm to split the model from the selected split-point to obtain the second part of the model and the names of the input tensors for that second part.

[0140] During the inference phase, the sender performs the inference for the first part of the model and passes intermediate data (possibly with appropriate metadata) to the receiver. The intermediate data is then output according to the output intermediate data format rules defined in the setup phase.

[0141] During the example inference phase, the receiver may perform one or more of the following actions: The receiver receives intermediate data and / or tensor name information according to the setup phase rules. The receiver receives the tensor payload according to the tensor structure format. For example, the payload may be in the form of [[Tensor1, Tensor2]], or, if tensor names are used, it may be... , In the form of... As mentioned above, the receiver assigns a list of dictionary tensors to the input names according to the setup phase rules. For example, The receiver feeds intermediate data to the inference stage and performs the second part of the model's inference.

[0142] Examples of graph parse algorithms that may be used in some embodiments are described in the following sections.

[0143] Graph analysis of linear graphs For some AI models (e.g., VGG16), this structure consists of only one branch. Figure 4A This diagram illustrates such a model. Each node has only one input and one output. The output of node "i" is the input of node "i+1".

[0144] In this case, such as Figure 8 As shown in the flowchart, the example embodiment allows the model to be split anywhere on the list of nodes and creates (at least) two split models.

[0145] For example, let "n" represent the number of nodes, and "k" be used as the index to identify the split. The model can be split at index "k", thus retaining the first "k" nodes for part I and the nodes from "k" to "n" for part II. As a result, the node list {0..n} of the model is split into model part I with node list {0..k-1} and model part II with node list {k..n}.

[0146] In the example model where Part I runs on the first device and Model Part II runs on the second device, both the "intermediate data" representing the intermediate output of Model Part I and the index "k" can be transmitted to the second device.

[0147] The following is a description of the algorithm implemented using a linear model through the Keras interface of the TensorFlow library.

[0148] Python code: Call # Perform inference using model_a on the first device # Transmit the intermediate data and input, which will serve as the result of inference model_a, to the second device. # Perform inference using model_b on the second device with the previously obtained intermediate data. # Send the results back to the first device.

[0149] Graph analysis of complex graphs with several branches Some implementations can be used with complex AI models that can contain many branches, such as YOLOv4, SSD_ResNet, and RetinaNet, among others. In such implementations, it is desirable to identify all the inputs required for Part II of the model. The input of one node can be the output of another node, or it can be constant data contained in the model file (e.g., the weights and biases of convolutional nodes; these are referred to as "initializers"). Furthermore, the output of one node can be used as the input of several nodes.

[0150] Figure 9A and 9B An example of node numbering is shown. Figures 9A-9B In the example, the output of node "MaxPool" (node ​​2) is used by the first nodes "Conv" (node ​​3) and "Add" (node ​​6).

[0151] In cases where it is expected that the graph can be split at any location, pending data resulting from this split should be considered.

[0152] exist Figures 9A-9B In the example, if the model happens to be in the first "Conv" function After being cut, the pending data will be: • Output of the MaxPool node • and the output of the Conv node .

[0153] Various algorithms can be used to identify pending data. Specific examples are described below as Example 1 and Example 2.

[0154] Example 1 In one example, AI computes directed acyclic graphs (such as...) Figure 10 The example shown consists of a set of nodes and edges.

[0155] Each node represents a separate AI / ML operation. Each node can be described using some or all of the following information.

[0156] • node.id('n' integer): where the numbering starts from the top / bottom direction and from left to right when resolving nodes, e.g., n3 node 3) • node.name: A string that identifies this node. • node.op_type: A string that identifies the AI ​​operation of the node. • node.input: A list of strings representing the tensor names expected by the node. • node.output: A list of strings representing the names of the tensors generated by the node.

[0157] Each edge represents a tensor that is output from the source node and input to the destination node. Each tensor can be described by some or all of the following information: • edge.id ('n''source.node'.'destination.node.id'): Represents the edge number that connects the source and destination nodes. (For example, edge e1.2 means the edge from node 1 to node 2.) • edge.tensor: A string representing the name of a tensor, or the output of the source node and input to the destination node.

[0158] The first process in this example is node and edge identification. This can be done using a node numbering process, which generates a unique number for each node in the graph. The example numbering rule is to start numbering nodes from top / bottom and left to right. The node identification process can produce unique node names, such as 'conv_3.1'. Edge identification can be performed to produce names that bind and carry the relationship between two existing node connections. For example, e1.5 could be used to identify the relationship between node 1 and node 5, which bypasses nodes 2, 3, and 4. Each node's unique number is associated with and bound to a unique node name 'for example, Conv_3', and AI operations are associated with the node number.

[0159] The second process in this example is a split operation performed on the graph as numbered in the first process above. For the split performed at node n3, the sets of edges belonging to path part 1 and part 2 can be listed in the following two lists.

[0160] • The first list is a list of direct n3 predecessor edges. Figure 10 In the example, this only includes edge e1.3.

[0161] • The second list is a list of edges, where, in Figure 10 In the example, node n i Connect to node n j , where i belongs to [1..2] and j belongs to [4..6]. Figure 10 In the example, this includes edges e1.5 and e2.4.

[0162] Based on these lists • Part 1 includes a subset of the graph preceding n3 (n1 and n2) with the output tensor of the edges (edge ​​1.3, edge e1.5, and edge e2.4) identified above.

[0163] • Part 2 comprises a subset of the graph, including n3 and the remaining nodes (n3, n4, n5, n6), with an input tensor containing the edges (edge ​​1.3, edge e1.5, and edge e2.4) identified above.

[0164] The method in Example 1 can be summarized as follows. The first process involves parsing and numbering the edges between nodes: • Node numbering: The graph is analyzed and the nodes are numbered from top to bottom and from left to right.

[0165] • Edge numbering: The graph is parsed and the nodes are numbered to represent the relationship between two nodes. For example, e 1.5 represents the relationship between node 1 and node 5.

[0166] For the split performed at node k, the set of edges belonging to path part 1 and part 2 can be listed as follows: • A list of direct predecessor edges reaching node k. This includes at most the set of edges e0.k..k-1. • Node n k-1 Directly connected to node n k+1 list of edges Based on these lists: • Part 1 includes a subset of the graph preceding node k and the output tensor of the previously identified edges, such as 'conv_output_0'. • Part 2 includes a subset of the graph that includes node k and the input tensor with the previously labeled edge 'conv_input_0'.

[0167] Example 2 Reference here Figure 11A The model description is a second example of an algorithm used to split the model. The model can be performed as follows.

[0168] • Select model M • Choose the split point k (e.g., n3). • Split model M into M1(n1,n2) at split point k. Figure 11B ) and M2(n3,n4,n5,n6,n7) Figure 11C ) • Create a list of all outputs for all nodes of the full model M, named "all_outputs" (e.g., e1.2, e1.3, e1.5, e2.4, e3.4, e4.5, e4.6, e5.7, e6.7). • Create a list “all_inputs_needed” of all inputs to model M2 from the outputs of other nodes (i.e., also existing in the list “all_outputs”); we do not retain inputs from the model itself as weights and biases (e.g., e1.3, e1.5, e2.4, e3.4, e4.5, e4.6, e5.7, e6.7). • Remove all inputs generated by the nodes of model M2 from the list “all_inputs_needed” (e.g., remove e3.4, e4.5, e4.6, e5.7, e6.7 => all_inputs_needed = {e1.3, e1.5, e2.4}).

[0169] In the Python implementation, Example 2 can be expressed as follows: # We first construct a list of all outputs for all nodes. # After splitting, we find all the inputs needed for each node. #We check whether this input was generated by the previous node or is constant data. # Then remove all entries generated by the nodes after the split. .

[0170] The output of this process is a list of node outputs, which represents a list of all outputs of the first part of the model and / or a list of all inputs of the second part of the model, excluding the inputs of the first part of the model.

[0171] In the first variant of Example 2 (here referred to as "Variation 1"), for some models and some splits, the input to M1 is still requested by model M2. For example, the first node does not process the input, but only some internal variables of the model. If a split is performed on the first node of the model, the input will be processed on the second part of the model. Figures 15A-15C The image below illustrates such an example. Figure 15A The diagram illustrates the original model M. In the original model M, the input image is the input to node n1 and also to node n4. This model can be split at node n3, resulting in... Figure 15B The first model part M1 shown, and Figure 15C The second model part M2 is shown. Each model part receives an image as input (to node n1 in model M1 and node n4 in model M2).

[0172] exist Figures 16A-16B The diagram illustrates another example, showing the split of the RetinaNet model at node five of the model's numbering scheme. Figure 16A The diagram shows the first part (M1) of the split model. The input to M1 is "input_images", and the output is tensors "Add_3012", "Mul_3013", "Add_3014", "Mul_3015", and "Add_3016". Figure 16B The diagram illustrates the second part (M2) of the split model. The inputs to M2 include the outputs of M1, namely “Add_3012”, “Mul_3013”, “Add_3014”, “Mul_3015”, “Add_3016”, and “input_images”, which are also inputs to M1.

[0173] In the second variant of Example 2 (here referred to as "Variation 2"), for some models and some splits, the final output of the model can be distributed between the two parts of the model. Figures 17A-17C The image shows an example of this. For example... Figure 17A As shown, the complete model M in this example has three final outputs: P1, P2, and P3. Figure 17B The diagram illustrates the first sub-model (M1) generated from the split model M, and Figure 17C The illustration shows the second sub-model generated by splitting model M. In this example, output P1 can be generated by sub-model M1, and outputs P2 and P3 can be generated by sub-model M2. For example, in an object detection model, P1 could represent the "box" around the detected item, P2 could be the label, and P3 could be the confidence score.

[0174] In another example, Figure 18A The illustration shows the properties of the first sub-model obtained by splitting the RetinaNet model at node 2248, and... Figure 18B The illustration shows the properties of the second sub-model obtained by splitting the RetinaNet model at node 2248. The final output of the combined model is "2712", "2713", and "2734". With the split at node 2248, the output of model M1 is "2712" and "2713", and the output of model M2 is "2734".

[0175] Variant 1 and Variant 2 of the example implementation can be implemented together. The pseudocode for such an implementation could be as follows: • Select model M • Choose the split point k (e.g., n3) • Split the model M into M1(n1,n2) and M2(n3,n4,n5,n6,n7) at the split point k. • Create a list of all outputs for all nodes of the full model M, named "all_outputs" (e.g., e1.2, e1.3, e1.5, e2.4, e3.4, e4.5, e4.6, e5.7, e6.7). • Create a list of graph inputs named "graph_input_lst" (e.g., "image"). • Create a list of graph outputs named "graph_output_lst" • Create a list of constant variable initializers named "initializers_lst" (e.g., w3, b3, w4, b4). • Create a list "part_II_inputs" of all inputs to model M2 that are not in the initializer list "initializers_lst" (e.g., images, e1.3, e1.5, e2.4, e3.4, e4.5, e4.6, e5.7, e6.7). • Remove all inputs generated by the nodes of model M2 from the list “part_II_inputs” (e.g., remove e3.4, e4.5, e4.6, e5.7, e6.7 => part_II_inputs = {image, e1.3, e1.5, e2.4}). • Optionally, the output of M1 can be constructed by removing items from the list "part_II_inputs" that also exist in the list "graph_input_lst" of the graph inputs (e.g., {e1.3,e1.5,e2.4}). This step can be skipped in implementations that expect to obtain all the intermediate data needed for the second part of the model. For some library implementations (e.g., the function `utils.extract_mode()` from the ONNX library), it may be desirable to remove items from the output that are also present in the inputs.

[0176] • If the final graph output (from “graph_output_lst”) is included in Part I, it can be included in Part I_outputs and removed from Part II_outputs.

[0177] • Return part_I_outputs, part_II_inputs, part_II_outputs.

[0178] The example Python implementations for both "Variant 1" and "Variant 2" can be implemented as follows: Returns the input used by the "if" node. For this type of node, some branches of the node ("then", or "else") may require external inputs that are not declared as inputs to "If" nodes. First, initialize the list of inputs containing the node "If". # Then parse the "else" and "then" branches to add additional input. # Expected "then_branch" or "else_branch" # Clean: Remove tensors from the input that also exist in the output Identify the I / O connection between the two split parts, i.e., the output of the first part / the input of the second part. ''' # We build the output set of all nodes in the full model for a reasonableness check. # We create a list of graph inputs for validity checks and to determine if the second part is using inputs from the first part. # We build a list of graph outputs to detect whether part I has an output that serves as a graph output. # Get constant variable initializer # We look for all the inputs needed for each node after the split; that is, we build a set of all inputs for the second part. # If it is not an initializer, then we retain this input. # Reasonableness Check # Then, remove all entries generated by the nodes after the split; that is, remove the entries generated by the second part itself. # Create a list of all output tensors of part I # The output of the first part is almost identical to the input of the second part, except: # - The input graph should not exist in the output of the first part, as it is not supported by the function onnx.utils.extract_mode(). # - If the final graph output is included in Part I, then it should be included in Part I and removed from the Part II output. # Returns all outputs from Part I and all inputs from Part I

[0179] In the example implementation, the output of this method is a list of three nodes: a list of all outputs of the first part of the model, a list of all inputs of the second part of the model, and a list of all outputs of the second part of the model.

[0180] In this example, nodes are resolved using op_type "If". Nodes of type "if" take a single-element tensor representing the condition as input. "If" nodes contain a "then branch" and an "else branch". Each of these branches contains an embedding graph. If the condition is true, the "then" branch is executed. If the condition is false, the "else" branch is executed. In these graphs, some nodes may involve tensors of the main graph without indicating such dependencies in the input of the "if node". Therefore, some embodiments can operate to browse both branches of the "if node" to detect hidden dependencies on external tensors.

[0181] The function GetInputsfromIfNode(node) above provides an example of a Python implementation.

[0182] In some embodiments, methods described in the pseudocode, such as those described below, return all the inputs required by a node, including the inputs expected by the "then branch" and "else branch": • Create an empty list “Input”.

[0183] • Update the list "Input" using the "Input" of the "if node".

[0184] • Create an empty list named "output".

[0185] • parse all nodes of the "then branch" and "else branch", update the "input" list with each input of each node, and update the "output" list with each output of each node.

[0186] • Remove all elements present in the "Output" list from the "Input" list.

[0187] • Returns the "Input" list. Intermediate metadata communication.

[0188] A branch split (split at the bottleneck) ) The example implementation operation is used to transmit both the split point identifier (index number "k" or node name) and "intermediate data" representing the intermediate output of the model. Figure 12 The illustration shows an example embodiment.

[0189] In some embodiments, the split number (k) is communicated synchronously (inline). In other embodiments, the split number is communicated asynchronously. This can be indicated during the setup phase.

[0190] When the split number is communicated asynchronously, the sender can communicate metadata including the split point identifier (number or name) and conventions regarding the split point markers (before, after). The sender performs the first part of the reasoning, encodes the intermediate data, and communicates it to the receiver. The receiver receives the metadata including the split point identifier and performs the split accordingly. The receiver receives the intermediate data and performs the second part of the reasoning.

[0191] When the split point number is communicated synchronously, the sender performs the first part of the reasoning and encodes the intermediate data. Then, the sender communicates metadata, including the split point identifier (number or name) and conventions regarding the split point markers (before, after), as well as the intermediate data. The receiver receives the metadata, the split point identifier (number or name), the conventions regarding the split point markers (before, after), and the intermediate data. The receiver then performs the second part of the reasoning.

[0192] Multi-branch splitting As an example, the ssd_resnet model can be split exactly before node number 106 in the model's numbering scheme. At this level of splitting, four branches are cut. The names of the pending data are: .

[0193] Figure 13A The output attributes of the first part of the illustrated model. Figure 13B The diagram illustrates the input attributes of the second part of the model. At the end of Part I inference, the result is four tensors. Therefore, in this case, there are four tensors to be conveyed, and it is expected that the receiver can identify which tensor is associated with which input. Different embodiments use different techniques to provide this information to the receiver.

[0194] In some embodiments, both the tensor itself and the tensor_name are communicated during the inference phase. In such embodiments, the sender communicates metadata during the setup phase, including split-point identifiers (numbers or names) and conventions regarding split-point markers (before, after). During the first part of the inference phase, the sender encodes intermediate data and communicates metadata including tensor names and the intermediate data for each tensor. As an example, the communicated information can be expressed as follows: .

[0195] During the setup phase, the receiver receives metadata including split point identifiers and performs splitting accordingly. During the inference phase, the receiver receives metadata including both tensor names and intermediate data for each tensor. Based on the metadata, the receiver reconstructs the dictionary. For example, a dictionary can be reconstructed as follows.

[0196] .

[0197] Then, the receiver proceeds with the second part of the reasoning.

[0198] In other embodiments, it is assumed that the order of tensor names (either by convention or explicitly during the setup phase) is known to the receiving side, and only the tensors themselves are communicated. In one such embodiment, the sender communicates metadata during the setup phase, including split-point identifiers (numbers or names) and conventions regarding split-point markers (before, after), and an ordered list of tensor names, such as below.

[0199] .

[0200] During the inference phase, the sender performs the first part of the inference, encodes the intermediate data, and transmits the intermediate data for each tensor, as shown below.

[0201] .

[0202] During the setup phase, the receiver receives metadata including split point identifiers and an ordered list of tensor names, performs the splits accordingly, and records the ordered list of tensor names. During the inference phase, the receiver receives intermediate data for each tensor and reconstructs the dictionary. For example, a dictionary can be reconstructed as follows.

[0203] .

[0204] Then, the receiver proceeds with the second part of the reasoning.

[0205] The following table summarizes examples of information that may be used in some embodiments.

[0206] Figures 14A-14C The illustration shows an example of splitting an AI model, specifically the SSD_resnet model. Figure 14AThe diagram illustrates the first part of the model. This first part of the model can be implemented on a first device. It receives input and provides two outputs, Conv_output_0 and MaxPool_output_0. The first device can then provide these outputs to a second device. (The diagram is repeated in the original text.) Figure 14B-14C As shown, the second device can implement the second part of the model. The second part of the model receives Conv_output_0 and MaxPool_output_0 as inputs and processes them accordingly.

[0207] Other embodiments A method according to some embodiments includes: obtaining a machine learning (ML) model; obtaining information indicating a split of the ML model into at least a first sub-model and a second sub-model; obtaining input information representing inputs to the second sub-model; and using the input information to execute the second sub-model.

[0208] In some embodiments, the ML model includes multiple nodes and multiple edges, with each edge connecting two nodes. In some embodiments, the nodes and edges of the ML model are arranged as a directed acyclic graph.

[0209] In some embodiments, the information indicating a split of the ML model includes information identifying nodes in the ML model. In some such embodiments, the information indicating a split of the ML model includes information identifying only one node in the ML model. In some embodiments, the information indicating a split includes information indicating whether the split occurs before or after the identified node.

[0210] In some embodiments, the input information includes at least one tensor. In some embodiments, the input information includes multiple tensors. In some embodiments, the input information further includes information indicating the name of each of the tensors.

[0211] In some embodiments, information indicating the splitting of the ML model and the input information are received from a first endpoint, and the second sub-model is executed by a second endpoint different from the first endpoint.

[0212] Some embodiments further include splitting the ML model into a first sub-model and a second sub-model based on information indicating the split.

[0213] In some embodiments, the ML model includes a plurality of nodes, the information indicating the split includes information identifying the nodes of the ML model, and splitting the ML model includes: determining a number assigned to each of the nodes; including all nodes with numbers less than the number of the identified node in a first sub-model; and including all nodes with numbers greater than the number of the identified node in a second sub-model. In some embodiments, the identified node is included in the first sub-model. In some embodiments, the identified node is included in the second sub-model.

[0214] Some embodiments further include: determining a set of inputs to nodes in the second sub-model; and determining a set of inputs to the second sub-model by excluding the outputs of nodes in the second sub-model from the set of inputs to nodes in the second sub-model.

[0215] In some embodiments, the input to the second sub-model of the ML model includes the input to the ML model.

[0216] In some embodiments, the output of the ML model includes at least one output of the first sub-model and at least one output of the second sub-model.

[0217] A method according to some embodiments includes: obtaining a machine learning (ML) model; providing information indicating a split of the ML model into at least a first sub-model and a second sub-model; obtaining input information representing inputs to the first sub-model; executing the first sub-model using the input information; and providing output information indicating the outputs of the first sub-model.

[0218] In some embodiments, the ML model includes multiple nodes and multiple edges, with each edge connecting two nodes. In some embodiments, the nodes and edges of the ML model are arranged as a directed acyclic graph.

[0219] In some embodiments, the information indicating a split of the ML model includes information identifying nodes in the ML model. In some such embodiments, the information indicating a split of the ML model includes information identifying only one node in the ML model. In some embodiments, the information indicating a split includes information indicating whether the split occurs before or after the identified node.

[0220] In some embodiments, the output information includes at least one tensor. In some embodiments, the output information includes multiple tensors.

[0221] In some embodiments, the output information further includes information indicating the name of each tensor.

[0222] In some embodiments, information indicating the splitting of the ML model and the output information are provided to a second endpoint, and the first sub-model is executed by a first endpoint different from the second endpoint.

[0223] Some embodiments further include splitting the ML model into a first sub-model and a second sub-model based on information indicating the split.

[0224] In some embodiments, the ML model includes a plurality of nodes, the information indicating the split includes information identifying the nodes of the ML model, and splitting the ML model includes: determining a number assigned to each of the nodes; including all nodes with numbers less than the number of the identified node in a first sub-model; and including all nodes with numbers greater than the number of the identified node in a second sub-model. In some embodiments, the identified node is included in the first sub-model. In some embodiments, the identified node is included in the second sub-model.

[0225] The method according to some embodiments includes: determining a set of inputs to nodes in a second sub-model; and determining the set of inputs to the second sub-model by excluding the outputs of nodes in the second sub-model from the set of inputs to nodes in the second sub-model.

[0226] In some embodiments, the input to the second sub-model of the ML model includes the input to the ML model.

[0227] In some embodiments, the output of the ML model includes at least one output of the first sub-model and at least one output of the second sub-model.

[0228] An apparatus according to some embodiments may include a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods described herein.

[0229] An apparatus according to some embodiments includes at least one processor configured to perform any of the methods described herein.

[0230] An apparatus according to some embodiments includes a computer-readable medium storing instructions that cause one or more processors to perform any of the methods described herein.

[0231] An apparatus according to some embodiments may include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform any of the methods described herein.

[0232] This disclosure describes a wide variety of aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described in detail and are generally described in a manner that may sound limiting, at least for the purpose of illustrating individual characteristics. However, this is for the purpose of clarity and does not limit the disclosure or scope of those aspects. In fact, all the different aspects can be combined and interchanged to provide further aspects. Furthermore, the aspects described can also be combined and interchanged with those described in earlier applications.

[0233] The aspects described and contemplated in this disclosure can be implemented in many different forms. While specific embodiments are illustrated, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of implementation. At least one aspect generally relates to encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions thereon stored for encoding or decoding data according to any of the described methods, and / or computer-readable storage media having a bitstream generated according to any of the described methods stored thereon.

[0234] This document describes various methods, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Furthermore, terms such as "first," "second," etc., can be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless specifically required, the use of such terms does not imply an ordering of the modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding and can occur, for example, before, during, or in a time period overlapping with the second decoding.

[0235] For example, various numerical values ​​may be used in this disclosure. Specific values ​​are for illustrative purposes, and the aspects described are not limited to these specific values.

[0236] The embodiments described herein can be implemented by computer software, or by a combination of hardware and software, implemented by a processor or other hardware. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, the processor can be of any type suitable for the technical environment and can encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0237] Various implementations involve decoding. As used in this disclosure, "decoding" can encompass all or part of a process performed, for example, on a received encoded sequence to produce a final output suitable for presentation. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also includes, or alternatively includes, processes performed by a decoder of the various implementations described in this disclosure.

[0238] Various implementations involve encoding. In a manner similar to the discussion above regarding “decoding,” the term “encoding,” as used in this disclosure, can encompass all or part of a process performed on, for example, the input to produce an encoded bitstream. In various embodiments, such a process includes one or more processes typically performed by an encoder, such as splitting, differential coding, transforming, quantizing, and entropy coding. In various embodiments, such a process also includes, or alternatively includes, processes performed by an encoder of the various implementations described in this disclosure.

[0239] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0240] The implementations and aspects described herein can be implemented, for example, in a method or process, apparatus, software program, data stream, or signal. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), the implementation of the features in question can be implemented in other forms (e.g., apparatus or program). An apparatus can be implemented, for example, in suitable hardware, software, and firmware. A method can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. A processor also includes communication devices, such as, for example, a computer, a cellular phone, a portable / personal digital assistant (“PDA”), and other devices that facilitate communication of information between end-users.

[0241] The references to "an embodiment" or "an embodiment" or "an implementation" or "an implementation," and other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the appearance of the phrases "in an embodiment" or "in an embodiment" or "in an implementation" or "in an implementation," and any other variations appearing throughout this disclosure, do not necessarily refer to the same embodiment.

[0242] Additionally, this disclosure may refer to the term "determine" for each piece of information. Determining information may include, for example, one or more of the following: estimation information, calculation information, prediction information, or information retrieved from memory.

[0243] In addition, this disclosure may refer to the various pieces of information “accessed”. Accessed information may include, for example, one or more of the following: receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0244] Additionally, this disclosure may refer to the term "receive" for each piece of information. Like "access," "receive" is intended to be a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) one or more of them. Furthermore, "receive" is generally referred to in one manner or another during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0245] To understand this, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended to as many listed items as possible.

[0246] Furthermore, as used herein, the term "signaling" specifically refers to instructing the corresponding decoder to do something. For example, in some embodiments, the encoder signals a particular one of several parameters used for region-based filter parameter selection for artifact removal filtering. In this way, in one embodiment, the same parameter is used on both the encoder and decoder sides. Thus, for example, the encoder can transmit (explicit signaling) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various embodiments by avoiding the transmission of any actual functionality. It should be understood that signaling can be done in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. Although the verb form of the term "signaling" has been referred to above, the term "signaling" can also be used as a noun herein.

[0247] The implementation can generate various signals that are formatted to carry information, such as information that can be stored or transmitted. This information may include, for example, instructions for performing a method, or data generated by one of the described embodiments. For example, the signal may be formatted to carry a bitstream of the described embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding the data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via various wired or wireless links. The signal may be stored on a processor-readable medium.

[0248] We have described many embodiments. The features of these embodiments may be provided individually or in any combination across various claim classes and types.

[0249] Note that the various hardware elements in one or more embodiments described herein are referred to as “modules”, which perform (i.e., execute, implement, etc.) the various functions described herein in connection with the respective modules. As used herein, a module includes hardware considered suitable for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices). Each described module may further include executable instructions for performing one or more functions described as being performed by the respective module, and note that these instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or the like, or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or the like, and may be stored in one or more suitable non-transitory computer-readable media, such as commonly referred to as RAM, ROM, etc.

[0250] Although features and elements have been described above in specific combinations, each feature or element may be used alone or in combination with other features and elements. Furthermore, the methods described herein can be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor storage devices, magnetic media (such as internal hard disks and removable disks), magneto-optical media, and optical media (such as CD-ROMs and digital versatile discs (DVDs)). The processor associated with the software can be used to implement a radio frequency transceiver used in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method comprising: Obtain information describing the graph structure of a machine learning (ML) model, wherein the ML model has multiple nodes; Obtain information indicating the splitting of the ML model into at least a first sub-model and a second sub-model, each of the nodes being in one of the sub-models; At the first endpoint, at least the first sub-model output is obtained by executing the first sub-model on the model input; Send the output of the first sub-model from the first endpoint to the second endpoint; and In response to determining that the model input is an input to at least one node in the second sub-model, the model input is sent to the second endpoint.

2. An apparatus comprising one or more processors, said one or more processors being configured to perform at least: Obtain information describing the graph structure of the machine learning (ML) model, where, The ML model has multiple nodes; Obtain information indicating the splitting of the ML model into at least a first sub-model and a second sub-model, each of the nodes being in one of the sub-models; At the first endpoint, at least the first sub-model output is obtained by executing the first sub-model on the model input; Send the output of the first sub-model from the first endpoint to the second endpoint; as well as In response to determining that the model input is an input to at least one node in the second sub-model, the model input is sent to the second endpoint.

3. The method of claim 1 or the apparatus of claim 2, further comprising obtaining at least a second sub-model output by executing the first sub-model on the model input; in, The ML model has multiple model outputs, and the second sub-model output is one of the model outputs.

4. The method according to claim 1 or claim 3 which is dependent on claim 1, or the apparatus according to claim 2 or claim 3 which is dependent on claim 2, wherein, Each of the nodes has a node input and a node output, and further includes determining a sub-model output set by identifying the node input of a node in the second sub-model that does not correspond to the node output of a node in the second sub-model, wherein the first sub-model output is in the sub-model output set.

5. The method according to claim 4, which is dependent on claim 1, or the apparatus according to claim 4, which is dependent on claim 2, wherein, Sending the first sub-model output from the first endpoint to the second endpoint includes sending the determined set of sub-model outputs to the second endpoint.

6. The method according to claim 1 or claims 3-5 which are dependent on claim 1, or the apparatus according to claim 2 or claims 3-5 which are dependent on claim 2, wherein, The ML model has an acyclic directed graph structure.

7. The method according to claim 1 or claims 3-6 which are dependent on claim 1, or the apparatus according to claim 2 or claims 3-6 which are dependent on claim 2, wherein, The output of the first sub-model is a tensor.

8. The method according to claim 1 or claims 3-7 which are dependent on claim 1, or the apparatus according to claim 2 or claims 3-7 which are dependent on claim 2, wherein, The information indicating the split includes information that identifies the split point node.

9. A method comprising: Obtain information describing the graph structure of a machine learning (ML) model, wherein the ML model has multiple nodes, and wherein the ML model is configured to generate at least a model output based on the model input; Obtain information indicating the splitting of the ML model into at least a first sub-model and a second sub-model, each of the nodes being in one of the sub-models; Obtain at least the first sub-model output from the first endpoint; In response to determining that the model input is an input to at least one of the nodes in the second sub-model, the model input is obtained from the first endpoint; and At the second endpoint, the model output is obtained by executing the second sub-model on the model input and the first sub-model output.

10. An apparatus comprising one or more processors, said one or more processors being configured to perform at least: Obtain information describing the graph structure of the machine learning (ML) model, where, The ML model has multiple nodes, and the ML model is configured to generate at least a model output based on the model input; Obtain information indicating the splitting of the ML model into at least a first sub-model and a second sub-model, each of the nodes being in one of the sub-models; Obtain at least the first sub-model output from the first endpoint; In response to determining that the model input is an input to at least one of the nodes in the second sub-model, the model input is obtained from the first endpoint; and At the second endpoint, the model output is obtained by executing the second sub-model on the model input and the first sub-model output.

11. The method of claim 9 or the apparatus of claim 10, further comprising obtaining at least a second sub-model output from the first endpoint; in, The ML model has multiple model outputs, and the second sub-model output is one of the model outputs.

12. The method according to claim 9 or claim 11 which is a dependent of claim 9, or the apparatus according to claim 10 or claim 11 which is a dependent of claim 10, wherein, Each of the nodes has a node input and a node output, and further includes determining a sub-model output set by identifying the node input of a node in the second sub-model that does not correspond to the node output of a node in the second sub-model, wherein the first sub-model output is in the sub-model output set.

13. The method according to claim 12, which is dependent on claim 9, or the apparatus according to claim 12, which is dependent on claim 10, wherein, Obtaining the first sub-model output from the first endpoint includes obtaining the determined set of sub-model outputs to the second endpoint.

14. The method according to claim 9 or claims 11-13 which are dependent on claim 9, or the apparatus according to claim 10 or claims 11-13 which are dependent on claim 10, wherein, The ML model has an acyclic directed graph structure.

15. The method according to claim 9 or claims 11-14 which are dependent on claim 9, or the apparatus according to claim 10 or claims 11-14 which are dependent on claim 10, wherein, The information indicating the split includes information that identifies the split point node.

Images