Divided Neural Network Computing

Abstract

The client device (104) and one or more such networked computing devices (112) coordinate to implement a client-initiated and device-initiated split-neural network computation scheme in which the neural network architecture of the neural network (116) is distributed, or "split," across the client device and one or more networked computing devices, whereby each device operates to perform at least one successive split-neural network portion (122, 124) of an overall neural network operation using input data (130), the intermediate result data (136) generated therefrom being subsequently communicated to the next device in sequence until a final result output (140) is generated by the final split-neural network portion of the sequence. The device generating the final result output can then take one or more actions based on the final result output or can forward the final result output to the device that initiated the split-neural network computation operation for further processing.

Description

Background Art

[0001] User equipment, wearable devices, and other client devices in cellular networks and other wireless networks are increasingly employing neural networks to perform specific operations at various protocol layers in order to improve performance efficiency or enhance user experience. For example, deep neural networks (DNNs) may be used in client devices to provide image analysis, transmission environment recognition, radio frequency signaling, and the like. However, client devices are often resource-constrained, which often makes it impractical to implement a neural network sufficiently complex to provide desired results. For example, a user device may have insufficient battery power, insufficient computational resources, or insufficient network capacity to perform DNN operations within an allocated time frame or at an intended level of complexity.

[0002] The present disclosure will be better understood, and numerous features and advantages will become apparent to those skilled in the art by reference to the accompanying drawings. The use of the same reference symbols in different drawings indicates similar or identical items.

Brief Description of the Drawings

[0003] [Figure 1] FIG. 1 illustrates an example wireless system that uses a split neural network computation scheme to split neural network computational load between two or more devices, in accordance with some embodiments. [Figure 2] FIG. 2 illustrates an example of a split neural network configuration using a client-first implementation, in accordance with some embodiments. [Figure 3] FIG. 3 illustrates another example of a split neural network configuration using an interleaved arrangement, in accordance with some embodiments. [Figure 4]This figure shows an exemplary hardware configuration of user equipment for the wireless system shown in Figure 1, according to several embodiments. [Figure 5] This figure shows an exemplary hardware configuration of the server of the wireless system shown in Figure 1, according to several embodiments. [Figure 6] This figure shows machine learning (ML) modules used in the segmented neural network portion, according to several embodiments. [Figure 7] This flowchart illustrates exemplary methods for segmented neural network computing in wireless systems, according to several embodiments. [Figure 8] This flowchart illustrates an example of a process for negotiating a partitioned neural network configuration between at least two devices, according to several embodiments. [Figure 9] This flowchart illustrates an example of performing machine learning operations using a neural network divided across two or more devices, according to several embodiments. [Figure 10] These are ladder signaling diagrams showing examples of operation of the methods in Figures 7 to 9, according to several embodiments. [Modes for carrying out the invention]

[0004] Client devices often avoid implementing complex DNNs or other neural networks due to various resource constraints. However, the networks to which they are connected often access servers or other computing devices with significantly fewer resource constraints. Taking advantage of this, in at least one embodiment, the client device and one or more such networked computing devices are coordinated to implement a partitioned neural network computation scheme in which the neural network architecture of the neural network is distributed or “partitioned” across the client device and one or more networked computing devices, so that each device operates to execute at least one consecutive portion of the overall DNN operation (hereinafter, the “partitioned neural network portion” or simply the “partitioned portion”), and the intermediate result data generated therefrom is then communicated sequentially to the next device until the final partitioned neural network portion of the sequence generates a final result output. The device that generates the final result output can then take one or more actions based on the final result output, or it can forward the final result output to a device that initiated a partitioned neural network computation operation for further processing.

[0005] In at least one embodiment, this partitioned neural network configuration is initiated by a client device. Therefore, in response to the initiation of an operation involving the use of the neural network, one of the client devices or networked computing devices determines a relevant set of current local conditions for the client device, such as its available computing resources (e.g., processor and memory), current battery conditions, current network conditions, and current thermal conditions. Furthermore, in some embodiments, there may be execution requirements related to the operation, such as latency conditions indicating the maximum allowable period for generating the final result output, quality of service (QoS) conditions indicating the maximum amount of time allowed to complete the operation, or precision conditions indicating a minimum level of precision or other performance level in the final result output. If the client device determines, considering its current local conditions, that it can satisfy the latency conditions, QoS conditions, precision conditions, or other such execution requirements, the client device may choose to perform the operation entirely locally, i.e., it may choose to perform the operation using a fully local neural network architecture for the neural network.

[0006] However, if one or more specified execution requirements are unlikely to be met by local execution alone, in at least some embodiments, either or both of the client device and / or networked computing devices determine a partitioned neural network configuration, where the neural network architecture of the neural network is allocated or "partitioned" between the client device and the networked computing devices (or multiple networked computing devices). The specific partitioning between devices may be based on current local conditions, execution requirements, or a combination thereof. For example, if there are requirements for relatively low availability of local computing resources, low battery reserves, and / or low latency, the partitioned neural network configuration may allocate more layers of the neural network architecture to one or more networked computing devices and fewer layers to the client device; on the other hand, if there is relatively high availability of local computing resources, high battery reserves, and high latency, the partitioned neural network configuration may allocate more layers of the neural network to itself and fewer layers to one or more networked devices.

[0007] In some embodiments, the initiating device then initiates negotiations with other devices(s) intended to be included in the divided neural network configuration by sending a partitioning computation request using the proposed partitioned neural network configuration. The other devices(s) may then consider their own resource constraints and execution requirements(s) when deciding whether to accept the proposed partitioned neural network configuration or to counter-propose a different partitioned neural network configuration that is more compatible with their resource constraints and execution requirements(s). Once the client devices and one or more networked computing resources have decided on an agreed-upon partitioned neural network configuration, each of the client devices and one or more networked computing resources implements its corresponding portion of the neural network architecture (i.e., its corresponding "partition") in its corresponding machine learning (ML) module. The first device in the sequence of partitioned neural network configurations then receives the input data for operation, processes the input data in its ML module, and provides the output as intermediate result data to the device having the next part of the sequence of neural network architectures.

[0008] In some embodiments, the division is a 2-way split, where one device (e.g., a networked computing device) has a first sequence of layers or other initial portion (e.g., a "first stage") of the neural network architecture, and the other device (e.g., a client device) has a last sequence of layers or other final portion (e.g., a "final stage") of the neural network architecture. In other embodiments, the division is an N-way split (N>2), and each of the N stages is assigned to a corresponding device. In some embodiments, each stage is assigned to a different device, for example, the first stage is assigned to one networked computing device, an intermediate stage is assigned to a second networked computing device, and the final stage is assigned to a client device. In other embodiments, the stages may be interleaved among a set of M devices (M<N). For example, the first stage and the final stage may be allocated to the client device, and the intermediate stages may be allocated to the networked computing device.

[0009] With this split computing approach for performing neural network operations, when a client device determines that a local-only solution may prevent achieving one or more conditions or objectives due to its current local conditions, the client device can appropriately utilize the resources of one or more available networked computing devices to assist in performing the neural network operations.

[0010] For the sake of clarity, the systems and technologies described herein are described in the illustrative context of a cellular network in which user equipment (UEs) operate as client devices and one or more servers operate together as networked computing devices. However, these references are for illustrative purposes only, and it will be understood that references to UEs, servers, or cellular networks also apply equally to other client devices, other networked computing devices, and other networks, respectively, unless otherwise noted.

[0011] Furthermore, for the sake of clarity and ease of understanding, the technology of this disclosure is described in exemplary embodiments in which the UE or other client device is the initiator of the neural network partitioning via partitioned neural network configuration and, therefore, the device that triggers the negotiation process. However, in other embodiments, it may be a networked computing device that causes the neural network partitioning between the networked computing device and the client device (and, in some examples, also between one or more additional networked computing devices). Thus, the references and descriptions to initiating or negotiating partitioned neural network configuration by the client device will be understood to apply instead to initiating or negotiating partitioned neural network configuration by the networked computing device using the guidelines provided herein.

[0012] Figure 1 shows an exemplary wireless communication network 100 using a segmented neural network computation scheme in several embodiments. In the illustrated example, the wireless communication network 100 is a cellular network including a network infrastructure 102 wirelessly connected to one or more client devices such as UE 104. The network infrastructure 102 includes a core network 106 coupled to one or more wide area networks (WANs) 108 or other packet data networks (PDNs) such as the Internet. The core network 106 is further connected to at least one base station (BS) 110. The BS 110 supports wireless communication with one or more wireless client devices such as UE 104 via radio frequency (RF) signaling using one or more applicable RATs as defined by one or more communication protocols or standards. Thus, the BS 110 acts as a wireless interface between one or more wireless devices and various networks and services provided by the network infrastructure 102, such as packet-switched (PS) data services and circuit-switched (CS) services.

[0013] BS110 can employ any of the following RATs: operating as a NodeB (or Base Transceiver Station (BTS)) of a Universal Mobile Telecommunications System (UMTS) RAT (also known as "3G"), operating as an Extended NodeB (eNodeB) of a 3G Partnership Project (3GPP®) Long-Term Evolution (LTE) RAT, or operating as a 5G NodeB ("gNB") of a 3GPP 5G New Radio (NR) RAT. UE104 then represents any of the various client devices capable of communicating with BS110 via the appropriate RAT, including, for example, mobile phones, tablet or laptop computers, desktop computers, video game systems, servers, network-enabled consumer electronics, network-enabled automotive communication systems, network-enabled smartwatches, or other wearable devices. The core network 106 and / or WAN 108 includes or has network access to one or more networked computing devices 112 that provide computing resources to the core network, WAN 108, and / or UE 104, such as one or more servers (or server farms), one or more workstations, etc. In the examples described herein, one or more networked computing devices 112 include one or more servers, and therefore the networked computing devices 112 are also referred to herein as “servers 112” for ease of reference.

[0014] In at least one embodiment, the UE104 is configured to implement a machine learning (ML) module (see Figure 6), which can be configured to implement one or more neural networks (or parts thereof) to provide neural network / machine learning capabilities to one or more software applications running on the UE104. Examples of such neural networks include deep neural networks (DNNs) such as convolutional neural networks (CNNs), artificial neural networks (ANNs), recurrent neural networks (RNNs), and generative adversarial networks (GANs). For example, a software application on the UE104 could use an ML module that implements a CNN to provide image detection / categorization, or an ML module that implements an RNN to provide speech detection. However, depending on the complexity or computational resources required to implement the entire neural network, the UE104 may not have sufficient resources to perform the neural network operation locally in a timely manner. For example, in the case of real-time speech detection and translation, the UE104 may not have sufficient computational resources to perform the speech detection and translation operation quickly enough to facilitate a satisfactory user experience. Alternatively or additionally, fully executing neural network operations on the UE104 may impose an unreasonable or unacceptable load on the UE104. For example, running a series of image classification operations using a DNN on the UE104 may excessively drain the battery of the UE104 or induce an unmanageable thermal load on the UE104.

[0015] Therefore, in at least one embodiment, the UE 104 and the network infrastructure 102 together can implement a partitioned neural network computation scheme, where the partitioned neural network configuration 114 is used to implement a neural network 116 that is distributed, or "partitioned," on the UE 104 and one or more servers 112, or other networked computing devices of the network 100, so that each device in the partitioned neural network configuration 114 operates to perform a portion of the overall neural network operation (i.e., a "partitioned neural network portion" or "partition"), and the intermediate results produced therefrom are then communicated sequentially to the next device until the final result output is produced by the last partition in the sequence. The partition assigned to the corresponding device can include any subset or partition of the elements of the neural network used by the partitioned neural network configuration 114. For example, many neural networks are configured as a sequence of layers including an input layer, one or more hidden layers, and an output layer, each layer having one or more nodes (e.g., neurons and / or perceptrons). In such cases, a divided neural network portion may include all the nodes of one or more layers of the neural network, some of the nodes of one or more layers of the neural network, or a combination thereof. In a particular neural network, the layers / nodes may be further organized, and the division of the neural network into corresponding divided neural network portions may occur according to such organization. For example, a GAN typically consists of generators and discriminators, and the division of a GAN across multiple devices may be done considering this configuration, such as the layers and nodes associated with the generators assigned to one or more devices, and the layers and nodes associated with the discriminators assigned to one or more other devices.

[0016] In this embodiment, the neural network partitioning is performed such that the partitioned neural network configuration 114 is implemented as a sequence of partitioned neural network parts representing the entire neural network, where each partitioned neural network part receives either an initial input or an output from a previous partitioned neural network part in the sequence as its input, and provides its output as input to the next partitioned neural network part in the sequence, or as the final result output, depending on the position of the partitioned neural network part in the sequence. In a two-way partition, the first device may be assigned, for example, the input layer and one or more hidden layers following the input layer, and the second device may be assigned, for example, the remaining hidden layers and the output layer. Which partition is assigned to which device may depend on various factors such as the availability of the compared computing resources, the type of neural network operation being performed, and the use of the output. For example, in the case of an image classification operation using image classification results used by an application in UE104, the server 112 may be assigned an initial partition having most of the input and hidden layers, while UE104 may be assigned a final partition having the remaining hidden layers and the output layer. In an N-way partition (N>2), each of the M devices (M>1) is assigned a partition portion, and in the case of N>M, at least one device is assigned multiple partition portions.

[0017] As described below, in some embodiments, the UE 104 acts to initiate the partitioning of the neural network to perform the neural network operation and determines how to perform the partitioning, either unilaterally or as a result of negotiation with the network infrastructure 102. The UE 104's decision to partition the neural network operation between the UE 104 and one or more servers 112 (or other networked computing devices) can be based on a variety of factors, including the scope or extent of the operation to be performed, the local resources of the UE 104 available to perform the neural network operation, one or more execution requirements (e.g., latency, QoS, precision thresholds), network bandwidth, or other network parameters. As a general guideline in some embodiments, the portion of the neural network that the UE 104 intends to offload to one or more servers 112 is inversely proportional to the strictness of the local resources and / or execution requirements(s) available to the UE 104, all else equal. However, it should be noted that in other embodiments, the server 112 may instead initiate the partitioning of the neural network using techniques similar to those described below.

[0018] When UE104 proposes / decides on this particular partition, UE104 can use one of a variety of methods to notify the server 112 of the assigned partitioned neural network portion(s). In some embodiments, the server 112 (and UE104) can access a repository 118 containing representations 120 of the neural networks, and UE104 can notify the server 112 of those partitioned neural network portions(s) assigned to the server 112, along with other relevant details such as initial weights and how the partitioned neural network portions are implemented in the server 112. For example, each representation 120 may be identified by a corresponding identifier, which is then referenced by UE104 and the server 112. Representation 120 can refer to a specific embodiment of a type of neural network (e.g., an RNN having a specific number of layers, nodes, weights, etc.), and UE 104 and Server 112 can use additional information to implement the partition of this specific embodiment into a determined partition, or representation 120 can represent a predetermined partition configuration template such as an RNN having a specific number of layers, nodes, weights, etc., and a predetermined partition configuration template of these elements between multiple partitioned neural network parts. Although shown as a separate component (e.g., a separate server in the network infrastructure 102), in some embodiments, the repository 118 may be implemented entirely or partially in Server 112. In other embodiments, UE 104 may provide Server 112 with information specifying how Server 112 implements the corresponding partitioned neural network part, e.g., one or more data structures representing the structure of the partitioned neural network part, its input format, its output format, the weights or other parameters of different nodes in the structure, etc.

[0019] As an example of this partitioned neural network computation scheme, Figure 1 shows an exemplary partitioned neural network configuration 114 of a neural network 116, in which the neural network 116 is divided into two parts, partitioned neural network parts 122 and 124. Partitioned neural network part 122 is assigned the input layer and the first five hidden layers of the exemplary neural network 116 (hence called the "initial partitioned neural network part 122"), and partitioned neural network part 124 is assigned the last two hidden layers and the output layer of the exemplary neural network 116 (hence called the "final partitioned neural network part 124"). With this configuration, the neural network 116 is divided into the sequence initial neural network part 122 → final neural network part 124. In this example, it is assumed that UE104 negotiates this partition with server 112 based on the current local resources available to UE104 and the requirements for the operation to be performed (e.g., ML-assisted image classification, speech detection, etc.), and that as a result of the negotiation with server 112, it uses the techniques described below. Furthermore, in this embodiment, the initial partitioned neural network portion 122 is assigned to server 112, and the final partitioned neural network portion 124 is assigned to UE104. Thus, the initial partitioned neural network portion 122 is implemented in server ML module 126 in server 112, and the final partitioned neural network portion 124 is implemented in UE ML module 128 in UE104.

[0020] To initiate an ML-assisted operation using this exemplary configuration, the software application on the UE104 generates input data 130-1 for input to the neural network 116. For example, if the ML-assisted operation is image classification, the input data 130-1 may include, for example, images captured by the camera on the UE104. In another example, if the ML-assisted operation is speech detection and translation, the input data 130-1 may include, for example, audio captured and digitized by the microphone on the UE104. The radio frequency (RF) interface 132 on the UE104 transmits the input data (as input data 130-2) to the corresponding RF interface 134 on the BS110, and the RF interface 134 then transmits that input data (as input data 130-3) to the server 112. At server 112, input data 130-3 is provided as input to server ML module 126, and server ML module 126 processes input data 130-3 in the input layer and the first five hidden layers of neural network 116 according to its assigned initial partitioned neural network portion 122. Thus, the output 136 of the fifth hidden layer node serves as the intermediate result data 138-1 of the initial partitioned neural network portion 122 (and therefore the output of server ML module 126). In this way, server 112 sends the intermediate result data (as intermediate result data 138-2) to BS110, which then wirelessly sends the intermediate result data 138-2 to UE104.

[0021] The UE 104 provides the intermediate result data (as intermediate result data 138-3) as input to a UE ML module 128 that implements the final split neural network portion 124. Accordingly, the output 136 represented by the intermediate result data 138-3 is provided as input to the corresponding nodes of the sixth hidden layer of the neural network 116, and is processed by the sixth, seventh, and eighth hidden layers and the output layer to generate a final result output 140 representing the final result output by the neural network 116 when the intermediate result data 138-3 is given as input. The software application of the UE 104 that initiated the ML-assisted operation, and / or other software applications of the UE 104, may then perform one or more actions in response to the final result output 140. For example, if the ML-assisted operation is a speech detection and translation operation, the final result output 140 may represent a translated representation of the input speech, such that the software application may operate the UE 104 to output an audio version of the translated representation via a speaker of the UE 104, and / or display a text representation via a display of the UE 104.

[0022] Figure 1 shows a specific two-way split of a neural network 116, where the initial part is assigned to server 112 and the final part to UE 104. However, the splitting of any given neural network can be performed in various ways depending on the needs and context. For example, UE 104 can implement the initial part of the split neural network configuration, while server 112 implements the final part. Figure 2 illustrates this example, where the split neural network configuration 214 splits the neural network 216 into an initial split portion 202 implemented in UE ML module 228 of UE 104 and a final split portion 204 implemented in server ML module 226 of server 112. In this approach, the software application on UE 104 provides input data 230 as input to UE ML module 228, so that a first subset of the initial and hidden layers of the neural network 216 processes the input data 230. The output 236 of the last hidden layer in the initial partition 202 is provided as an intermediate result output 238 transmitted to the server 112 via the base station 110. The server 112 provides the intermediate result output 238 as input to the first hidden layer in the final partition 204, so that this hidden layer, subsequent hidden layers, and output layers process the intermediate result output 238 to produce a final result output 240. The final result output 240 may then be sent back to the UE 104 for use by one or more software applications in the UE, and / or the final result output 240 may be used by the server 112 or any other component of the network infrastructure 102 to perform one or more remote actions 242. For example, the neural network 116 can be used to generate augmented display information for an augmented reality (AR) display, and thus one or more remote actions 242 may be the display of the generated augmented display information on the AR display (one result of the final result output 240).As another example, the neural network 116 can be used to generate VR content, such as virtual reality (VR) display content or VR audio content (an example of the final result output 240), and one or more remote actions 242 may display or output the generated VR content. As yet another example, the neural network 116 can be used to improve or enhance cellular connectivity between the UE 104 and the BS 110 (for example, by controlling certain aspects of the UE's RF antenna array based on input sensor data). Thus, one or more remote actions 242 may include, for example, changing the operation of the RF antenna array based on the final result output 240.

[0023] Furthermore, as described above, in some embodiments, the split configuration may be an N-way split (N>2), and each of the M devices obtains at least one split neural network configuration. FIG. 3 shows an example of such a case, where the split neural network configuration 314 splits the neural network 316 into three split parts, that is, an initial split part 302 implemented in the UE ML module 328 of the UE 104, an intermediate split part 304 implemented in the server ML module 326 of the server 112, and a final split part 306 implemented in the UE ML module 328 of the UE 104 (or a second instance of the UE ML module of the UE 104). In this example, the software application of the UE 104 provides input data 330 as an input to the UE ML module 228 for processing by the initial split part 302. The output 336-1 of the last hidden layer in the initial split part 302 is provided as a first intermediate result output 338-1 that is transmitted to the server 112 via the base station 110. The server 112 provides the first intermediate result output 338-1 as an input to the first hidden layer in the intermediate split part 304, whereby this hidden layer and subsequent hidden layers of the intermediate split part 304 process the first intermediate result output 338-1. The output 336-2 of the last hidden layer in the intermediate split part 304 is provided as a second intermediate result output 338-2 that is transmitted to the UE 104 via the base station 110. Next, the UE 104 provides the second intermediate result output 338-2 as an input to the first hidden layer in the final split part 306, and this hidden layer, the subsequent hidden layers, and the output layer in the final split part 306 process this input to generate a final result output 340. The final result output 340 may then be processed by one or more software applications at the UE 104, and in response, one or more actions are taken at the UE 104.

[0024] Figure 4 shows exemplary hardware configurations of the UE104 (as a typical client device) in several embodiments. Note that the illustrated hardware configurations represent the processing and communication components most directly related to the neural network-based processes described herein, and omit certain components that are well known to be frequently implemented in such electronic devices, such as displays, peripherals, and external power supplies.

[0025] In the illustrated configuration, the UE104 includes an RF interface 402, which has one or more antennas 404 and one or more modems, which support one or more radio access technologies (RATs), such as a 3GPP 4th Generation Long-Term Evolution (4G LTE) compliant RAT, a 3GPP 5th Generation New Radio (5G NR) compliant RAT, an IEEE 802.11 compliant RAT, etc. The UE104 further includes one or more processors 406, a set of sensors 408, a user interface (UI) 410, and one or more batteries 412 or other power sources. The one or more processors 406 may include, for example, one or more central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), or other application-specific integrated circuits (ASICs). For example, the processor 406 may include an application processor (AP) used by the UE 104 to run the operating system and various user-level software applications, as well as one or more processors used by the baseband processor of the modem or RF interface 402. The set of sensors 408 may include, for example, satellite positioning sensors such as Global Positioning System (GPS) sensors, Global Navigation Satellite System (GNSS) sensors, inertiaThis may include measuring unit (IMU) sensors, visual and odor sensors, accelerometers, gyroscopes, barometers, altimeters, tilt sensors or other inclinometers, and ultra-wideband (UWB) based sensors. Other examples of sensor types in sensor set 408 may include sensors to determine the current operating status of UE104, such as battery level sensors, temperature sensors, and screen mode sensors. UI410 includes components for interface with the user, such as a display, keyboard or touch panel, buttons, microphone, and speaker.

[0026] UE104 further includes one or more computer-readable media 414, which include any of the various media used by electronic devices to store data and / or executable instructions, such as random access memory (RAM), read-only memory (ROM), cache, flash memory, solid-state drive (SSD), or other mass storage devices. For ease of explanation and brevity, computer-readable media 414 are referred to herein as “memory 414” considering that system memory or other memory is frequently used to store data and instructions for execution by processor 406, but references to “memory 414” are understood to apply equally to other types of storage media unless otherwise noted. One or more memories 414 of UE104 are used to store one or more sets of executable software instructions and related data for operating one or more processors 406 and other components of UE104, and perform various functions described herein and attributed to UE104. The set of executable software instructions includes, for example, an operating system (OS) 416 and various drivers (not shown), as well as one or more user-level software applications 418, which operate the UE104 hardware via the OS 416 or otherwise interact with it.

[0027] The set of executable software instructions further includes one or more of the Neural Network Management Module 420 and the Partitioned Computation Management Module 422. The Neural Network Management Module 420 implements one or more ML modules (e.g., ML module 128, Figure 1) that utilize one or more neural networks (or parts thereof) in the UE104, as described in detail below. The Partitioned Computation Management Module 422 operates to monitor the status of local resources in the UE104, such as battery status, available memory status, available processor resources, and network status, and based on these local resource statuses (and / or other considerations), it determines whether to implement the neural network as a locally-only neural network (i.e., entirely in the UE104) or as a partitioned neural network configuration to use the neural network to perform one or more operations. Furthermore, if a partitioned neural network configuration is adopted, the partitioning computation management module 422 can operate to manage negotiations regarding the implementation of partitioned portions by one or more servers 112 of the network infrastructure 102, and can coordinate with the neural network management module 420 to implement the partitioned portion(s) assigned to the UE 104 in the negotiated partitioned neural network configuration.

[0028] To facilitate the operation of the UE104 as described herein, one or more memories 414 of the UE104 may further store data related to these operations. This data may include, for example, one or more neural network architecture configurations 424, as well as device data (not shown) including, for example, user data, multimedia data, beamforming codebooks, software application configuration information, etc. Each neural network architecture configuration 424 includes one or more data structures containing data and other information representing the corresponding architecture and / or parameter configuration used by the neural network management module 420 to form the corresponding neural network or a segment of the UE104. The information contained in the neural network architecture configuration 424 includes parameters that specify, for example, the total connected layer neural network architecture, the convolutional layer neural network architecture, the recurrent neural network layers, the number of connected hidden neural network layers, the input layer architecture, the output layer architecture, the number of nodes used by the neural network, the coefficients used by the neural network (such as weights and biases), kernel parameters, the number of filters used by the neural network, the stride / pooling configuration used by the neural network, the activation function of each neural network layer, the interconnections between neural network layers, and the neural network layers to skip. Thus, the neural network architecture configuration 424 contains any combination of neural network forming components (e.g., architecture and / or parameter configurations) that can be used to create a neural network architecture configuration (e.g., a combination of one or more neural network forming components) that defines and / or forms a DNN or other neural network.

[0029] Figure 5 shows exemplary hardware configurations of server 112 (as a typical networked computing resource) in several embodiments. Note that the illustrated hardware configurations represent the processing and communication components most directly related to the neural network-based processes of the embodiments described herein, and certain components that are well understood to be frequently implemented in such electronic devices have been omitted. Furthermore, note that while the figures shown represent an embodiment as a single server, the functions, and therefore the hardware components, may instead be distributed across multiple servers or other networked computing devices in a manner that performs the functions described herein. Thus, references to the functions of a single server 112 may also apply to equivalent functions in multiple servers 112 or other networked computing devices unless otherwise specified.

[0030] In the illustrated configuration, the server 112 includes a network interface 502 for wired and / or wireless connections to the BS 110 and other components of the network infrastructure 102. The server 112 may further include one or more processors 506, such as one or more CPUs, GPUs, TPUs, or other ASICs. The server 112 further includes one or more computer-readable media 508, which include any of the various media used by electronic devices to store data and / or executable instructions, such as RAM, ROM, cache, flash memory, SSD, or other mass storage devices. Similar to the memory 414 of UE 104, for the sake of ease and brevity, the computer-readable media 508 is referred to herein as “memory 508” considering that it frequently uses system memory or other memory to store data and instructions for execution by the processors 506, but references to “memory 508” are understood to apply equally to other types of storage media unless otherwise noted.

[0031] One or more memories 508 of server 112 are used to store one or more sets of executable software instructions and related data for operating one or more processors 506 and other components of server 112, and perform various functions described herein, which are attributed to server 112 individually or as a set of server 112. Examples of sets of executable software instructions include the OS 510 and various drivers (not shown), as well as various software applications. The sets of executable software instructions further include one or more of the neural network management module 512 and partitioning management module 514. Similar to the neural network management module 420 and partitioning management module 422 of UE 104, these modules 512, 514 operate to implement a neural network or its partition(s) in server 112. Specifically, the neural network management module 512 implements one or more ML modules (e.g., ML module 126, Figure 1) that utilize one or more neural networks (or their partitions) for the server 112, while the partition calculation management module 514 negotiates with the UE 104 regarding the implementation of the partitions on the server 112 and operates to coordinate with the neural network management module 512 in order to implement the partition(s) assigned to the server 112 in the negotiated partitioned neural network configuration.

[0032] One or more memories 508 further store various information, such as one or more neural network architecture configurations 524 representing trained neural network architecture configurations that can be used by the ML module of the server 112 (e.g., ML module 126, Figure 1). Thus, similar to the neural network architecture configuration 424 in Figure 4, each neural network architecture configuration 524 includes one or more data structures containing data and other information that represent the corresponding architecture and / or parameter configuration used by the neural network management module 512 of the server 112 to form the corresponding part(s) or whole of a neural network.

[0033] Figure 6 shows exemplary machine learning (ML) modules 600 for implementing a neural network or its partitioned portion(s) according to several embodiments. As described herein, both UE104 and server 112 implement one or more ML modules to implement one or more partitioned neural network portions, respectively. Thus, ML module 600 represents an exemplary module for implementing one or more of these partitioned neural network portions (or entire neural networks).

[0034] In the illustrated example, the ML module 600 implements at least a portion 602 of a deep neural network (DNN) model having a group of connected nodes (e.g., neurons and / or perceptrons) organized into one or more layers. The nodes between layers can be configured in various ways, such as a partially connected configuration where a first subset of nodes in a first layer is connected to a second subset of nodes in a second layer, or a fully connected configuration where each node in the first layer is connected to each node in the second layer. Neurons process input data to produce continuous output values, such as any real number between 0 and 1. In some cases, the output value indicates how close the input data is to a desired category. Perceptrons perform linear classification, such as binary classification, on the input data. Nodes, whether neurons or perceptrons, can use a variety of algorithms to generate output information based on adaptive learning. Using part 602, ML module 600 performs various different types of analysis, including single linear regression, multiple linear regression, logistic regression, stepwise regression, binary classification, multi-class classification, multivariate adaptive regression splines, local estimation scatter plot smoothing, and more.

[0035] When part 602 includes the entire DNN model, it includes two or more layers: an input layer, one or more hidden layers, and an output layer. When part 602 includes a segment of the DNN model, the layers included in part 602 depend on the position of the segment in the sequence of segments that together constitute the DNN model. If part 602 is the initial segment, it includes an initial layer and one or more adjacent hidden layers immediately following the initial layer. If part 602 is the final segment, it includes an output layer and one or more adjacent hidden layers immediately preceding the output layer. If part 602 is an intermediate segment, it includes one or more adjacent hidden layers.

[0036] For the sake of clarity, the illustrated portion 602 of the DNN model includes three layers 604, 606, and 608, which may be implemented, for example, as an input layer and two adjacent hidden layers following the input layer in an initial partitioned portion embodiment, or as three adjacent hidden layers in an intermediate partitioned portion embodiment, or as an output layer and two adjacent hidden layers preceding the output layer in a final partitioned portion embodiment. However, in actual embodiments, the partitions implemented by the ML module 600 typically have more layers. Each layer has any number of nodes, and the number of nodes between layers may be the same or different. That is, the input layer may have the same and / or different number of nodes as the output layer, the output layer may have the same and / or different number of nodes as one or more hidden layers, and so on. For example, node 610 corresponds to one of several nodes contained in the input layer (which may be represented by layer 604), and the node performs a separate and independent computation. As will be further described, the node receives input data and processes the input data using one or more algorithms to generate output data. Typically, the algorithm includes weights and / or coefficients that change based on adaptive learning. Therefore, the weights and / or coefficients reflect the information learned by the neural network. Each node can, in some cases, decide whether to pass the processed input data to one or more subsequent nodes. For example, after processing the input data, node 610 can decide whether to pass the processed input data to one or both of nodes 612 and 614 of an adjacent hidden layer (layer 608 in this example). Alternatively or additionally, node 610 passes the processed input data to nodes based on the layer-connected architecture. This process can be repeated across multiple layers until part 602 produces an output using a node (e.g., node 616) in the final layer (layer 608) of part 602. To facilitate the input of incoming data and the output of generated output data, the ML module 600 may further include an input interface 618 and an output interface 620.The input interface 618 receives input data 622 and operates to distribute the individual data contained in input data 622 to the corresponding nodes of the first layer 604 according to some predetermined format. Similarly, the output interface 620 receives the individual outputs of the nodes of the final layer 608 and operates to provide them as output data 624 according to some predetermined format. The format used for the input interface 618 and the output interface 620 depends on the neural network used and the layers of the neural network implemented in section 602. For example, the initial section. against The input data 622 represents the initial data that is input for processing by the neural network, and therefore the input interface 618 operates to distribute the initial data to the nodes of the input layer (layer 604) accordingly. However, in the case of the intermediate or final partition, the input data 622 represents the output from the upstream hidden layer, and therefore the input data 622 represents the intermediate result data, and the input interface 618 is configured to distribute the individual data of the intermediate result data as input to the nodes of the first hidden layer (layer 604) according to the neural network architecture. Similarly, for the initial or intermediate partition, the output data 624 is the individual output of the node of the last hidden layer (layer 608) in the ML module 600, and therefore represents the intermediate result data, but for the final partition, the output data 624 is the final result output provided by the output layer (layer 608) of the neural network.

[0037] Neural networks can also use various architectures that determine which nodes in the neural network are connected, how data is advanced and / or held within the neural network, what weights and coefficients are used to process the input data, and how the data is processed. These various elements collectively describe neural network architecture configurations, such as the neural network architecture configurations briefly described above. For example, recurrent neural networks, such as Long Short-Term Memory (LSTM) neural networks, form cycles between node connections to hold information from the previous part of the input data sequence. The recurrent neural network then uses the held information for the subsequent part of the input data sequence. Another example is a feedforward neural network, which advances connections by transferring information without forming cycles to hold information. Although described in the context of node connections, it should be understood that neural network architecture configurations can include various parameter configurations that affect how other neural networks process input data.

[0038] The neural network architecture configuration of a neural network can be characterized by various architectures and / or parameter configurations. To illustrate this, Part 602 considers an example in which part or all of a CNN is implemented. In general, a convolutional neural network corresponds to a type of DNN in which layers process and filter input data using convolutional operations. Thus, a CNN architecture configuration can be characterized, for example, by pooling parameters(s), kernel parameters(s), weights, and / or layer parameters(s).

[0039] Pooling parameters correspond to parameters that specify pooling layers within a convolutional neural network, reducing the dimensionality of the input data. For example, a pooling layer can combine the outputs of nodes in a first layer into the inputs of nodes in a second layer. Alternatively or additionally, pooling parameters specify how and where the neural network pools data within the layers of data processing. For example, a pooling parameter indicating "maximum pooling" configures the neural network to pool data by selecting the maximum value from a group of data generated by the nodes of the first layer, and using that maximum value as the input to a single node in the second layer. A pooling parameter indicating "average pooling" configures the neural network to generate an average value from a group of data generated by the nodes of the first layer, and using that average value as the input to a single node in the second layer.

[0040] Kernel parameters indicate the filter size (e.g., width and height) used to process the input data. Alternatively or additionally, kernel parameters specify the type of kernel method used to filter and process the input data. Support vector machines, for example, correspond to kernel methods that use regression analysis to identify and / or classify data. Other types of kernel methods include Gaussian processes, canonical correlation analysis, and spectral clustering methods. Thus, kernel parameters can indicate the filter size and / or the type of kernel method to apply to the neural network. Weight parameters specify the weights and biases used by the algorithm within the node to classify the input data. In some embodiments, the weights and biases are trained parameter configurations, such as parameter configurations generated from training data. Layer parameters specify the layer connections and / or layer types, such as fully connected layer types, which indicate that all nodes of a first layer (e.g., output layer 608) are connected to all nodes of a second layer (e.g., hidden layer 606); partially connected layer types, which indicate which nodes of the first layer are disconnected from the second layer; and activated layer types, which indicate which filters and / or layers are activated within the neural network. Alternatively or additionally, layer parameters specify the type of node layer, such as normalization layer type, convolutional layer type, and pooling layer type.

[0041] While pooling parameters, kernel parameters, weight parameters, and layer parameters are discussed in this context, it should be noted that other parameter settings can be used to form a DNN that conforms to the guidelines provided herein. Thus, the neural network architecture configuration can include any appropriate type of configuration parameters applicable to the DNN that influence how the DNN processes input data and produces output data.

[0042] Referring here to Figures 7-10, a method 700 for implementing a partitioned neural network configuration of a neural network used to perform ML operations on behalf of a software application or other process on a UE or other client device is shown according to several embodiments. For illustrative purposes, the method 700 and accompanying examples are described in an exemplary context of a wireless communication network 100 comprising a UE 104 acting as a client device and a server 112 representing one or more co-networked computing devices, but the method is not limited to the context of this particular embodiment. Furthermore, the ladder diagram 1000 in Figure 10 is referenced below inline with the description of the method 700 in Figures 7-9 as an example of the operation of the method 700 for an exemplary two-way partitioned neural network configuration to facilitate understanding. Furthermore, it should be noted that the order of operations described with reference to the method 700 is for illustrative purposes only, and operations may be performed in a different order, and one or more operations may be omitted, or one or more additional operations may be included in the method shown.

[0043] As described above, in the embodiment, a software application of UE104 (e.g., a user-level application 418 or OS416 or other kernel / driver-level process) utilizes the neural network to perform corresponding actions, and the results are used by that software application or other software applications to perform one or more actions. If UE104 has sufficient resources to implement the neural network entirely locally while achieving corresponding goals (e.g., a maximum latency goal or a battery consumption goal), UE104 may implement the entire neural network in UE104's ML module and use this ML module, thus configured, to perform the actions. However, if UE104 cannot implement the neural network entirely locally while satisfying one or more specified goals, UE104 may instead attempt to implement the neural network as a partitioned neural network configuration that leverages the computing resources of one or more servers 112 of the network infrastructure 102.

[0044] Therefore, Method 700 begins in Block 702, in which a software application on UE104 initiates the execution of an ML operation (Block 1002, Figure 10) that utilizes an identified neural network to generate output results based on input data provided or referenced by the software application. Using the example above, this operation could be an image classification operation using a specific CNN, where the input to the CNN is image data representing the image to be classified. Initiating an ML operation can include, for example, a software application issuing a call to the application programming interface (API) of OS416 to support such operations. Simultaneously, the partitioning management module 422 monitors the status of the UE104's local resources, including available memory, processor utilization, remaining battery capacity, current device usage, and other indications of current or future local resource availability (Block 1004, Figure 10). Therefore, in response to the initiation of an ML operation, in Block 704, the partitioning management module 422 determines the current conditions of UE104 based on these monitoring operations. Furthermore, the partitioned computation management module 422 identifies arbitrary execution requirements related to ML operation, such as execution latency limits, execution power consumption limits, and computation resource consumption limits.

[0045] Next, in block 706, the partitioning management module 422 determines whether a local-only implementation or partitioned configuration of the identified CNN is more suitable for the determined current conditions and any execution requirements. For example, if there are sufficient local power and computing resources available at UE104, a relatively long time limit for generating the final result output, and the bandwidth of the wireless network is limited, the partitioning management module 422 may decide to implement the neural network entirely locally and thus run the ML operation only at UE104. However, if there are insufficient computing or power resources available at UE104, a strong wireless connection with BS110, and a relatively short time limit, the partitioning management module 422 may decide to implement the neural network as a partitioned configuration in which one or more parts of the neural network are assigned to UE104 and one or more other parts of the neural network are assigned to server 112 (or multiple servers 112). This decision can be made in any of several ways. For example, the decision may be implemented algorithmically (e.g., via a weighted sum equation), via a lookup table (LUT), or via other less resource-intensive trained neural networks. For instance, UE104 may model the performance of a proposed partition configuration to determine whether it satisfies the UE's power consumption limits while also meeting QoS (e.g., latency) requirements. If it does, UE104 may select the proposed partition configuration; otherwise, UE104 may select another proposed partition configuration for similar modeling.

[0046] If the partitioning management module 422 determines that a local-only implementation of the neural network is appropriate or sufficient, in block 708, the partitioning management module 422 instructs the neural network management module 420 to implement an ML module configured to implement the entire architectural configuration of the identified neural network, and UE104 uses this ML module to perform ML operations.

[0047] Returning to block 706, if the partitioning management module 422 determines that the partition configuration of the neural network is appropriate (block 1006, Figure 10), then in block 710, the partitioning management module 422 performs a negotiation process and, given the respective resource constraints of UE 104 and server(s) 112, determines a partition of the neural network between UE 104 and server(s) 112 (or multiple servers 112) that is acceptable to UE 104 and server(s) 112.

[0048] Briefly referring to Figure 8, exemplary embodiments of the negotiation process in block 710 are shown according to several embodiments. In this exemplary approach, the negotiation process in block 802 follows a decision to implement a partition configuration using a partition calculation management module 422 that determines or identifies a specific neural network architecture to be implemented to perform ML operations. This includes determining the type of neural network (e.g., CNN, RNN, ANN, GAN, etc.), as well as parameters of the neural network type such as the number of layers, the number of nodes in each layer, the operations performed at each node, the weights and other parameters of each node, and the connections between nodes. Furthermore, the following describes the negotiation process for a client-initiated partition, although in other embodiments, the partition may be initiated by the server 112, in which case a similar process may be carried out, but it should be noted that the roles of the server 112 and UE 104 are reversed with respect to the initiator and responder.

[0049] Once the neural network architecture is identified, in block 804, the partitioning computation management module 422 then determines a proposed partition configuration for the neural network architecture based on the current UE conditions, execution requirements, etc. This proposed partition configuration, as well as the decision of whether or not to partition the neural network, can be determined or selected algorithmically, such as via a LUT or other selection structure, or via a smaller neural network. In some embodiments, several predetermined partition options are available, and the partitioning computation management module 422 selects one of the predetermined partition options as the proposed partition configuration. For example, one partition option may be a two-way partition that evenly divides the hidden layers between the server 112 and the UE 104, another partition option may be a two-way partition that allocates a larger proportion of the hidden layers to the server 112 than to the UE 104. Yet another partition option may be a three-way partition that allocates initial and final partition portions with a small proportion of the hidden layers to the UE 104, and an intermediate partition portion with the majority of the hidden layers to the server 112. In this example, UE104 can select the partition option that best suits the current UE's conditions while satisfying the stated execution requirements. In other embodiments, the partition computation management module 422 dynamically determines the proposed partition, for example, by allocating a number of layers to the portion allocated to the server 112 in inverse proportion to the available computational resources of UE104, using a simple example. Once the proposed partition is identified, UE104 then sends a partition request 1008 (Figure 10) to the server 112 via BS110. The partition request 1008 may include a description or identifier of the proposed partition (e.g., a data structure that identifies relevant parameters that adequately describe the partition portion that UE104 is proposing for the server 112 to implement, or an identifier to a predefined partition option available from the repository 118 as a representation of a neural network 120). The partition request 1008 may also include information relating to the requested partition proposal, such as the current conditions of UE104 and / or the execution requirements that led to the particular proposed partition.Additionally or alternatively, server 112 may obtain some or all of the current conditions of UE 104 from the latest RRC UE capability information message sent to BS110 by UE 104 in response to a Radio Resource Control (RRC) UE capability query message from BS110.

[0050] In block 806, the partition calculation management module 514 of server 112 (or BS110, or other network component acting as an intermediary for server 112) evaluates the proposed partition represented by partition request 1008 to decide whether to accept the proposed partition or to counter-propose a modified proposed partition. When deciding whether to accept the proposed partition, the partition calculation management module 514 may consider the current resource constraints and stated execution requirements of the server itself. For example, if server 112 can dedicate the resources necessary to implement the partitioned neural network portion allocated to server 112 based on the proposal and can satisfy the execution requirements stated in the proposed partition, the partition calculation management module 514 may accept the proposed partition and indicate this acceptance in block 808 by sending a partition calculation permission message 1010 (Figure 10) to UE104 via BS110.

[0051] Returning to block 806, if the partitioning management module 514 determines that server 112 cannot allocate the necessary computing resources to implement the proposed partition, or that the proposed partition does not meet the stated performance requirements, then in block 810, the partitioning management module 514 may determine a partitioning desuggestion that proposes a modified partition configuration of the neural network. For example, if server 112 cannot dedicate sufficient resources to the original proposed partition, the partitioning management module 514 may desuggest a desuggestion partition that allocates fewer layers of the neural network architecture to server 112. As another example, if server 112 can allocate sufficient resources, but the partitioning management module 514 determines that the original proposed partition does not meet latency requirements or other performance requirements, then the partitioning management module 514 may determine a desuggestion partition that is more likely to meet latency requirements or other performance requirements. To make it so A modified partition may be counter-proposed. Next, the server 112 sends a representation of the partition counter-proposal to the UE 104. In block 812, the partition calculation management module 422 of the UE 104 evaluates the counter-proposal, and if it is acceptable, in block 814 sends an acceptance message to the server 112, to which the server 112 responds by issuing a partition calculation permission message 1010. If the counter-proposal is not acceptable, in block 816, the UE 104 may terminate the negotiation and attempt to perform the ML operation entirely locally (i.e., by implementing the entire neural network in the ML module of the UE 104), or fall back by returning a No Action (NOP) or an error message to the startup software application to indicate that the ML operation cannot be performed under the current conditions. Alternatively, in some embodiments, the negotiation continues until a mutually acceptable partition is identified or until a threshold number of counter-proposals for the partition are sent. reverse One or more rounds of the proposed procedure may be performed between UE104 and server 112.

[0052] Returning to Figure 7, based on the partition configuration proposal agreed upon by Server 112 and UE 104, Server 112 and UE 104 proceed to implement their respective partition(s). However, to do so, Server 112 and UE 104 require the neural network architecture details (e.g., layers, nodes, connections, weights, and other parameters) for each partition to be implemented as a corresponding ML module. In some examples, the negotiation process provides the distribution of this information. For example, when the negotiation process results in mutual agreement to use a given partitioned neural network configuration from, for example, Repository 118, the partition implementation details may be explicitly included in the negotiation messaging, or they may subsequently be accessible from Repository 118 by either or both Server 112 or UE 104 once the agreed-upon partitioned neural network configuration is determined. Alternatively, if the partitioned neural network configuration is determined ad hoc by UE104, or if a counter-proposal partition from server 112 is accepted by UE104, the implementation details of the partition(s) to be implemented by server 112 and / or UE104 may be included in the negotiation messaging (e.g., in partition request 1008 or permission message 1010, respectively). However, in some embodiments, the negotiation may include the negotiation of the overall partition but not specific details such as weights or other parameters of a particular node or layer. In such cases, as represented by block 712, UE104 sends one or more configuration messages to server 112 via BS110, which include the implementation details of the partition(s) of the neural network (hereinafter, "server-side partition(s)") that can be implemented by server 112 according to the negotiated partition configuration.With the implementation details in place, in block 714, UE104 is configured to implement one or more ML modules (e.g., UE ML module 128, Figure 1) to implement one or more partitions (hereinafter, "UE-side partitions") assigned to UE104, and in block 716, server 112 is configured to implement one or more ML modules (e.g., server ML module 126, Figure 1) to implement one or more server-side partitions. With UE104 and server 112 configured to implement their respective partitioned neural network portions of the agreed partitioned neural network configuration for the neural network, in block 718, server 112 and UE104 work together to perform partitioned computation ML operations via the neural network distributed between server 112 and UE104.

[0053] Figure 9 shows exemplary embodiments of the partitioned computation ML operation execution process for block 718 according to several embodiments. As described above, a partitioned neural network configuration can include a sequence of two or more partitioned neural network portions, and the initial partition portion of the sequence may be assigned to either the UE 104 or the server 112, depending on the implementation, as is the case with the final partition portion of the sequence. Therefore, for the purpose of describing the exemplary implementation of the partitioned computation ML operation execution process in Figure 9 below, one of the UE 104 or server 112 that implements the initial partition portion in a given partitioned configuration will be referred to herein as the “first device”, and the other of the UE 104 or server 112 that implements the next partition portion following the initial partition portion of the sequence will be referred to as the “second device”, referring to the order in which the devices join in the sequence of partitioned configurations. In the example of Figure 10 showing this embodiment, the server 112 is the first device, the UE 104 is the second device, the partitioned configuration is a two-way partitioned configuration, the server 112 implements the initial partition portion, and the UE 104 implements the final partition portion.

[0054] A typical neural network receives input data, processes the input data in various layers or other structures of the neural network, and then provides a final output as a result of this processing of the input data. Depending on the purpose of the ML operation being performed and the origin of the input data for the neural network implemented as a partitioned configuration by UE104 and server 112, the input data may be supplied by UE104, server 112, other components, or a combination thereof. Regardless of the source, in order to start the execution of the ML operation, the input data is provided to a first device implementing the initial partitioned portion. Thus, in block 902, one or more sources of input data transmit or otherwise provide first device access to the input data used for the ML operation. For example, in a partitioned configuration where server 112 implements the initial partitioned portion (i.e., server 112 is the first device and UE104 is the second device), but the ML operation requires input data acquired by UE104 (e.g., images captured by UE104), UE104 may transmit image data to server 112 via BS110. As another example, the image data could be an image supplied from a web page (for example, the ML operation is an image matching search), and therefore, instead of directly providing the image data, UE104 sends a web address or other pointer to the image and the web page, and server 112 uses the pointer to retrieve the image data from the web page.

[0055] In a cellular network, or other networks such as the wireless communication network 100 illustrated in Figure 1, it will be understood that BS110 and UE104 may use one or more resource allocation schemes to facilitate the efficient and timely transmission of data uplink or downlink between UE104 and BS110. Such schemes may be relevant when UE104 provides some or all of the input data to server 112 for input to the initial layers of a neural network in server 112, especially when there are maximum latency requirements imposed on the ML operation. Therefore, following the partition calculation negotiation process, before sending input data from UE104 to server 112 via BS110, UE104 may notify BS110 of the need for its uplink transmission for timely wireless transmission of input data from UE104 to BS110. To illustrate, referring briefly to Figure 10, following the receipt of the partition calculation permission message 1010, in block 1012, UE 104 identifies the input data to be provided by UE 104 to server 112, determines the uplink (UL) data requirements for transmitting this input data, and buffers the input data in one or more UL buffers. In block 1012, UE 104 sends a UL transmission request for input data to BS 110 in the form of a UL buffer status report (BSR) message 1014, informing BS 110 of the amount of data in its UL buffer for UL transmission. In response to the UL BSR message 1014, BS 110 sends a UL permission to UE 104, for example, in the form of a UL downlink control information (DCI) message 1016, which informs UE 104 of the parameters that UE 104 will use when transmitting the data in its UL buffer (including input data) to BS 110, such as physical layer resource allocation and power control commands. In block 1018, UE104 wirelessly transmits the input data as UL data to BS110 using the parameters indicated in UL DCI message 1016, and BS110 forwards the input data to server 112.

[0056] Referring back to Figure 9, upon providing input data to the first device, in block 904, the first device receives the input data and processes it in the initial division section implemented in the ML module of the first device (block 1020, Figure 10) to generate output data representing the intermediate result of the ML operation. In block 906, the intermediate result (block 1022, Figure 10) is sent to the second device. In block 908, the second device receives the intermediate result and provides it as input data to the next division section implemented in the ML module of the second device, and the ML module processes the input data (block 1024, Figure 10) to generate output data. If, as indicated by decision block 910, this next division is the final division in the sequence of divisions, the output data represents the final result of the neural network, and therefore the final result of the ML operation. Thus, one or both of the final results are sent to the first device in block 912, so that the first device can take one or more actions in response to the final results, or the second device can take one or more actions in block 914 in response to the final results (block 1026, Figure 10). However, if this next division is an intermediate division in the sequence of divisions, the output data represents an intermediate result, and therefore, in block 916, the intermediate result is sent to a first device (or a third device if three or more devices are implemented in the division neural network configuration), thereby providing these intermediate results as input data to the next division configuration in the sequence to generate output data, which may then be the final result of the neural network if this next division configuration is the last division configuration in the sequence and is processed accordingly, or if there is still one or more additional divisions remaining in the sequence, the output data may be provided to the next device to be used as input data for the next division configuration in the sequence until the final division configuration in the sequence outputs the final result of the neural network, and so on.

[0057] Embodiments of this disclosure may also be better understood by considering the following non-limiting embodiments.

[0058] Example 1: A method performed on a computer, comprising, in a first device, dividing a neural network into partitioned neural network configurations for at least the first device and a second device based on one or more current sets of conditions of the first device, wherein the partitioned neural network configurations specify the distribution of elements of the neural network architecture of the neural network between at least a first neural network portion of the first device and a second neural network portion of the second device, and the method further comprises implementing the first neural network portion in the first device, transmitting a representation of the second neural network portion to the second device, and processing first data in the first neural network portion to produce a first output.

[0059] Example 2: The method according to Example 1, wherein the first data includes input data for the neural network, the first output includes intermediate result data, and the method further includes transmitting the intermediate result data to the second device for processing by the second neural network portion.

[0060] Example 3: The method according to Example 2, further comprising receiving the final result output of the neural network from the second device and performing at least one action on the first device in response to the final result output.

[0061] Example 4: The method according to Example 1, further comprising receiving the first data from the second device, wherein the first data includes intermediate result data generated by the second neural network portion in the second device, and the first output is the final result output of the neural network.

[0062] Example 5: The method according to Example 4, further comprising performing at least one action on the first device based on the final result output of the neural network.

[0063] Example 6: The method according to Example 4, further comprising transmitting input data from the first device to the second device for processing by the second neural network portion in the second device.

[0064] Example 7: The method according to any one of Examples 1 to 6, wherein the set of one or more current conditions includes at least one of battery conditions, network conditions, thermal conditions, computing resource availability, or specified quality of service conditions.

[0065] Example 8: The method of any one of Examples 1 to 7, further comprising determining latency conditions for performing neural network operations using the neural network, and determining the partitioned neural network configuration of at least the first device and the second device based on the latency conditions.

[0066] Example 9: Based on one or more sets of current conditions or at least one of the latency conditions, First The method according to Embodiment 8, further comprising determining whether to fully perform the neural network operation on the device or to perform the neural network operation by using a partitioned configuration of the neural network, and determining the partitioned neural network configurations for at least the first device and the second device in response to the decision to use a partitioned configuration of the neural network.

[0067] Example 10: The method according to any one of Examples 1 to 9, further comprising sending a partitioning request to the second device, the partitioning request proposing the partitioned neural network configuration, and the first neural network portion being implemented in the first device in response to receiving permission for the partitioning request from the second device.

[0068] Example 11: Determining the partitioned neural network configuration for at least the first and second devices comprises sending a partition calculation request to the second device, the partition calculation request proposing a different partitioned neural network configuration based on one or more sets of current conditions, and the determination further comprises receiving a partition calculation inverse proposal from the second device, and the partition calculation Countermeasure The proposal is the method according to any one of Examples 1 to 9, wherein the proposal proposes the partitioned neural network configuration, and the first neural network portion is implemented in the first device in response to accepting the partitioned computation inverse proposal based on the one or more set of current conditions.

[0069] Example 12: The method according to one of Examples 10 or 11, wherein the partitioning calculation request further indicates latency conditions for performing neural network operations using the partitioned neural network configuration.

[0070] Example 13: The method according to any one of Examples 1 to 12, wherein the representation of the second neural network portion includes at least one of data describing the neural network architecture of the second neural network portion, or an identifier for a predetermined partitioned neural network configuration of the neural network represented in a repository accessible to the second device.

[0071] Example 14: A method performed on a computer, comprising: a second device receiving instructions from a first device for a partitioned neural network configuration of at least the first device and the second device, wherein the partitioned neural network configuration specifies the distribution of elements of a neural network architecture of a neural network between at least a first neural network portion of the first device and a second neural network portion of the second device, and the method further comprises: implementing the second neural network portion on the second device; receiving first data from the first device; and processing the first data in the second neural network portion to produce a first output.

[0072] Example 15: The method according to Example 14, wherein the first data includes input data for the neural network, the first output includes intermediate result data, and the method further includes transmitting the intermediate result data to the first device for processing in the first neural network portion.

[0073] Example 16: The method according to Example 14, wherein the first data includes intermediate result data generated by the first neural network portion in the first device, and the first output is the final result output of the neural network.

[0074] Example 17: The method according to Example 16, further comprising at least one of the following: performing at least one action on the second device in response to the final result output, or transmitting the final result output to the first device.

[0075] Example 18: The method according to any one of Examples 14 to 17, wherein receiving the instruction for a partitioned neural network configuration includes receiving a partition calculation request proposing the partitioned neural network configuration, the method further includes determining whether to allow the partition calculation request, and the second neural network portion is implemented in the second device in response to the decision to allow the partition calculation request.

[0076] Example 19: The method of Example 18, further comprising receiving instructions for latency conditions for performing neural network operations using the neural network from the first device, and determining whether to allow the partition calculation request, based on the latency conditions.

[0077] Example 20: The method according to any one of Examples 1 to 19, wherein the second neural network portion is implemented in the second device in response to receiving a representation of the second neural network portion from the first device, and the representation of the second neural network portion includes at least one of data representing the neural network architecture of the second neural network portion or an identifier for a predetermined partitioned neural network configuration of the neural network represented in a repository accessible to the second device.

[0078] Example 21: The method according to any one of Examples 1 to 20, wherein the first device is a user device, and the second device is one or more servers connected to a network that can be wirelessly accessed by the user device.

[0079] Example 22: The method according to any one of Examples 1 to 21, wherein the neural network is a deep neural network (DNN) model.

[0080] Example 23: The partitioned neural network configuration is the method according to any one of Examples 1 to 22, wherein a first set of continuous layers of the neural network is distributed to the first neural network portion, a second set of continuous layers of the neural network is distributed to the second neural network portion, and the second set is adjacent to the first set.

[0081] Example 24: The method according to any one of Examples 1 to 22, wherein the partitioned neural network configuration includes a two-way partition comprising an initial neural network portion and a final neural network portion, the initial neural network portion being assigned to one of the first device or the second device, and the final neural network portion being assigned to the other of the first device or the second device.

[0082] Example 25: The method according to any one of Examples 1 to 22, wherein the partitioned neural network configuration includes a three-way partition comprising an initial neural network portion, an intermediate neural network portion, and a final neural network portion, the initial neural network portion and the final neural network portion are assigned to either the first device or the second device, and the intermediate neural network portion is assigned to the other of the first device or the second device.

[0083] Example 26: A device comprising a network interface, at least one processor coupled to the network interface, and a memory for storing executable instructions, wherein the executable instructions are configured to operate the at least one processor to perform the method described in any of Examples 1 to 25.

[0084] Example 27: The device according to Example 26, wherein the first device is a user device on a cellular network, and the second device is one or more servers on a network that the user device can access wirelessly.

[0085] Example 28: The device is the first device, as described in either Example 26 or 27.

[0086] Example 29: The device is the second device, as described in either Example 26 or 27.

[0087] In some embodiments, certain aspects of the techniques described above may be implemented by one or more processors of a processing system that runs the software. The software includes one or more sets of executable instructions, which are stored or otherwise tangibly embodied on a non-temporary computer-readable storage medium. The software may include instructions and specific data that, when executed by one or more processors, cause one or more processors to perform one or more aspects of the techniques described above. The non-temporary computer-readable storage medium may include, for example, magnetic or optical disk storage devices, and solid-state storage devices such as flash memory, cache, random access memory (RAM), or a single or multiple non-volatile memory devices, and similar. The executable instructions stored on the non-temporary computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats that can be interpreted or otherwise executed by one or more processors.

[0088] Computer-readable storage media may include any storage media, or combinations of storage media, that are accessible by a computer system while in use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tapes, or magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical system (MEMS) based storage media. Computer-readable storage media may be embedded in a computing system (e.g., system RAM or ROM), fixedly mounted to a computing system (e.g., magnetic hard drives), removable mounted to a computing system (e.g., optical disks or Universal Serial Bus (USB) based flash memory), or connected to a computer system via a wired or wireless network (e.g., network-accessible storage (NAS)).

[0089] In addition to the foregoing, it should be noted that not all activities or elements described above are required, and that certain activities or parts of devices may not be required, or that one or more additional activities may be performed, or that one or more additional elements may be included. Furthermore, the order in which the activities are listed does not necessarily indicate the order in which they are performed. Also, concepts are described with reference to specific embodiments. However, those skilled in the art will understand that various modifications and variations can be made without departing from the scope of this disclosure as set forth in the claims below. Accordingly, this specification and the drawings should be considered illustrative rather than restrictive, and all such variations are intended to be included within the scope of this disclosure.

[0090] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, benefits, advantages, and solutions to problems, as well as any features(if any) that may produce or make more prominent any benefit, advantage, or solution, shall not be construed as material, necessary, or essential features of any or all claims. Furthermore, the specific embodiments disclosed above are illustrative only. The subject matter of the disclosed invention may be modified and implemented in different but equivalent forms that are obvious to those skilled in the art who have an interest in teaching herein. No limitation is intended to any configuration or design details shown herein other than those described in the claims below. Accordingly, it is clear that the specific embodiments disclosed above may be changed or modified, and all such variations are considered within the scope of the subject matter of the disclosed invention. Consequently, the protection sought herein is as described in the claims below.

Claims

1. A method performed on a computer, in a first device, The method includes dividing the neural network into at least a set of current conditions for the first device and a second device, wherein the divided neural network configuration specifies the distribution of elements of the neural network architecture of the neural network between at least a first neural network portion of the first device and a second neural network portion of the second device, and the method further includes Implementing the first neural network portion in the first device, Transmitting the representation of the second neural network portion to the second device, The first neural network portion processes the first data in order to generate the first output, Sending a partitioning computation request including the proposed partitioned neural network configuration to the second device, Receiving instructions from the second device indicating either to accept the proposed segmented neural network configuration or to specify a modified segmented neural network configuration, A method performed on a computer, comprising determining or implementing the partitioned neural network configuration based on the received instructions.

2. The first data includes the input data of the neural network, The first output includes intermediate result data, The above method further, Transmitting the intermediate result data to the second device for processing by the second neural network portion. including, The method according to claim 1.

3. Receiving the final result output of the neural network from the second device, In response to the final result output, the first device performs at least one action, The method according to claim 2, further comprising:

4. The process further includes receiving the first data from the second device, wherein the first data includes intermediate result data generated by the second neural network portion in the second device. The first output is the final output of the neural network. The method according to claim 1.

5. Based on the final result output of the neural network, the first device performs at least one action. The method according to claim 4, further comprising:

6. The second device transmits input data from the first device to the second device for processing by the second neural network portion. The method according to claim 4, further comprising:

7. The method according to claim 1, wherein the set of one or more current conditions includes at least one of battery conditions, network conditions, thermal conditions, computing resource availability, or specified quality of service conditions.

8. The process involves determining the latency conditions for executing the neural network operation using the aforementioned neural network, Based on the latency conditions, the partitioned neural network configuration of at least the first device and the second device is determined. The method according to claim 1, further comprising:

9. Based on one or more sets of current conditions or at least one of the latency conditions, a determination is made as to whether to fully execute the neural network operation on the first device or to execute the neural network operation by using a partitioned configuration of the neural network. In response to the decision to use the partitioned neural network configuration, the partitioned neural network configurations for at least the first device and the second device are determined. The method according to claim 8, further comprising:

10. A method performed on a computer, wherein in a first device, The method includes dividing the neural network into at least a set of current conditions for the first device and a second device, wherein the divided neural network configuration specifies the distribution of elements of the neural network architecture of the neural network between at least a first neural network portion of the first device and a second neural network portion of the second device, and the method further includes Implementing the first neural network portion in the first device, Transmitting the representation of the second neural network portion to the second device, The first neural network portion processes the first data in order to generate the first output, Includes, The process further includes sending a partitioning request to the second device, the partitioning request proposing the partitioned neural network configuration, The first neural network portion is a method implemented on the first device, executed by a computer, in response to receiving permission for the partitioning request from the second device.

11. A method performed on a computer, wherein in a first device, The method includes dividing the neural network into at least a set of current conditions for the first device and a second device, wherein the divided neural network configuration specifies the distribution of elements of the neural network architecture of the neural network between at least a first neural network portion of the first device and a second neural network portion of the second device, and the method further includes Implementing the first neural network portion in the first device, Transmitting the representation of the second neural network portion to the second device, The first neural network portion processes the first data in order to generate the first output, Includes, Determining the partitioned neural network configuration of at least the first device and the second device is: This includes sending a partitioning request to the second device, the partitioning request proposing different partitioned neural network configurations based on one or more sets of current conditions, and the determination further includes: This includes receiving a partitioning inverse proposal from the second device, the partitioning inverse proposal proposing the partitioned neural network configuration, A method, implemented on the first device and executed by a computer, in response to the first neural network portion accepting the partition calculation inverse proposal based on one or more sets of current conditions.

12. The method according to claim 10, wherein the partitioning calculation request further indicates latency conditions for performing neural network operations using the partitioned neural network configuration.

13. The method according to claim 1, wherein the representation of the second neural network portion includes at least one of data describing the neural network architecture of the second neural network portion, or an identifier for a predetermined partitioned neural network configuration of the neural network represented in a repository accessible to the second device.

14. A method performed on a computer, on a second device, The method includes receiving instructions from a first device for a partitioned neural network configuration of at least the first device and the second device, wherein the partitioned neural network configuration specifies the distribution of elements of the neural network architecture of the neural network between at least the first neural network portion of the first device and the second neural network portion of the second device, and the method further includes: Implementing the second neural network portion in the second device, Receiving first data from the first device, The second neural network portion processes the first data in order to generate a first output, Includes, Receiving instructions for the partitioned neural network configuration includes receiving a partitioning operation request that proposes the partitioned neural network configuration, The aforementioned method, The process involves deciding whether to accept the proposed partitioned neural network configuration or to propose a modified partitioned neural network configuration. This includes sending an instruction to the first device indicating either to accept the proposed divided neural network configuration or to specify the modified divided neural network configuration, The second neural network portion is a method implemented in the second device and executed by a computer in response to the decision.

15. The first data includes the input data of the neural network, The first output includes intermediate result data, The above method further, Transmit the intermediate result data to the first device for processing in the first neural network portion. including, The method according to claim 14.

16. The first data includes intermediate result data generated by the first neural network portion in the first device. The first output is the final output of the neural network. The method according to claim 14.

17. In response to the final result output, the second device performs at least one action, or To transmit the final result output to the first device, The method according to claim 16, further comprising at least one of the following.

18. A method performed on a computer, wherein in a second device, The method includes receiving instructions from a first device for a partitioned neural network configuration of at least the first device and the second device, wherein the partitioned neural network configuration specifies the distribution of elements of the neural network architecture of the neural network between at least the first neural network portion of the first device and the second neural network portion of the second device, and the method further includes: Implementing the second neural network portion in the second device, Receiving first data from the first device, The second neural network portion processes the first data in order to generate a first output, Includes, Receiving the instruction for a partitioned neural network configuration includes receiving a partitioning computation request that proposes the partitioned neural network configuration, The method further includes determining whether to allow the partition calculation request, The second neural network portion is a method implemented in the second device, which is executed by a computer, in response to a decision to allow the partitioning computation request.

19. The first device further includes receiving instructions for latency conditions for performing neural network operations using the neural network, The decision of whether to allow the partition calculation request is based on the latency condition. The method according to claim 18.

20. The method according to claim 1, wherein the second neural network portion is implemented in the second device in response to receiving a representation of the second neural network portion from the first device, and the representation of the second neural network portion includes at least one of data representing the neural network architecture of the second neural network portion, or an identifier for a predetermined partitioned neural network configuration of the neural network represented in a repository accessible to the second device.

21. The method according to claim 1, wherein the first device is a user device, and the second device is one or more servers connected to a network that can be wirelessly accessed by the user device.

22. The method according to claim 1, wherein the neural network is a deep neural network (DNN) model.

23. The method according to claim 1, wherein the partitioned neural network configuration distributes a first set of continuous layers of the neural network to the first neural network portion, and distributes a second set of continuous layers of the neural network to the second neural network portion, the second set being adjacent to the first set.

24. The method according to claim 1, wherein the partitioned neural network configuration includes a two-way partition comprising an initial neural network portion and a final neural network portion, wherein the initial neural network portion is assigned to one of the first device or the second device, and the final neural network portion is assigned to the other of the first device or the second device.

25. The method according to claim 1, wherein the partitioned neural network configuration includes a three-way partition comprising an initial neural network portion, an intermediate neural network portion, and a final neural network portion, wherein the initial neural network portion and the final neural network portion are assigned to one of the first device or the second device, and the intermediate neural network portion is assigned to the other of the first device or the second device.

26. Network interface and At least one processor coupled to the network interface, A memory for storing executable instructions, wherein the executable instructions are configured to operate the at least one processor to perform the method according to any one of claims 1 to 25. device.

27. The device according to claim 26, wherein the first device is a user device of a cellular network, and the second device is one or more servers of a network that the user device can access wirelessly.

28. The device according to claim 26, wherein the device is the first device.

29. The device according to claim 26, wherein the device is the second device.

30. A program that causes at least one processor to perform the method according to any one of claims 1 to 25.

Images