Quantitative perception joint training for hardware capability of edge device

Abstract

A processor-implemented method for quantizing perceptual joint training includes quantizing, by a server, a global model. The global model is quantized at a plurality of different quantization levels for each of the one or more sub-network models to generate one or more quantized sub-network models. The one or more sub-network models are assigned to one or more devices of the plurality of devices according to device processing capabilities. And the server distributes the quantized sub-network model to one or more devices in the plurality of devices. The server receives local data based model updates from the one or more devices. The server generates an updated global model from an aggregate function based on the model update from each of the one or more devices.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 465,034, filed September 11, 2023, entitled “QUANTIZATION-AWARE FEDERATEDTRAINING TO ADDRESS EDGE DEVICES HARDWARE CAPABILITIES”, the entire disclosure of which is expressly incorporated herein by reference. Technical Field

[0003] This disclosure relates in general to neural networks, and more specifically to quantized perceptual joint training for edge device hardware capabilities. Background Technology

[0004] Federation learning is a method for collaboratively training neural networks across multiple edge devices without the need to collect data at a central location. Because of the distributed training (where the raw data is not shared among edge devices), federated learning is beneficial for applications where privacy is a critical factor. Federation learning aims to address differential privacy, continuous learning, and personalization by allowing edge (or terminal) devices to perform training locally using collected data and only sending weight updates instead of the original data.

[0005] While joint learning frameworks can address these fundamental problems, training on devices is challenging and can be resource-intensive, particularly in terms of memory and computation. Therefore, some resource-constrained devices may be prevented from participating in the joint learning process. This limitation on participation can lead to model bias and performance degradation. Summary of the Invention

[0006] This disclosure is set forth in the independent claims. Some aspects of this disclosure are described in the dependent claims.

[0007] In various aspects of this disclosure, a processor-implemented method includes quantizing a global model by a server. The global model is quantized at multiple different quantization levels for each of one or more sub-network models to generate one or more quantized sub-network models. The one or more sub-network models are assigned to one or more devices among a plurality of devices based on device processing capabilities. The processor-implemented method further includes distributing the quantized sub-network models to at least one device among the plurality of devices by the server. The processor-implemented method also includes receiving model updates based on local data from the at least one device by the server. The processor-implemented method further includes generating an updated global model by the server based on the model updates from each of the at least one device, according to an aggregation function.

[0008] Some aspects of this disclosure relate to an apparatus having at least one memory and one or more processors coupled to the at least one memory. The processors are configured to quantize a global model by a server. The global model is quantized at multiple different quantization levels for each of one or more sub-network models to generate one or more quantized sub-network models. The one or more sub-network models are assigned to one or more devices among a plurality of devices based on device processing capabilities. The processor is also configured to distribute the quantized sub-network models to at least one of the plurality of devices by the server. The processor is further configured to receive model updates based on local data from the at least one device by the server. The processor is further configured to include components for generating an updated global model by the server based on the model updates from each of the at least one device according to an aggregation function.

[0009] In various aspects of this disclosure, a processor-implemented method includes receiving a quantized subnetwork model from a server by a device. The quantized subnetwork model corresponds to a global model. The global model is quantized at multiple different quantization levels for each of one or more subnetwork models to generate one or more quantized subnetwork models. The one or more subnetwork models are assigned to one or more devices among a plurality of devices based on the device's processing capabilities. The processor-implemented method further includes generating a model update for the quantized subnetwork model by the device based on local data. The processor-implemented method also includes sending the model update to the server by the device, the server generating an updated global model based on the model update.

[0010] Some aspects of this disclosure relate to an apparatus having at least one memory and one or more processors coupled to the at least one memory. The processors are configured to receive quantized subnetwork models from a server. The quantized subnetwork models correspond to a global model. The global model is quantized at multiple different quantization levels for each of the one or more subnetwork models to generate one or more quantized subnetwork models. The one or more subnetwork models are assigned to one or more devices among a plurality of devices based on the device's processing capabilities. The processor is also configured to generate model updates for the quantized subnetwork models based on local data. The processor is further configured to send the model updates to the server, which generates an updated global model based on the model updates.

[0011] The aspects as a whole include, as described substantially with reference to the accompanying drawings and description, and as illustrated in the accompanying drawings and description, methods, apparatus, systems, computer program products, non-transitory computer-readable media, user equipment, base stations, wireless communication devices, and processing systems.

[0012] The features and technical advantages of the examples according to this disclosure have been summarized rather extensively above in order to better understand the detailed description that follows. Additional features and advantages will be described. The disclosed concepts and specific examples can be readily used as the basis for modifying or designing other structures for achieving the same purpose as this disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The characteristics of the disclosed concepts, in both their organization and manner of operation, and the associated advantages, will be better understood by considering the following description in conjunction with the accompanying drawings. Each drawing in the accompanying drawings is for illustrative and descriptive purposes and not intended to define limitations of the claims. Attached Figure Description

[0013] The features, substance, and advantages of this disclosure will become more apparent when understood in conjunction with the accompanying drawings, in which the same reference numerals are consistently used for identification.

[0014] Figure 1 Example implementations of designing neural networks using a system-on-a-chip (SoC) including a general-purpose processor, according to certain aspects of this disclosure, are illustrated.

[0015] Figure 2A , Figure 2B and Figure 2C This is a diagram illustrating various aspects of a neural network according to this disclosure.

[0016] Figure 2D This is a diagram illustrating an exemplary deep convolutional network (DCN) according to various aspects of this disclosure.

[0017] Figure 3 This is a block diagram illustrating an exemplary deep convolutional network (DCN) according to various aspects of this disclosure.

[0018] Figure 4 This is a block diagram illustrating an exemplary software architecture that enables modularization of artificial intelligence (AI) functions according to various aspects of this disclosure.

[0019] Figure 5 This is a high-level block diagram illustrating an example system for hardware-aware joint learning according to various aspects of this disclosure.

[0020] Figure 6A and 6B This is a flowchart illustrating an example process for hardware-aware joint learning according to various aspects of this disclosure.

[0021] Figure 7 This is a block diagram illustrating example processes for quantization-sensory joint training according to various aspects of this disclosure.

[0022] Figure 8 This is a diagram illustrating example pseudocode for server-side quantization-aware joint training according to various aspects of this disclosure.

[0023] Figure 9 This is a block diagram illustrating example processes for collaborative quantitative sensing joint training according to various aspects of this disclosure.

[0024] Figure 10 This is a diagram illustrating example pseudocode for collaborative quantization-aware joint training according to various aspects of this disclosure.

[0025] Figure 11 This is a flowchart illustrating an example processor implementation of a method for server-based quantization-aware joint training according to various aspects of this disclosure.

[0026] Figure 12 This is a flowchart illustrating an example processor implementation of a method for collaborative quantization-aware joint training according to various aspects of this disclosure. Detailed Implementation

[0027] The detailed description that follows, taken in conjunction with the accompanying drawings, is intended as a description of various configurations and not as representing only configurations in which the described concepts can be practiced. To provide a comprehensive understanding of the various concepts, the detailed description includes specific details. However, it will be apparent to those skilled in the art that these concepts can be practiced without these specific details. In some instances, to avoid obscuring such concepts, well-known structures and components are shown in block diagram form.

[0028] Based on the teachings, those skilled in the art will recognize that the scope of this disclosure is intended to cover any aspect of this disclosure, whether implemented independently of or in combination with any other aspect of this disclosure. For example, an apparatus or method may be implemented using any number of the aspects described. Furthermore, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or differ from the various aspects of this disclosure described. It should be understood that any aspect of this disclosure may be embodied by one or more elements of the claims.

[0029] The word “exemplary” is used to mean “serving as an example, instance, or illustration.” Any aspect described as “exemplary” need not be interpreted as superior to or better than other aspects.

[0030] While specific aspects have been described, numerous variations and substitutions of these aspects fall within the scope of this disclosure. Although some benefits and advantages of preferred aspects have been mentioned, the scope of this disclosure is not intended to be limited to a particular benefit, use, or purpose. Rather, aspects of this disclosure are intended to be broadly applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated by way of example in the accompanying drawings and the following description of preferred aspects. The detailed description and drawings are merely illustrative and not limiting of this disclosure, the scope of which is defined by the appended claims and their equivalents.

[0031] Federation learning is a decentralized form of machine learning in which one or more local clients (e.g., terminal devices) collaboratively train a statistical model under the orchestration of a central device (e.g., a server, serving cell, parameter server, etc.) while keeping the training data local and maintaining the privacy of the local client data. In other words, machine learning models (such as deep neural networks) are trained on raw data collected from multiple local datasets included in the terminal device, without receiving or accessing the raw data.

[0032] In other words, federated learning enables users (or end devices) to train machine learning models in a distributed manner. Each end device can use its local dataset to train a local model and then send model updates to a central server. For example, in each round of the federated learning process, the parameter server can select several users and send copies of the global machine learning model to the selected users. Each local training iteration of the federated learning process can be called an epoch, and each round of communication with the server can be called a communication round. Each end device uses its own dataset to compute the parameters of the model and feeds back the corresponding updates (e.g., weight updates) to the parameter server. The parameter server aggregates all end device updates and determines the update for the global model by, for example, averaging the aggregated end device updates or other techniques. The parameter server broadcasts the new parameters of the global model to the selected users in the next round of the federated learning process. Because no localized data is sent, federated learning is beneficial for applications where privacy is a factor.

[0033] As described, federated learning involves utilizing cross-device platforms. Distribution P A dataset of data points (Among them, for example, To learn with matrix tensor parameters Server models (such as neural networks) can be used without directly accessing device-specific datasets. This is achieved by defining a loss function for each end device. The total security risk can be written as:

[0034] This objective corresponds to a loss at each data point. joint dataset Minimize empirical risks. In joint learning, reducing communication costs is beneficial. Therefore, for devices... Each device in the process can optimize the weight parameters within the target's internal optimization process. Perform multiple gradient updates to obtain a result with weight parameters. The local model. These multiple gradient updates can be referred to as local epochs (such as the amount of data through the entire local dataset), abbreviated as... E Then, each terminal device can communicate its local weight to the server. Corresponding updates. Furthermore, the server in rounds... For example, by averaging the parameters of the local model. To update the global model.

[0035] While federated learning frameworks can address these fundamental issues, training them on devices is challenging and can be costly in terms of memory and computational resources.

[0036] Due to memory and processing power requirements, many devices may be unable to participate in federated learning training due to their hardware capabilities. Edge devices in federated learning can be mobile devices (“User Equipment (UE)”) that may have inherent differences in capabilities (or characteristics). For example, the hardware capabilities of different UEs may include the number and type of processors, and the amount and type of memory (e.g., speed). Dynamic hardware capabilities may include available or anticipated power (e.g., battery power), available or anticipated computing resources (e.g., for concurrently running applications), and available or anticipated communication bandwidth. Thus, hardware capabilities can be dynamic. Furthermore, the same UE may be able to train different (types) of models at different times. Hardware limitations may prevent devices with lower capabilities from obtaining the benefits of federated learning. Moreover, hardware limitations may cause model bias, as some users may be unable to utilize federated learning due to devices with limited hardware capabilities. Such limitations may lead to poor model performance (e.g., incorrect classification).

[0037] To address these and other challenges, various aspects of this disclosure relate to quantized perception joint training.

[0038] Figure 1 An example implementation of a System-on-Chip (SOC) 100 is illustrated, which may include a Central Processing Unit (CPU) 102 or a multi-core CPU configured for quantized perceptual joint training. Variables (e.g., neural signals and synaptic weights), system parameters associated with the computing device (e.g., a weighted neural network), latency, frequency bin information, and task information may be stored in a memory block associated with a Neural Processing Unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with a Graphics Processing Unit (GPU) 104, a memory block associated with a Digital Signal Processor (DSP) 106, a memory block 118, or may be distributed across multiple blocks. Instructions executed at the CPU 102 may be loaded from the program memory associated with the CPU 102 or from memory block 118.

[0039] SOC 100 may also include additional processing blocks tailored for specific functions, such as GPU 104, DSP 106, connectivity block 110 (which may include fifth-generation (5G) connectivity, fourth-generation LTE (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and multimedia processor 112 capable of, for example, detecting and recognizing gestures. In one implementation, the NPU is implemented within a CPU, DSP, and / or GPU. SOC 100 may also include sensor processor 114, image signal processor (ISP) 116, and / or navigation module 120, which may include a global positioning system.

[0040] The SOC 100 may be based on the ARM instruction set. In one aspect of this disclosure, instructions loaded into the CPU 102 may include code to quantize a global model by a server. This global model is quantized at multiple different quantization levels for each of one or more sub-network models to generate one or more quantized sub-network models. These one or more sub-network models are assigned to one or more devices among a plurality of devices based on device processing capabilities. The instructions loaded into the CPU 102 may also include code to distribute the quantized sub-network models to at least one of the plurality of devices by the server. The instructions loaded into the CPU 102 may additionally include code to receive model updates based on local data from the at least one device by the server. The instructions loaded into the CPU 102 may also include code to generate an updated global model by the server based on the model updates from each of the at least one device, according to an aggregation function.

[0041] In some aspects, the instructions loaded into CPU 102 may include code for the device to receive a quantized subnetwork model from a server. This quantized subnetwork model corresponds to a global model. The global model quantizes each of the one or more subnetwork models at multiple different quantization levels to generate one or more quantized subnetwork models. These one or more subnetwork models are assigned to one or more devices among a plurality of devices based on the device's processing capabilities. The instructions loaded into CPU 102 may also include code for the device to generate a model update for the quantized subnetwork model based on local data. The instructions loaded into CPU 102 may additionally include code for the device to send this model update to the server, which generates an updated global model based on the model update.

[0042] Deep learning architectures perform object recognition tasks by learning to represent inputs at progressively higher levels of abstraction in each layer, thereby constructing useful feature representations of the input data. In this way, deep learning addresses a major bottleneck of traditional machine learning. Before deep learning, machine learning methods for object recognition problems often relied heavily on human-designed features, possibly in conjunction with shallow classifiers. Shallow classifiers could be two-class linear classifiers, where a weighted sum of feature vector components is compared to a threshold to predict which class the input belongs to. Human-designed features could be templates or kernels customized for a specific problem domain by engineers with domain expertise. In contrast, while deep learning architectures can learn to represent features similar to those that human engineers might design, this requires training. Furthermore, deep networks can learn to represent and recognize novel types of features that humans might not have considered.

[0043] Deep learning architectures can learn hierarchical structures of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power at specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes in visual data or combinations of sounds in auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0044] Deep learning architectures perform particularly well when applied to problems with a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to identify features such as wheels, windshields, and others. These features can then be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0045] Neural networks can be designed to have multiple connectivity patterns. In feedforward networks, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in higher layers. As described above, hierarchical representations can be built in successive layers of a feedforward network. Neural networks can also have recurrent or feedback (also known as top-down) connections. In recurrent connections, the output from a neuron in a given layer can be passed to another neuron in the same layer. Recurrent architectures can help identify patterns across more than one block of input data that is sequentially delivered to the neural network. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be helpful when the recognition of higher-level concepts can aid in discerning specific lower-level features of the input.

[0046] The connections between layers in a neural network can be fully connected or locally connected. Figure 2A An example of a fully connected neural network 202 is illustrated. In the fully connected neural network 202, neurons in the first layer can transmit their outputs to each neuron in the second layer, so that each neuron in the second layer will receive inputs from each neuron in the first layer. Figure 2B An example of a locally connected neural network 204 is illustrated. In the locally connected neural network 204, neurons in the first layer can connect to a finite number of neurons in the second layer. More generally, the locally connected layers of the locally connected neural network 204 can be configured such that each neuron in the layer will have the same or similar connectivity pattern, but the connection strength can have different values ​​(e.g., 210, 212, 214, and 216). The connectivity pattern of locally connected layers can produce spatially different receptive fields in higher layers because higher-layer neurons in a given region can receive inputs that are tuned to the characteristics of a restricted portion of the total input to the network through training.

[0047] An example of a locally connected neural network is a convolutional neural network. Figure 2C An example of a convolutional neural network 206 is illustrated. Convolutional neural network 206 can be configured such that the connection strength associated with the input for each neuron in the second layer is shared (e.g., 208). Convolutional neural networks may be well-suited for problems where the spatial location of the input is meaningful.

[0048] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2D A detailed example of a DCN 200 designed to recognize visual features from an image 226 input from an image capture device 230 (such as an in-vehicle camera) is illustrated. The DCN 200 in this example can be trained to identify traffic signs and the numbers provided on them. Of course, the DCN 200 can be trained for other tasks, such as identifying lane markings or traffic lights.

[0049] Supervised learning can be used to train the DCN 200. During training, an image (such as image 226 of a speed limit sign) can be presented to the DCN 200, and forward passes can then be computed to produce output 222. The DCN 200 may include a feature extraction part and a classification part. Upon receiving image 226, convolutional layer 232 can apply a convolutional kernel (not shown) to image 226 to generate a first set 218 of feature maps. As an example, the convolutional kernel used for convolutional layer 232 may be a 5x5 kernel that generates a 28x28 feature map. In this example, because four different feature maps are generated in the first set of feature maps 218, four different convolutional kernels are applied to image 226 at convolutional layer 232. Convolutional kernels may also be referred to as filters or convolutional filters.

[0050] The first set of feature maps 218 can be subsampled by a max-pooling layer (not shown) to generate a second set of feature maps 220. The max-pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220 (e.g., 14x14) is smaller than the size of the first set of feature maps 218 (e.g., 28x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 can be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0051] exist Figure 2D In the example, the second set of feature maps 220 is convolved to generate a first feature vector 224. Furthermore, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature of the second feature vector 228 may include a number corresponding to a possible feature of image 226, such as "sign", "60", and "100". A softmax function (not shown) can convert the numbers in the second feature vector 228 into probabilities. Thus, the output 222 of DCN 200 can be the probability that image 226 includes one or more features.

[0052] In this example, the probabilities for "sign" and "60" in output 222 are higher than the probabilities for other numbers in output 222 (such as "30", "40", "50", "70", "80", "90", and "100"). Before training, output 222 generated by DCN 200 may be incorrect. Therefore, the error between output 222 and the target output can be calculated. The target output is the ground truth of image 226 (e.g., "sign" and "60"). The weights of DCN 200 can then be adjusted so that output 222 of DCN 200 is more closely aligned with the target output.

[0053] To adjust the weights, the learning algorithm computes the gradient vector of the weights. The gradient indicates by how much the error will increase or decrease as the weights are adjusted. At the top layers, the gradient corresponds directly to the values ​​of the weights connecting the activated neurons in the penultimate layer to the neurons in the output layer. In lower layers, the gradient depends on the values ​​of the weights and the error gradient computed in the higher layers. The weights can then be adjusted to reduce the error. This method of adjusting weights is called "backpropagation" because it involves "passing backward" through the neural network.

[0054] In practice, the error gradient of the weights can be calculated using a small number of examples to make the calculated gradient approximate the true error gradient. This approximation method is called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the entire system stops decreasing or until the error rate reaches a target level. After learning, a new image (e.g., a speed limit sign in image 226) can be presented to DCN 200, and output 222 can be generated through the forward pass of DCN 200. This output can be considered as an inference or prediction of DCN 200.

[0055] Deep Belief Networks (DBNs) are probabilistic models that include multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of training datasets. DBNs are obtained by stacking layers of Restricted Boltzmann Machines (RBMs). An RBM is a type of artificial neural network that learns a probability distribution from a set of inputs. Because RBMs can learn a probability distribution without information about the class each input should be classified into, they are often used for unsupervised learning. Using a hybrid paradigm of supervised and unsupervised learning, the bottom RBM of a DBN can be trained unsupervised and used as a feature extractor, while the top RBM can be trained supervisedly (on the joint distribution of inputs from the previous layer and the target class) and used as a classifier.

[0056] DCN is a network of convolutional networks configured with additional pooling and normalization layers. DCN has achieved state-of-the-art performance on many tasks. DCN can be trained using supervised learning, where both the input and output targets are known for many paradigms and are used to modify the network's weights using gradient descent.

[0057] DCNs can be feedforward networks. Furthermore, as described above, connections from neurons in the first layer of a DCN to a set of neurons in the next higher layer are shared across neurons in the first layer. The feedforward and shared connections of a DCN can be used for fast processing. For example, the computational cost of a DCN may be much smaller than that of a similarly sized neural network that includes recurrent or feedback connections.

[0058] The processing at each layer of a convolutional network can be thought of as a spatially invariant template or base projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, then a convolutional network trained on that input can be thought of as three-dimensional, with two spatial dimensions along the image's axes and a third dimension capturing color information. The output of the convolutional connections can be thought of as forming a feature map in the next layer, where each element in the feature map (e.g., 220) receives input from a range of neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values ​​in the feature map can be further processed with nonlinearities (such as correction, maximum value (0,x)). Values ​​from neighboring neurons can be further pooled, which corresponds to downsampling and provides additional local invariance and dimensionality reduction. Normalization corresponding to whitening can also be applied through lateral inhibition between neurons in the feature map.

[0059] Figure 3 This is a block diagram illustrating a DCN 350. A DCN 350 can include multiple layers of different types based on connectivity and weight sharing. For example... Figure 3 As shown, DCN 350 includes convolutional blocks 354A and 354B. Each convolutional block in convolutional blocks 354A and 354B can be configured with a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a max pooling layer (MAX POOL) 360.

[0060] Although only two of the convolutional blocks 354A and 354B are shown, this disclosure is not limited thereto, and any number of convolutional blocks 354A and 354B may be included in the DCN 350 according to design preferences.

[0061] Convolutional layer 356 may include one or more convolutional filters that can be applied to the input data to generate feature maps. Normalization layer 358 can normalize the output of the convolutional filters. For example, normalization layer 358 can provide whitening or lateral suppression. Max pooling layer 360 can provide spatial downsampling aggregation to achieve local invariance and dimensionality reduction.

[0062] Parallel filter banks of deep convolutional networks can be loaded onto an SOC 100 (e.g., Figure 4 The CPU 102 or GPU 104 of the SOC 100 can be used to achieve high performance and low power consumption. In an alternative implementation, a parallel filter bank can be loaded onto the DSP 106 or ISP 116 of the SOC 100. Additionally, the DCN 350 can access other processing blocks that may exist on the SOC 100, such as the sensor processor 114 and navigation module 120, which are dedicated to sensors and navigation, respectively.

[0063] The DCN 350 may also include one or more fully connected layers 362 (FC1 and FC2). The DCN 350 may also include logistic regression (LR) layers 364. Weights (not shown) to be updated are located between each of the layers 356, 358, 360, 362, and 364 of the DCN 350. The output of each layer (e.g., 356, 358, 360, 362, and 364) can be used as input to the next layer in the DCN 350 (e.g., 356, 358, 360, 362, and 364) to learn hierarchical feature representations from the input data 352 (e.g., images, audio, video, sensor data, and / or other input data) supplied at the first convolutional block in convolutional block 354A. The output of the DCN 350 is a classification score 366 of the input data 352. The classification score 366 may be a set of probabilities, where each probability is the probability that the input data includes a feature from the feature set.

[0064] Figure 4 This is a block diagram illustrating an exemplary software architecture 400 with modular artificial intelligence (AI) functionality. Using this architecture, applications can be designed to enable various processing blocks of the SOC 420 (e.g., CPU 422, DSP 424, GPU 426, and / or NPU 428) to support quantized perceptual joint training for AI applications 402 according to various aspects of this disclosure. Architecture 400 can be included, for example, in a computing device such as a smartphone.

[0065] AI application 402 can be configured to invoke functions defined in user space 404, which may, for example, provide the detection and recognition of a scene indicating the current location of the computing device (including architecture 400). For example, AI application 402 may configure microphones and cameras differently depending on whether the recognized scene is an office, lecture hall, restaurant, or an outdoor environment such as a lake. AI application 402 may make requests to compiled program code associated with libraries defined in the AI ​​Function Application Programming Interface (API) 406. This request may ultimately rely on the output of a deep neural network configured to provide inferred responses based on, for example, video and location data.

[0066] The runtime engine 408 (which may be compiled code of the runtime framework) may further be accessible to the AI ​​application 402. For example, the AI ​​application 402 may cause the runtime engine to request inferences at specific time intervals or triggered by events detected by the application's user interface. When the runtime engine provides an inference response, it may then signal to the operating system (such as the Linux kernel 412) in the operating system (OS) space 410 running on the SOC 420. The operating system may then enable sequential relaxation of quantization to be performed on the CPU 422, DSP 424, GPU 426, NPU 428, or some combination thereof. The CPU 422 may be directly accessible by the operating system, while other processing blocks may be accessible via drivers (such as drivers 414, 416, or 418 for the DSP 424, GPU 426, or NPU 428, respectively). In an exemplary example, a deep neural network may be configured to run on a combination of processing blocks such as the CPU 422, DSP 424, and GPU 426, or may run on the NPU 428.

[0067] According to certain aspects of this disclosure, each fully connected layer in the fully connected layer 362 may be configured to determine the parameters of the model based on one or more desired functional features of the model, and to further adapt, tune, and update the determined parameters to evolve the one or more functional features toward the desired functional features.

[0068] As indicated above, Figures 1 to 4 This is provided as an example. Other examples are available relative to... Figures 1 to 4 The examples described are different.

[0069] Figure 5 This is a high-level block diagram illustrating an example system 500 for hardware-aware joint learning according to various aspects of this disclosure. (Refer to...) Figure 5System 500 includes a server 502 for managing the federated learning model. System 500 also includes multiple terminal devices 504a-z. Each terminal device 504a-z may include a mobile communication device, such as a smartphone, tablet, electric vehicle, or Internet of Things (IoT) device. Each terminal device (e.g., 504a-z) may have a different hardware configuration, which may include dynamic hardware capabilities. For example, some terminal devices (e.g., 504a) may be configured with a graphics processing unit (GPU), a neural processing unit (NPU), a digital signal processing unit (DSP), or may have different memory configurations. Accordingly, each terminal device (e.g., 504a-z) may have different capabilities for operating the federated learning model or performing on-device training of the federated learning model. According to various aspects of this disclosure, each terminal device (e.g., 504a-z) may be configured to evaluate the current hardware capabilities of the device. The evaluation of the current hardware capabilities may be based, for example, on physical hardware configuration (e.g., GPU, NPU, etc.) and processing power. In some respects, current hardware capabilities can be evaluated or determined based on the current workload or other performance metrics of the terminal devices (504a-z). Terminal devices 504a-z can transmit indications of their current hardware capabilities. Furthermore, server 502 can adapt an initial or fully functional joint learning model for each terminal device (e.g., 504a-z) based on the current hardware capabilities. For example, server 502 can compress the fully functional joint learning model (e.g., via pruning, quantization, or one or more other model compression techniques). The compressed model can be delivered to a specific terminal device (e.g., 504a-z).

[0070] The terminal device (e.g., 504a-z) can continue to monitor its current hardware capabilities and update server 502 so that it can continue to provide models commensurate with those capabilities. In doing so, server 502 can provide a federated learning model at an optimal level suitable for the terminal device (e.g., 504a-z). That is, server 502 can be configured to run federated learning models on the terminal device with reduced or (to some extent) no degradation in model performance. This can beneficially reduce model latency and power consumption. Consequently, the user experience and enjoyment when the model is executed in the background can also be improved.

[0071] Additionally, the joint learning model can be improved as more end devices can participate in the joint learning process. Each end device (e.g., 504a-z) can individually retrain the model on its own based on locally collected data. Each end device (e.g., 504a-z) determines model updates (e.g., weight updates) and transmits such updates to server 502. In some aspects, the end devices can choose how frequently to provide model updates (e.g., weight updates) to the server. The frequency can be based on current hardware capabilities. In one example, a powerful smartphone can provide weight updates when it is charging or not under heavy workload. In another example, for end devices with lower hardware capabilities (such as IoT devices or other battery-powered devices where battery resources are more important), the update frequency can be adapted to keep the device running for longer. Furthermore, server 502 jointly trains the joint learning model based on the updates received from the end devices (e.g., 504a-z).

[0072] Figure 6A and 6B These are flowcharts illustrating example processes 600 and 650 for hardware-aware joint learning according to various aspects of this disclosure. (Refer to...) Figure 6A In box 602, the server (e.g., Figure 5 As shown in 502, it can be directed to the collection of participating terminal devices (e.g., Figure 5 The 504a-z diagram illustrates a joint learning model. This joint learning model can be, for example, an artificial neural network (e.g., Figure 3 (See 350 shown in the figure). The joint learning model can be a top-level model (e.g., a fully functional model). In various ways, the joint learning model can be generated based on the top-level hardware capabilities of the participating end devices. For example, the server can investigate the participating end devices and determine the top-level hardware capabilities. The server can then generate a top-level model based on the top-level hardware capabilities.

[0073] Each participating end device can assess its current hardware capabilities. In box 604, the participating end device can determine whether its current hardware capabilities are suitable for on-device training. On-device training suitability can be assessed based on various key performance indicators (KPIs). KPIs may include, for example, inference per second (IPS), double data rate read / write bandwidth, power consumption, memory footprint, or other performance indicators. In the first example, a threshold can be applied to determine whether the current hardware capabilities are suitable for on-device training (e.g., greater than 50,000 IPS). In the second example, the server can declare a set of models and a set of hardware specifications for running each model. Thus, the end device can determine whether its current hardware capabilities conform to or meet the specifications for the declared models, and (in some respects) determine the model best suited to other current hardware capabilities.

[0074] The terminal device can determine its current hardware capabilities based on its physical hardware configuration. Additionally, in some aspects, the current hardware capabilities can be determined based on the current workload, estimated completion time, or other performance metrics. If the current hardware capabilities are suitable for on-device training, then in box 606, the device retains the model (e.g., the top-level model). The device can operate on this model using locally collected data. Additionally, the device can perform on-device training based on locally collected data. Furthermore, the device can transmit weight updates computed during on-device training to a server (not shown).

[0075] If a device (e.g., 504b) determines that its current hardware capabilities may not be suitable for training on the device, then in box 608, the device may send a notification to the server. This notification may include an indication of the device's current hardware capabilities. Alternatively, in some aspects, the end device may also indicate that its current hardware capabilities are suitable for models that are more complex than the declared model.

[0076] In response to this notification, in box 610, the server can adapt the model to adjust its complexity. For example, in some aspects, the server can compress the top-level model. The server can use one or more of pruning, quantization, or other compression or model personalization techniques to compress the top-level model. The server can then deliver the adapted model to the end device.

[0077] After this, process 600 can return to box 604 to evaluate whether the current hardware capabilities are suitable for training on the device based on the adapted model.

[0078] In this way, process 600 can be applied iteratively until each device can successfully train a joint learning model on the device.

[0079] However, since current hardware capabilities can change, for example, based on changes in hardware configuration or workload, this process can be repeated continuously or periodically in some respects. In this way, model complexity can be updated and (in some respects) optimized based on the current hardware capabilities of each device.

[0080] In other aspects (not shown), the server transmits a representation of the model to the set of participating terminal devices, rather than the entire model. In these aspects, the terminal devices can determine whether they are capable of participating in a training epoch of the model based on this representation. If so, the terminal device accordingly sends a message to the server, which then transmits the initial model to the capable terminal devices.

[0081] Reference Figure 6BIn box 652, process 650 provides a server capable of generating a joint learning model with multiple categories or levels. This joint learning model with multiple categories or levels can have different levels of model complexity. This joint learning model with multiple categories or levels can be based on the different hardware capabilities of the participating end devices. For example, the joint learning model with multiple categories or levels can be based on different dimensions such as hardware processors (e.g., GPUs, NPUs, DSPs, etc.), floating-point weights, fixed-point weight quantization, edge pruning, etc.

[0082] In box 654, the terminal device may determine its current hardware capabilities. For example, the terminal device may determine whether its current hardware capabilities are suitable for training on the device. In various aspects, the current hardware capabilities may be determined based on the physical hardware configuration. Additionally, in some aspects, the current hardware capabilities may also be determined based on, for example, the workload (e.g., the application being executed), estimated workload completion, or other performance metrics. In still other aspects, the server may transmit an evaluation function to participating devices (e.g., terminal device 504z) to discover their hardware capabilities. The evaluation function may be a program executed on the terminal device. The output of the program captures the hardware capabilities of the terminal device at a given time or over a time period. The terminal device reports its hardware capabilities back to the server. The terminal device may use the evaluation function over time (periodically or event-driven) and may notify the server to negotiate a new model based on its current hardware capabilities.

[0083] In box 656, an end device (e.g., 504z) may notify the server of its current hardware capabilities. For example, the end device may report its hardware capabilities back to the server based on the output of an evaluation function. In box 658, the server may select a category or level of joint learning model for each end device based on its current hardware capabilities. The selected category or model may be an estimate of a model trained on the device that is compatible with its current hardware capabilities. In box 660, the server may send the model of the selected category to the end device.

[0084] Each terminal device can then perform on-device training based on the received model. Accordingly, the terminal device can collect data and operate a local model, and each participating device can be retrained on-device (e.g., according to a loss function) to generate local model updates (e.g., weight updates). Furthermore, the device can transmit the weight updates determined during on-device training to the server. Additionally, the server can update the joint learning model for each category or level based on the weight updates for the terminal device (e.g., 504a-z). For example, the server can use a weight update methodology (e.g., weight averaging) to update the weights of the joint learning model for each category. The server can also transmit the updated model categories to the respective devices.

[0085] The process can return to box 654 to repeat the evaluation of the current hardware capabilities. Accordingly, the server can provide a class of models to the end device in response to any change in the current hardware capabilities. In some aspects, the end device can initiate a model query to the server. For example, in the presence of a change in its current hardware capabilities (e.g., a change in physical hardware configuration or a change in workload), the end device can request the server to select a new model based on the current hardware capabilities in light of that change. In one example, the end device may be able to handle a more complex model because an incomplete process previously running on it has been completed. On the other hand, the device may have a new process competing for hardware resources and thus may be able to accommodate a less complex model.

[0086] In various ways, the terminal device can also train multiple classes or levels of a jointly trained ANN. For example, similar to a server, when the terminal device has considerable processing power (e.g., hardware configuration with numerous measurement resources) and the current workload is below a threshold (e.g., less than 10 percent of processing capacity), the terminal device can train multiple classes or levels of a jointly trained ANN. Additionally, the terminal device can also provide these jointly trained classes or levels of ANNs to, for example, servers or other terminal devices. For instance, when the device is charging overnight, with little or no competition for terminal device resources from other applications, the terminal device can train multiple models without impacting device performance and user experience.

[0087] Thus, the described dynamic method enables terminal devices (e.g., Figure 5 The 504a-z models can continue to participate in and benefit from the federated learning framework, regardless of the progress of competition for hardware resources on these devices. Furthermore, federated learning training can benefit from increasing the number of devices that can contribute to weight updates, thereby improving the federated learning model.

[0088] Various aspects of this disclosure relate to quantization-aware joint training. Weight updates can be generated to leverage hardware-aware joint machine learning. In various aspects of this disclosure, a server can perform quantization-aware joint training. In some aspects, a device and a server can jointly perform quantization-aware joint training.

[0089] In various aspects of this disclosure, one or more subnetwork models can be generated by using techniques such as one-off networks or by manually creating subnetwork models through pruning, to suit the hardware capabilities of a set of devices participating in joint machine learning.

[0090] parameter This is a set of subnetwork models ordered from smallest to largest size. Smaller subnetwork models can be assigned to devices with less hardware capabilities based on the following criteria: (1) the device can be assigned the largest subnetwork model that fits the device's memory footprint and is based on the device's processing power, or (2) the device can be assigned the largest subnetwork model that is an upper limit on the training time on the device. Furthermore, smaller subnetwork models can be included within larger subnetwork models. For example, S1 can be a subset of S2, S2 can be a subset of S3, etc.

[0091] Given a server and a set N A collection of devices The server may have already collected data. D And each device Can have local datasets The server can train an aggregated global model independently or jointly with the device. The device's local model Can be at a given fixed quantization level Training is conducted under the following conditions, in which The quantization level is represented by minimizing the summation loss:

[0092] in It is a loss function. It is the model quantization function, and the local model It can have different quantization levels. Quantization function Element-wise quantizable vector and return symbols and The latter is rounded to the endpoints of the containing interval.

[0093] In Formula 3, the first term It is a cross-device loss, and the second item It is an aggregated global model Data on the server The losses.

[0094] Figure 7 This is a block diagram illustrating an example process 700 for quantization-sensory joint training according to various aspects of this disclosure. (See reference) Figure 7 Example process 700 includes server 702, a first device (device 1 704a), and a second device (device 2 704b). For ease of illustration, two devices are shown. However, this disclosure is not limited thereto and may include any number of devices.

[0095] Server 702 can select one or more sub-network models (e.g., 706a, 706b) to train. In each training iteration (e.g., training round)... t Server 702 can send the quantized sub-network model to the corresponding device (e.g., 704a, 704b). That is, server 702 can generate a quantized global model for each device (e.g., 704a, 704b), where the global model is quantized at a quantization level based on the sub-network model. Thus, server 702 can send the quantized global model... Distribute the data to device 1 704a, which is assigned sub-network model 1, and quantize the global model. Distributed to device 2 704b (at box 710a) which is assigned subnetwork model 2.

[0096] Device 1 704a and Device 2 704b can train locally quantized subnetwork models for subnetwork 1 (at box 706a) and subnetwork 2 (at box 706b), respectively (e.g., , Device 1 704a and Device 2 704b can independently train and update their respective local sub-network models using their local data by minimizing the loss function given by the following equation:

[0097] Devices 1 704a and 2 704b can send the corresponding updated subnetwork model (e.g., weight updates) to server 702. Figure 7 In the example, devices 1 704a and 2 704b may not perform quantization on the updated subnetwork model.

[0098] Furthermore, server 702 can aggregate updated sub-network models (e.g., weight updates) from devices 1 704a and 2 704b (at box 712). Server 702 can then generate a global model update based on the aggregated weight updates. For example, server 702 can perform a joint averaging process as given below:

[0099] In some respects, in Formula 5, simply dividing by the number of devices is replaced. Masks can be applied (For example, a mask) Replaceable Mask It can have a global model Same size. Mask Each value can correspond to the iteration The number of devices that update specific parameters during the period.

[0100] Updating the global model Subsequently, in some respects, server 702 can fine-tune the updated global model (at box 714). For example, server 702 can use server data by minimizing the loss function given by the following equation. D To further train and update the global model:

[0101] In various ways, fine-tuning can be performed using a set of common data (such as data from backup devices, for example). Fine-tuning the updated global model can improve and, in some ways, optimize the updated global model for server data. D A trade-off between the use of local data on the corresponding devices (e.g., 702a, 702b) and the fine-tuning enables knowledge sharing between server 702 and devices (e.g., 702a, 702b).

[0102] Subsequently, example process 700 can be repeated. For example, at box 710b, the updated global model is fine-tuned ( Quantization can be performed according to the quantization level of different sub-network models. Example process 700 can be repeated according to design preferences or until model convergence is achieved.

[0103] Therefore, in Figure 7 In the server-side quantization-aware joint training method, server 702 independently performs model quantization (e.g., floating-point quantization or fixed-point quantization). That is, server 702 can perform model quantization, while devices (e.g., 704a, 704b) do not. In this way, the computational cost (e.g., processor workload, memory consumption, or power consumption) of each device (e.g., 704a, 704b) can be reduced.

[0104] A potential drawback of the server-based quantized-aware joint training method may be the quantized sub-network model generated by the server 702 (e.g., ...). It may be biased towards server data. D (For example, if server data) D Data from devices (e.g., 704a, 704b) The distribution offset between them increases. Since server 702 performs model quantization independently, only server data can be considered during the quantization process (quantization-aware training). D .

[0105] Therefore, to address potential biases, regularization can be performed in some ways to reduce data bias, thus benefiting the server (e.g., 702) and improving the performance (e.g., accuracy) and utility of the subnetwork model. For example, regularization can be implemented in the local subnetwork model. The quantized subnetwork model distributed by server 702 Add regularization terms between As shown below:

[0106] in It is the regularization parameter.

[0107] By applying regularization parameters This can reduce local model fine-tuning on devices (e.g., 704a, 704b) and server data usage. D The data offset between the optimized quantized sub-network models is thus optimized. Therefore, knowledge sharing across servers 702 and devices (e.g., 704a, 704b) can be smoothed out.

[0108] Figure 8 This is a diagram illustrating example pseudocode 800 for server-side quantization-aware joint training according to various aspects of this disclosure. For example... Figure 8 As shown, in line 2, at the server (e.g., 702), global model parameters can be initialized. In line 3, each device (e.g., 704a, 704b) can be assigned one of multiple sub-network models. For example, a sub-network model can be assigned to a device (e.g., 704a, 704b) based on its hardware capabilities (e.g., memory capacity and / or processing power).

[0109] In line 4, quantization-aware joint training can be repeatedly performed by the server (e.g., 702). T Next, one or more sub-network models can be selected for training. As shown in line 5, the sub-network models can be, for example, randomly selected. The server (e.g., 702) can send the quantized sub-network models to each device (e.g., 704a, 704b) assigned the selected sub-network models. In line 8, each device (e.g., 704a, 704b) can then train a local model (as shown in line 11) and can transmit updated sub-network models (e.g., weight updates) to the server (e.g., 702). In line 9, the server (e.g., 702) can aggregate the received updated sub-network models to generate an updated global model. For example, the updated sub-network models can be aggregated and averaged (e.g., via a joint averaging process) to generate an updated global model. In some respects, the updated global model can be fine-tuned (in line 10). Thereafter, the process returns to line 4 and can be repeated.

[0110] Figure 9 This is a block diagram illustrating an example process 900 for collaborative quantitative sensing joint training according to various aspects of this disclosure. (See reference) Figure 9Server 902 can select one or more sub-network models to train. In each training iteration (e.g., training rounds) t Server 902 can send the quantized sub-network model to the corresponding device (e.g., 904a, 904b). In other words, server 902 can generate a quantized global model for each device, where the global model is quantized at different levels based on the sub-network model. Thus, server 902 can send the quantized global model... The distributed global model is assigned to the first device (device 1 904a) of the first sub-network model (sub-network model 1) and quantized. Distributed to the second device (device 2 904b) assigned to the second sub-network model (sub-network model 2) (at box 910a).

[0111] Device 1 904a and Device 2 904b can train locally quantized subnetwork models for subnetwork 1 (at box 906a) and subnetwork 2 (at box 906b), respectively (e.g., , Device 1 904a and Device 2 904b can independently train and update their local sub-network models using their respective local data by minimizing the loss function given by Equation 4.

[0112] Device 1 904a and Device 2 904b can train local models for sub-network 1 (at box 906a) and sub-network 2 (at box 906b), respectively (e.g., , Device 1 904a and Device 2 904b can each quantize their respective model parameters (e.g., weights) (at boxes 908a and 908b). In some respects, considering the resource constraints of the devices (e.g., 904a, 904b), quantization may not be performed by the devices (e.g., 904a, 904b) in each training epoch to reduce device computational costs.

[0113] Devices (e.g., 904a and 904b) can transmit the quantized subnetwork model back to server 902. However, if quantization is not performed by devices (904a, 904b) for a given training epoch, devices (904a, 904b) can transmit the trained subnetwork model back to server 902.

[0114] Server 902 may receive a quantized, updated subnetwork model generated for server 902 by devices (904a, 904b). Server 902 may then, for example, use a joint averaging process (at box 912) to aggregate the quantized, updated subnetwork model and generate an updated global model. The updated global model may also be fine-tuned to further improve the updated global model (e.g., increase accuracy) (at box 914). Then, at box 910b, the server may perform model quantization, and this process may be repeated for subsequent training epochs.

[0115] Figure 10 This is a diagram illustrating example pseudocode 1000 for collaborative quantization-aware joint training according to aspects of this disclosure. Figure 10 Example pseudocode 1000 is similar to Figure 8 The example pseudocode 800 is shown. However, in example pseudocode 1000, both the server (e.g., 902) and the devices (e.g., 904a, 904b) can perform model quantization. For example, in line 7, the server (e.g., 902) can send the quantized subnetwork model to each device (e.g., 904a, 904b) separately. Furthermore, the devices (e.g., 904a, 904b) can also perform quantization in one or more training epochs (in line 11). The devices (904a, 904b) can send the quantized, updated subnetwork model back to the server (e.g., 902), as shown in line 15.

[0116] In some respects, regularization can be performed to reduce the impact on server data. or device data Data bias. Since both the server (e.g., 902) and the device (e.g., 904a, 904b) can perform a portion of model quantization, this bias can favor either the server (e.g., 902) or the device (e.g., 904a, 904b). Therefore, regularization can be added to both local and global training as follows:

[0117] in These are device regularization parameters.

[0118] Figure 11 This is a flowchart illustrating an example processor implementation of method 1100 for server-based quantization-aware joint training according to various aspects of this disclosure. For example, the processor-implemented method 1100 may be executed by one or more processors, such as CPUs (e.g., 102, 422), GPUs (e.g., 104, 426), DSPs (e.g., 106, 424), and / or NPUs (e.g., 108, 428)).

[0119] like Figure 11 As shown, at box 1102, the server quantizes the global model. This global model is quantized at multiple different quantization levels for each of one or more sub-network models to generate one or more quantized sub-network models. These one or more sub-network models are assigned to one or more devices among a plurality of devices based on device processing capabilities. For example, as referenced... Figure 7 As described, in each training iteration (e.g., training rounds) t Server 702 can send the quantized sub-network model to the corresponding device (e.g., 704a, 704b). Server 702 can generate a quantized global model for each device (e.g., 704a, 704b), wherein the global model is quantized at a quantization level according to the sub-network model.

[0120] At box 1104, the server distributes the quantized subnetwork model to at least one of the multiple devices. (See reference...) Figure 7 As described, server 702 can quantize the global model. Distribute the data to device 1 704a, which is assigned sub-network model 1, and quantize the global model. Distributed to device 2704b (at box 710a) which is assigned subnetwork model 2.

[0121] At box 1106, the server receives model updates based on local data from the at least one device. For example, as referenced... Figure 7 As described, devices 1 704a and 2 704b can train locally quantized subnetwork models for subnetwork 1 (at box 706a) and subnetwork 2 (at box 706b), respectively (e.g., , Furthermore, devices 1 704a and 2 704b can send the corresponding updated subnetwork model (e.g., weight update) to server 702.

[0122] At box 1108, the server generates an updated global model based on the model update from each of the at least one device, according to an aggregation function. For example, as referenced... Figure 7 As described, server 702 can aggregate updated sub-network models (e.g., weight updates) from devices 1704a and 2 704b (at box 712). Server 702 can then generate a global model update based on the aggregated weight updates. In one example, server 702 can generate the update by performing a joint averaging process relative to the aggregated model updates. In some aspects, the updated global model can be fine-tuned using, for example, common data from backup devices with the same device type and processing capabilities.

[0123] Figure 12 This is a flowchart illustrating an example processor implementation of method 1200 for collaborative quantization-aware joint training according to various aspects of this disclosure. For example, the processor-implemented method 1200 may be executed by one or more processors, such as CPUs (e.g., 102, 422), GPUs (e.g., 104, 426), DSPs (e.g., 106, 424), and / or NPUs (e.g., 108, 428)).

[0124] refer to Figure 12 At box 1202, the device receives a quantized sub-network model from the server. This quantized sub-network model corresponds to the global model. The global model quantizes each of the one or more sub-network models at multiple different quantization levels to generate one or more quantized sub-network models. These one or more sub-network models are assigned to one or more devices among a plurality of devices based on the device's processing capabilities. For example, as referenced... Figure 9 As described, server 902 can quantize the global model. The distributed global model is assigned to the first device (device 1 904a) of the first sub-network model (sub-network model 1) and quantized. The distributed data is assigned to the second device (device 2 904b) (at box 910a) of the second sub-network model (sub-network model 2). The quantized global model is then assigned to the first device (device 1 904a) of the first sub-network model (sub-network model 1). and the quantized global model It is generated for each device assigned to the corresponding sub-network model, and the global model is quantized at the quantization level according to the sub-network model.

[0125] At box 1204, the device generates model updates for the quantized sub-network model based on local data. For example, as referenced... Figure 9 As described, devices 1 904a and 2 904b can independently train and update their local sub-network models using their respective local data by minimizing the loss function given by the following equation, as shown in Equation 4.

[0126] At box 1206, the GIA device sends the model update to the server. The server then uses this model update to generate an updated global model. See, for example, a reference. Figure 9 As described, the devices (e.g., 904a and 904b) can transmit the quantized subnetwork model back to the server 902.

[0127] Specific implementation examples are provided in the following numbered clauses.

[0128] 1. A processor-implemented method executed by one or more processors, the processor-implemented method comprising: The server quantizes the global model, which quantizes each of the one or more sub-network models at multiple different quantization levels to generate one or more quantized sub-network models, which are then assigned to one or more devices among a plurality of devices based on the device's processing capabilities. The server distributes the quantized sub-network model to at least one of the plurality of devices; The server receives model updates based on local data from the at least one device; and The server generates an updated global model based on the model updates from each of the at least one device, according to an aggregation function.

[0129] 2. The processor-implemented method according to Clause 1, further comprising: using public data from a backup device to fine-tune the updated global model.

[0130] 3. The processor-implemented method according to Clause 1 or 2, further comprising: quantizing the updated global model by the server for each of the one or more sub-network models at the plurality of different quantization levels to generate one or more quantized updated sub-network models.

[0131] 4. The processor-implemented method according to any one of Clauses 1 to 3, the processor-implemented method further comprising: applying a regularization process to the updated global model to reduce bias in the server data.

[0132] 5. The method implemented by the processor according to any one of Clauses 1 to 3, wherein the model update from the at least one device is a quantized model update based on the quantization of the at least one device.

[0133] 6. A processor-implemented method executed by one or more processors, the processor-implemented method comprising: The device receives a quantized sub-network model from the server. The quantized sub-network model corresponds to a global model. The global model quantizes each of the one or more sub-network models at multiple different quantization levels to generate one or more quantized sub-network models. The one or more sub-network models are assigned to one or more devices among a plurality of devices according to the device's processing capabilities. The device generates model updates for the quantized sub-network model based on local data; and The device sends the model update to the server, and the server generates an updated global model based on the model update.

[0134] 7. The processor-implemented method according to Clause 6, wherein the model update based on the device uses an aggregation function to generate the updated global model.

[0135] 8. The processor-implemented method according to Clause 6 or 7, the processor-implemented method further comprising: quantizing the model update by the device to generate a quantized model update, the quantized model update being used to generate the updated global model.

[0136] 9. The processor-implemented method according to any one of Clauses 6 to 8, the processor-implemented method further comprising: applying a first regularization process to the updated global model to reduce a first bias on server data, or applying a second regularization process to the quantized model update to reduce a second bias on device data.

[0137] 10. A processor-implemented method according to any one of claims 6 to 9, the processor-implemented method further comprising: repeating the receiving, generating, and transmitting for a plurality of training epochs; and wherein the quantization is performed based on a subset of the training epochs based on the device processing capabilities.

[0138] 11. An apparatus comprising: At least one memory; and At least one processor, coupled to the at least one memory, is configured to: The server quantizes the global model, which quantizes each of the one or more sub-network models at multiple different quantization levels to generate one or more quantized sub-network models, which are then assigned to one or more devices among a plurality of devices based on the device's processing capabilities. The server distributes the quantized sub-network model to at least one of the plurality of devices; The server receives model updates based on local data from the at least one device; and The server generates an updated global model based on the model updates from each of the at least one device, according to an aggregation function.

[0139] 12. The apparatus according to Clause 11, wherein the at least one processor is further configured to: fine-tune the updated global model using public data from the backup device.

[0140] 13. The apparatus according to clause 11 or 12, wherein the at least one processor is further configured to: quantize the updated global model by the server for each of the one or more sub-network models at the multiple different quantization levels to generate one or more quantized updated sub-network models.

[0141] 14. The apparatus according to any one of clauses 11 to 13, wherein the at least one processor is further configured to apply a regularization process to the updated global model to reduce bias in the server data.

[0142] 15. The apparatus according to any one of clauses 11 to 14, wherein the model update from the at least one device is a quantized model update based on the quantization of the at least one device.

[0143] 16. An apparatus comprising: At least one memory; and At least one processor, coupled to the at least one memory, is configured to: The device receives a quantized sub-network model from the server. The quantized sub-network model corresponds to a global model. The global model quantizes each of the one or more sub-network models at multiple different quantization levels to generate one or more quantized sub-network models. The one or more sub-network models are assigned to one or more devices among a plurality of devices according to the device's processing capabilities. The device generates model updates for the quantized sub-network model based on local data; and The device sends the model update to the server, and the server generates an updated global model based on the model update.

[0144] 17. The apparatus according to Clause 16, wherein the model update based on the apparatus uses an aggregation function to generate the updated global model.

[0145] 18. The apparatus according to clause 16 or 17, wherein the at least one processor is further configured to: quantize the model update by the apparatus to generate a quantized model update, the quantized model update being used to generate the updated global model.

[0146] 19. The apparatus according to any one of clauses 16 to 18, wherein the at least one processor is further configured to: apply a first regularization process to the updated global model to reduce a first bias on server data, or apply a second regularization process to the quantized model update to reduce a second bias on device data.

[0147] 20. The apparatus according to any one of clauses 16 to 19, wherein the at least one processor is further configured to: repeat the receiving, the generating, and the transmitting for a plurality of training epochs; and wherein the apparatus performs the quantization based on the apparatus processing capacity of a subset of the training epochs.

[0148] In one aspect, the quantization component, distribution component, receiving component, generation component, determination component, and / or transmission component may be CPU 102, program memory associated with CPU 102, dedicated memory block 118, fully connected layer 362, and / or routing connection processing unit 216 configured to perform the described functions. In another configuration, the aforementioned components may be any module or any device configured to perform the functions described by the aforementioned components.

[0149] The various operations of the methods described above can be performed by any suitable component capable of performing the corresponding function. These components may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, in the cases where operations are illustrated in the accompanying drawings, these operations may have corresponding paired components with similar numbering plus functional components.

[0150] As used, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), assertion, etc. Additionally, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "determine" can include parsing, selecting, choosing, building, etc.

[0151] As used, the phrase "at least one of the items in the list" refers to any combination of these items, including a single member. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc.

[0152] The various exemplary logic blocks, modules, and circuits described in this disclosure can be implemented or executed using a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic component, discrete hardware component, or any combination thereof designed to perform the described functions. While the general-purpose processor may be a microprocessor, in alternative embodiments, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0153] The steps of the methods or processes described in this disclosure can be implemented directly in hardware, in a software module executed by a processor, or in a combination of both. The software module can reside in any form of storage medium known in the art. Some examples of usable storage media include random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs, and the like. The software module may include a single instruction or multiple instructions and may be distributed across several different code segments, across different programs, and across multiple storage media. The storage medium may be coupled to the processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium may be integral with the processor.

[0154] The disclosed method includes one or more steps or actions for implementing the described method. The steps and / or actions of the method may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of a particular step and / or action may be modified without departing from the scope of the claims.

[0155] The described functionality can be implemented in hardware, software, firmware, or any combination thereof. If implemented in hardware, an example hardware configuration may include a processing system within the device. This processing system may utilize a bus architecture. Depending on the specific application and overall design constraints of the processing system, the bus may include any number of interconnect buses and bridges. The bus can link various circuits together, including processors, machine-readable media, and bus interfaces. The bus interface can be used to connect network adapters, etc., to the processing system via the bus. The network adapter can be used to implement signal processing functions. In some respects, user interfaces (e.g., keypads, displays, mice, joysticks, etc.) may also be connected to the bus. The bus may also link various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are well known in the art and will not be described further.

[0156] A processor may be responsible for managing the bus and general-purpose processing, including executing software stored on a machine-readable medium. A processor may be implemented using one or more general-purpose processors and / or special-purpose processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuitry that can execute software. Software should be interpreted broadly as instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. By way of example, a machine-readable medium may include random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, disks, optical disks, hard disks, or any other suitable storage medium, or any combination thereof. A machine-readable medium may be embodied as a computer program product. A computer program product may include packaging material.

[0157] In a hardware implementation, machine-readable media can be part of a processing system separate from the processor. However, as those skilled in the art will readily understand, machine-readable media, or any portion thereof, can be external to the processing system. By way of example, machine-readable media may include transmit lines, carrier waves modulated by data, and / or computer components separate from the device, all accessible to the processor via a bus interface. Alternatively or additionally, machine-readable media, or any portion thereof, may be integrated into the processor, such as in the case of a cache and / or a general-purpose register file. Although the various components discussed may be described as having a specific location, such as local components, they may also be configured in various ways, such as certain components being configured as part of a distributed computing system.

[0158] The processing system may be configured as a general-purpose processing system having one or more microprocessors providing processor functionality and external memory providing at least a portion of machine-readable medium, all of which are linked together with other supporting circuitry via an external bus architecture. Alternatively, the processing system may include one or more neuromorphic processors for implementing the described neuron and nervous system models. As another alternative, the processing system may be implemented using an application-specific integrated circuit (ASIC) having a processor, bus interface, user interface, supporting circuitry, and at least a portion of machine-readable medium integrated on a single chip, or using one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic components, discrete hardware components, or any other suitable circuitry, or any combination of circuitry capable of performing the various functionalities described throughout this disclosure. Those skilled in the art will recognize how best to implement the described functionality of the processing system depends on the specific application and the overall design constraints imposed on the system as a whole.

[0159] Machine-readable media may include multiple software modules. These software modules include instructions that, when executed by a processor, enable the processing system to perform various functions. Software modules may include send and receive modules. Each software module may reside in a single storage device or be distributed across multiple storage devices. For example, when a triggering event occurs, a software module may be loaded from a hard disk drive into RAM. During the execution of a software module, the processor may load some of the instructions into a cache to improve access speed. One or more cache lines may then be loaded into a general-purpose register file for processor execution. When the functionality of a software module is referred to below, it will be understood that such functionality is implemented by the processor when executing the instructions from that software module. Furthermore, it should be understood that aspects of this disclosure result in improvements to the functionality of a processor, computer, machine, or other system implementing such aspects.

[0160] If implemented in software, the functions may be stored as one or more instructions or codes on or transmitted through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one location to another. A storage medium can be any available medium accessible to a computer. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and is accessible to a computer. Additionally, any connection is also appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, optical fiber, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwave, then such coaxial cable, optical fiber, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. The disks and optical discs used include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs. ® Optical discs, where magnetic disks typically reproduce data magnetically, and optical discs reproduce data optically using lasers. Therefore, in some aspects, computer-readable media may include non-transitory computer-readable media (e.g., tangible media). Furthermore, in other aspects, computer-readable media may include transient computer-readable media (e.g., signals). Combinations of the above should also be included within the scope of computer-readable media.

[0161] Therefore, certain aspects may include a computer program product for performing the presented operations. For example, such a computer program product may include a computer-readable medium on which instructions are stored (and / or encoded) that can be executed by one or more processors to perform the described operations. In some aspects, the computer program product may include packaging material.

[0162] Furthermore, it should be understood that modules and / or other suitable components for performing the described methods and techniques may be downloaded and / or otherwise obtained by the user terminal and / or base station where applicable. For example, such devices can be coupled to a server to facilitate the delivery of components for performing the described methods. Alternatively, the various methods described can be provided via storage components (e.g., RAM, ROM, physical storage media such as CDs or floppy disks) so that the user terminal and / or base station can obtain the various methods once the storage component is coupled to or provided to the device. In addition, any other suitable techniques suitable for providing the described methods and techniques to the device may be utilized.

[0163] It should be understood that the claims are not limited to the precise configurations and components illustrated above. Various modifications, variations, and alterations may be made to the arrangement, operation, and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. A processor-implemented method executed by one or more processors, the processor-implemented method comprising: The server quantizes the global model, which quantizes each of the one or more sub-network models at multiple different quantization levels to generate one or more quantized sub-network models, which are then assigned to one or more devices among a plurality of devices based on the device's processing capabilities. The server distributes the quantized sub-network model to at least one of the plurality of devices; The server receives model updates based on local data from the at least one device; as well as The server generates an updated global model based on the model updates from each of the at least one device, according to an aggregation function.

2. The processor-implemented method according to claim 1, further comprising: The updated global model is fine-tuned using public data from the backup device.

3. The processor-implemented method according to claim 1, further comprising: The server quantizes the updated global model at the multiple different quantization levels for each of the one or more sub-network models to generate one or more quantized updated sub-network models.

4. The processor-implemented method according to claim 1, further comprising: A regularization process is applied to the updated global model to reduce bias in the server data.

5. The processor-implemented method of claim 1, wherein the model update from the at least one device is a quantized model update based on the quantization of the at least one device.

6. A processor-implemented method executed by one or more processors, the processor-implemented method comprising: The device receives a quantized sub-network model from the server. The quantized sub-network model corresponds to a global model. The global model quantizes each of the one or more sub-network models at multiple different quantization levels to generate one or more quantized sub-network models. The one or more sub-network models are assigned to one or more devices among a plurality of devices according to the device's processing capabilities. The device generates model updates for the quantized sub-network model based on local data. as well as The device sends the model update to the server, and the server generates an updated global model based on the model update.

7. The processor-implemented method of claim 6, wherein the model update based on the device uses an aggregation function to generate the updated global model.

8. The processor-implemented method according to claim 6, further comprising: The device quantizes the model update to generate a quantized model update, which is used to generate the updated global model.

9. The processor-implemented method according to claim 8, further comprising: A first regularization process is applied to the updated global model to reduce a first bias on the server data, or a second regularization process is applied to the quantized model update to reduce a second bias on the device data.

10. The processor-implemented method according to claim 8, further comprising: The receiving, generating, and sending processes are repeated for multiple training rounds; and The quantization is performed in a subset of the training rounds based on the processing capabilities of the device.

11. An apparatus comprising: At least one memory; and At least one processor, coupled to the at least one memory, is configured to: The server quantizes the global model, which quantizes each of the one or more sub-network models at multiple different quantization levels to generate one or more quantized sub-network models, which are then assigned to one or more devices among a plurality of devices based on the device's processing capabilities. The server distributes the quantized sub-network model to at least one of the plurality of devices; The server receives model updates based on local data from the at least one device; as well as The server generates an updated global model based on the model updates from each of the at least one device, according to an aggregation function.

12. The apparatus of claim 11, wherein the at least one processor is further configured to: fine-tune the updated global model using common data from the backup device.

13. The apparatus of claim 11, wherein the at least one processor is further configured to: quantize the updated global model by the server for each of the one or more sub-network models at the plurality of different quantization levels to generate one or more quantized updated sub-network models.

14. The apparatus of claim 11, wherein the at least one processor is further configured to apply a regularization process to the updated global model to reduce bias in the server data.

15. The apparatus of claim 11, wherein the model update from the at least one device is a quantized model update based on the quantization of the at least one device.

16. An apparatus comprising: At least one memory; and At least one processor, coupled to the at least one memory, is configured to: The device receives a quantized sub-network model from the server. The quantized sub-network model corresponds to a global model. The global model quantizes each of the one or more sub-network models at multiple different quantization levels to generate one or more quantized sub-network models. The one or more sub-network models are assigned to one or more devices among a plurality of devices according to the device's processing capabilities. The device generates model updates for the quantized sub-network model based on local data. as well as The device sends the model update to the server, and the server generates an updated global model based on the model update.

17. The apparatus of claim 16, wherein the model update based on the apparatus uses an aggregation function to generate the updated global model.

18. The apparatus of claim 16, wherein the at least one processor is further configured to: quantize the model update by the apparatus to generate a quantized model update, the quantized model update being used to generate the updated global model.

19. The apparatus of claim 18, wherein the at least one processor is further configured to: apply a first regularization process to the updated global model to reduce a first bias on server data, or apply a second regularization process to the quantized model update to reduce a second bias on device data.

20. The apparatus of claim 18, wherein the at least one processor is further configured to: repeat the receiving, the generating, and the transmitting for a plurality of training epochs; and wherein the apparatus performs the quantization in a subset of the training epochs based on the apparatus's processing capabilities.

Images