Quantized cognitive association training to address the hardware capabilities of edge devices

Abstract

A processor implementation method for quantized cognitive association training includes a server quantizing a global model. The global model is quantized at multiple different quantization levels for each of the one or more subnetwork models in order to generate one or more quantized subnetwork models. The one or more subnetwork models are assigned to one or more of multiple devices according to their processing capabilities. The server distributes the quantized subnetwork models to one or more of the multiple devices. The server receives model updates from one or more devices based on local data. Based on the model updates from each of the one or more devices, the server generates an updated global model according to an aggregate function.

Description

[Technical Field]

[0001] (Cross-reference of related applications)

[0001] This application claims priority to U.S. Patent Application No. 18 / 465,034, filed on 11 September 2023, entitled "QUANTIZATION-AWARE FEDERATED TRAINING TO ADDRESS EDGE DEVICES HARDWARE CAPABILITIES," the disclosure of which is expressly incorporated in its entirety by reference.

[0002]

[0002] Aspects of this disclosure generally relate to neural networks, and more particularly to quantized cognitive association training to address the hardware capabilities of edge devices. [Background technology]

[0003]

[0003] Federated learning is a technique for collaboratively training neural networks across multiple edge devices without collecting data at a central location. Federated learning is beneficial for applications where privacy is a critical factor, as it allows for decentralized training where raw data is not shared among edge devices. Federated learning aims to address differential privacy, continuous learning, and personalization by having edge (or end) devices perform training locally using the collected data and send only weight updates, not raw data.

[0004]

[0004] While federative learning frameworks can address these fundamental problems, performing training on devices can be difficult and burdensome in terms of memory and computational resources. As a result, some resource-limited devices may be prevented from participating in the federative learning process. Such limitations on participation can lead to model bias and reduced model performance. [Overview of the Initiative]

[0005]

[0005] Each of the present disclosures is described in an independent claim. Some aspects of the present disclosures are described in dependent claims.

[0006]

[0006] In various aspects of the present disclosure, the processor implementation method includes a server quantizing a global model. The global model is quantized at a plurality of different quantization levels for each of the one or more subnetwork models in order to generate one or more quantized subnetwork models. The one or more subnetwork models are assigned to one or more of the plurality of devices according to the device processing capacity. The processor implementation method also includes a server distributing the quantized subnetwork models to at least one of the plurality of devices. The processor implementation method further includes a server receiving model updates from at least one device based on local data. The processor implementation method further includes a server generating an updated global model according to an aggregate function based on the model updates from each of the at least one device.

[0007]

[0007] Some aspects of the present disclosure relate to a device having at least one memory and one or more processors coupled to at least one memory. The processor(s) are configured by a server to quantize a global model. The global model is quantized at a plurality of different quantization levels for each of the one or more subnetwork models in order to generate one or more quantized subnetwork models. The one or more subnetwork models are assigned to one or more of the plurality of devices according to the device processing capacity. The processor(s) are also configured by a server to deliver the quantized subnetwork models to at least one of the plurality of devices. The processor(s) are further configured by a server to receive model updates from at least one device based on local data. The processor(s) are further configured by a server to include means for generating an updated global model according to an aggregate function based on the model updates from each of the at least one device.

[0008]

[0008] In various aspects of the present disclosure, a processor implementation method includes a device receiving a quantized subnetwork model from a server. The quantized subnetwork model corresponds to a global model. The global model is quantized at multiple different quantization levels for each of the one or more subnetwork models in order to generate one or more quantized subnetwork models. The one or more subnetwork models are assigned to one or more of a plurality of devices according to the device processing capacity. The processor implementation method also includes a device generating a model update of the quantized subnetwork model based on local data. The processor implementation method further includes a device sending the model update to a server, which generates an updated global model based on the model update.

[0009]

[0009] Some aspects of the present disclosure relate to a device having at least one memory and one or more processors coupled to at least one memory. The processor(s) are configured by the device to receive quantized subnetwork models from a server. The quantized subnetwork models correspond to a global model. The global model is quantized at multiple different quantization levels for each of the one or more subnetwork models in order to generate one or more quantized subnetwork models. The one or more subnetwork models are assigned to one or more of the multiple devices according to the device processing capacity. The processor(s) are also configured by the device to generate model updates of the quantized subnetwork models based on local data. The processor(s) are further configured by the device to send the model updates to a server, which generates an updated global model based on the model updates.

[0010]

[0010] The embodiments will be described in general with reference to the accompanying drawings and this specification, and will include methods, apparatus, systems, computer program products, non-temporary computer-readable media, user equipment, base stations, wireless communication devices, and processing systems as shown in the accompanying drawings and this specification.

[0011]

[0011] In the foregoing, the features and technical advantages of the embodiments according to the present disclosure have been outlined fairly broadly, so that the following "Detailed Description of Embodiments" may be better understood. Additional features and advantages will be described. The disclosed concepts and specific examples can be readily utilized as a basis for modifying or designing other structures to carry out the same purposes of the present disclosure. Such equivalent structures do not depart from the scope of the appended claims. The characteristics of the disclosed concepts, as well as both their configuration and method of operation, together with associated advantages, will be better understood by considering the following description in connection with the accompanying drawings. Each of the drawings is provided for purposes of illustration and description, and is not provided as a definition of the limitations of the claims.

Brief Description of the Drawings

[0012]

[0012] The features, properties, and advantages of the present disclosure will become more apparent from the detailed description set forth below when read in conjunction with the drawings, in which like reference numerals identify corresponding elements throughout. [Figure 1]

[0013] illustrates an exemplary implementation for designing a neural network using a system-on-a-chip (SOC) including a general-purpose processor, in accordance with some aspects of the present disclosure. [Figure 2ABC]

[0014] FIG. 2A is a diagram illustrating a neural network according to aspects of the present disclosure. FIG. 2B is a diagram illustrating a neural network according to aspects of the present disclosure. FIG. 2C is a diagram illustrating a neural network according to aspects of the present disclosure. [Figure 2D]

[0015] is a diagram illustrating an exemplary deep convolutional network (DCN), in accordance with aspects of the present disclosure. [Figure 3]

[0016] is a block diagram illustrating an exemplary deep convolutional network (DCN), in accordance with aspects of the present disclosure. [Figure 4]

[0017] FIG. 2 is a block diagram illustrating an exemplary software architecture that allows artificial intelligence (AI) functions to be modularized, in accordance with aspects of the present disclosure. [Figure 5]

[0018] FIG. 4 is a high-level block diagram illustrating an exemplary system for hardware-aware federated learning, in accordance with aspects of the present disclosure. [Figure 6A]

[0019] FIG. 8 is a flow diagram illustrating an exemplary process for hardware-aware federated learning, in accordance with aspects of the present disclosure. [Figure 6B] FIG. 11 is a flow diagram illustrating an exemplary process for hardware-aware federated learning, in accordance with aspects of the present disclosure. [Figure 7]

[0020] FIG. 15 is a block diagram illustrating an exemplary process for quantization-aware federated training, in accordance with various aspects of the present disclosure. [Figure 8]

[0021] FIG. 19 is a diagram illustrating exemplary pseudocode for server-side quantization-aware federated training, in accordance with various aspects of the present disclosure. [Figure 9]

[0022] FIG. 23 is a block diagram illustrating an exemplary process for collaborative quantization-aware federated training, in accordance with various aspects of the present disclosure. [Figure 10]

[0023] FIG. 27 is a diagram illustrating exemplary pseudocode for collaborative quantization-aware federated training, in accordance with various aspects of the present disclosure. [Figure 11]

[0024] FIG. 31 is a flow diagram illustrating an exemplary processor-implemented method for server-based quantization-aware federated training, in accordance with various aspects of the present disclosure. [Figure 12]

[0025] FIG. 35 is a flow diagram illustrating an exemplary processor-implemented method for collaborative quantization-aware federated training, in accordance with various aspects of the present disclosure. [Description of Embodiments]

[0013]

[0026] The embodiments for carrying out the invention described below with respect to the attached drawings illustrate various configurations and do not represent the only configuration in which the described concepts can be put into practice. The embodiments for carrying out the invention include specific details intended to provide a complete understanding of the various concepts. However, it will be clear to those skilled in the art that these concepts can be put into practice even without these specific details. In some examples, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0014]

[0027] Those skilled in the art will understand, based on the teachings, that the scope of this disclosure is intended to encompass any aspect of the disclosure, whether implemented independently of or in combination with any other aspect of the disclosure. For example, an apparatus may be implemented or a method may be practiced using any number of the described aspects. In addition, the scope of this disclosure shall encompass any such apparatus or method practiced using other structures, functions, or structures and functions in addition to or other than the various aspects of the disclosure described. It will be understood that any aspect of the disclosed disclosure may be embodied by one or more elements of the claims.

[0015]

[0028] The word "exemplary" is used to mean "serving as an example, case, or illustration." Any aspect described as "exemplary" should not necessarily be interpreted as being preferable or more favorable than other aspects.

[0016]

[0029] While specific embodiments are described, many variations and substitutions of these embodiments fall within the scope of this disclosure. While some advantages and benefits of preferred embodiments are stated, the scope of this disclosure is not limited to any particular advantage, use, or purpose. Rather, the embodiments of this disclosure are broadly applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated as examples in the figures and the following description of preferred embodiments. The embodiments and drawings for carrying out the invention are not limiting but merely illustrative of this disclosure, and the scope of this disclosure is defined by the appended claims and their equivalents.

[0017]

[0030] Federative learning is a decentralized form of machine learning in which one or more local clients (e.g., end devices) collaboratively train a statistical model under the orchestration of a central device (e.g., a server, serving cell, parameter server, etc.) while keeping the training data localized and maintaining the privacy of local client data. In other words, machine learning models such as deep neural networks are trained on raw data collected from multiple local datasets contained within the end device, without receiving or accessing the raw data.

[0018]

[0031] In other words, federated learning allows users (or end devices) to train a machine learning model in a distributed manner. Each end device uses its local dataset to train its local model and can then send model updates to a central server. For example, in each round of the federated learning process, the parameter server may select several users and send a copy of the global machine learning model to the selected users. Each local training iteration in the federated learning process may be called an epoch, and each communication round with the server may be called a communication round. Each end device uses its own dataset to compute the model parameters and feeds back the corresponding updates (e.g., weight updates) to the parameter server. The parameter server aggregates all end device updates and determines the update of the global model, for example, by averaging the aggregated end device updates or by other techniques. In the next round of the federated learning process, the parameter server broadcasts the new parameters of the global model to the selected users. Due to the non-transmission of localized data, federated learning is beneficial for applications where privacy is a factor.

[0019]

[0032] As explained, associative learning uses matrix tensor parameters w to create server models such as neural networks for P data points D={(x1,y1),...,(x P ,y P This involves training using the dataset )}, which is distributed across end devices N, and in the equation, for example, D=D1U...UD without directly accessing device-specific datasets. N The loss function L is calculated for each end device. N (D N By defining ;w), the total security risk can be written as follows: U

[0020]

number

[0021]

[0033] This objective corresponds to empirical risk minimization across a combined dataset D with a loss L(·) for each data point. In federative learning, reducing communication costs is beneficial. Therefore, multiple gradient updates for the weight parameter w in the internal optimization of the objective can be performed for each of the devices N, and thus the weight parameter w N Obtain a local model with the following characteristics: Multiple gradient updates are sometimes called local epochs, such as the amount of data that passes through the entire local dataset, represented by the abbreviation E. Each end device then obtains the local weights w S The server can then communicate the corresponding update. The server can then, for example, average the parameters of the local model.

[0022]

number

[0023] We will update the global model in Round T.

[0024]

[0034] While federative learning frameworks can address these fundamental problems, training them on devices can be challenging and burdensome in terms of memory and computational resources.

[0025]

[0035] Due to the memory and processing power requirements, many devices may be unable to participate in federative learning training due to their hardware capabilities. Edge devices in federative learning can be mobile devices ("user equipment, UEs") that may have inherent differences in capabilities (or characteristics). For example, hardware capabilities for different UEs may include the number and type of processors, as well as the amount and type of memory (e.g., speed). Dynamic hardware capabilities may include available or expected power (e.g., battery power), available or expected computing resources (e.g., based on concurrently running applications), and available or expected communication bandwidth. Thus, hardware capabilities can be dynamic. Furthermore, the same UE may be able to train different (types of) models at different times. Hardware limitations may prevent devices with lower capabilities from enjoying the benefits of federative learning. In addition, hardware limitations may bias the models because some users may be unable to utilize federative learning due to a limited number of devices with sufficient hardware capabilities. Such limitations may result in insufficient model performance (e.g., inaccurate classification).

[0026]

[0036] To address these and other challenges, aspects of this disclosure focus on quantized cognitive association training.

[0027]

[0037] Figure 1 shows exemplary implementations of a system-on-a-chip (SOC) 100, which may include a central processing unit (CPU) 102 or a multi-core CPU configured for quantized cognitive federation training, according to several aspects of the present disclosure. Variables (e.g., neural signals and synaptic weights), system parameters associated with computing devices (e.g., neural networks with weights), delays, frequency bin information, and task information may be stored in memory blocks associated with a neural processing unit (NPU) 108, memory blocks associated with the CPU 102, memory blocks associated with a graphics processing unit (GPU) 104, memory blocks associated with a digital signal processor (DSP) 106, memory block 118, or distributed across multiple blocks. Instructions executed in the CPU 102 may be loaded from program memory associated with the CPU 102 or from memory block 118.

[0028]

[0038] The SOC100 may also include a GPU104, a DSP106, a connectivity block110 which may include fifth-generation (5G) connectivity, fourth-generation long-term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, and additional processing blocks adapted to specific functions, such as a multimedia processor112 which can detect and recognize gestures. In one implementation, the NPU is implemented in the CPU, DSP, and / or GPU. The SOC100 may also include a navigation module120 which may include a sensor processor114, image signal processors (ISPs)116, and / or a global positioning system.

[0029]

[0039] The SOC100 may be based on the ARM instruction set. In one aspect of this disclosure, the instructions loaded into the CPU102 may include code for quantizing a global model, provided by the server. The global model is quantized at multiple different quantization levels for each of the one or more subnetwork models in order to generate one or more quantized subnetwork models. The one or more subnetwork models are assigned to one or more of the multiple devices according to their device processing capabilities. The instructions loaded into the CPU102 may also include code for distributing the quantized subnetwork models to at least one of the multiple devices, provided by the server. The instructions loaded into the CPU102 may further include code for receiving model updates from at least one device based on local data, provided by the server. The instructions loaded into the CPU102 may also include code for generating an updated global model according to an aggregate function, based on the model updates from each of the at least one device.

[0030]

[0040] In some embodiments, the instructions loaded into the CPU 102 may include code for the device to receive a quantized subnetwork model from a server. The quantized subnetwork model corresponds to a global model. The global model is quantized at multiple different quantization levels for each of the one or more subnetwork models in order to generate one or more quantized subnetwork models. The one or more subnetwork models are assigned to one or more of multiple devices according to the device's processing capacity. The instructions loaded into the CPU 102 may also include code for the device to generate model updates of the quantized subnetwork model based on local data. The instructions loaded into the CPU 102 may further include code for the device to send the model updates to the server, which generates an updated global model based on the model updates.

[0031]

[0041] Deep learning architectures can perform object recognition tasks by learning to represent inputs at successively higher levels of abstraction within each layer, thereby constructing useful feature representations of the input data. In this way, deep learning addresses a major bottleneck in traditional machine learning. Before the advent of deep learning, machine learning methods for object recognition problems relied heavily on human-designed features, perhaps in combination with shallow classifiers. A shallow classifier could be, for example, a two-class linear classifier where a weighted sum of feature vector components is compared to a threshold to predict which class an input belongs to. Human-designed features could be templates or kernels adapted to a specific problem domain by engineers with domain expertise. In contrast, deep learning architectures can learn through training to represent features similar to those that human engineers could design. Furthermore, deep networks can learn to represent and recognize novel types of features that humans might not have considered.

[0032]

[0042] Deep learning architectures can learn feature hierarchies. When presented with visual data, for example, the first layer may learn to recognize relatively simple features such as edges in the input stream. In another example, when presented with auditory data, the first layer may learn to recognize spectral power at a specific frequency. A second layer, inputting the output of the first layer, may learn to recognize combinations of features such as simple shapes in the case of visual data, or combinations of sounds in the case of auditory data. For example, higher layers may learn to represent complex shapes in visual data or words in auditory data. Even higher layers may learn to recognize common visual objects or spoken phrases.

[0033]

[0043] Deep learning architectures can perform particularly well when applied to problems with natural hierarchical structures. For example, classifying motorized vehicles may benefit from first learning to recognize wheels, windshields, and other features. These features can then be combined in different ways in higher layers to recognize cars, trucks, and airplanes.

[0034]

[0044] Neural networks can be designed using various connectivity patterns. In a feedforward network, each neuron in a given layer communicates with neurons in higher layers, passing information from lower to higher layers. As mentioned above, a hierarchical representation can be constructed within the consecutive layers of a feedforward network. Neural networks can also have recursive or feedback (also called top-down) connectivity. In recursive connectivity, the output from a neuron in a given layer can communicate with another neuron in the same layer. Recursive architectures can be useful when recognizing patterns that span two or more chunks of input data delivered to the neural network in a sequence. Connectivity from a neuron in a given layer to a neuron in a lower layer is called feedback (or top-down) connectivity. Networks with a large number of feedback connections can be useful when the recognition of a high-level concept can help identify specific low-level features of the input.

[0035]

[0045] The connections between layers in a neural network can be fully connected or locally connected. Figure 2A shows an example of a fully connected neural network 202. In a fully connected neural network 202, neurons in the first layer can communicate their outputs to all neurons in the second layer, and as a result, each neuron in the second layer receives input from all neurons in the first layer. Figure 2B shows an example of a locally connected neural network 204. In a locally connected neural network 204, neurons in the first layer can be connected to a limited number of neurons in the second layer. More generally, the locally connected layers of a locally connected neural network 204 can be configured such that each neuron in the layer has the same or similar connectivity pattern but different connection strengths (e.g., 210, 212, 214, and 216). Since higher-layer neurons within a given region can receive inputs that, through training, are tuned to the properties of a limited portion of the total inputs to the network, local connectivity patterns can give rise to spatially distinct receptive fields within the higher layers.

[0036]

[0046] An example of a locally connected neural network is a convolutional neural network. Figure 2C shows an example of a convolutional neural network 206. The convolutional neural network 206 may be configured such that the connection strengths associated with the input for each neuron in the second layer (e.g., 208) are shared. Convolutional neural networks may be suitable for problems where the spatial location of the input is meaningful.

[0037]

[0047] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2D shows a detailed example of DCN200, designed to recognize visual features from an image 226 input from an image capture device 230, such as an in-vehicle camera. In this example, DCN200 can be trained to identify traffic signs and the numbers written on them. Of course, DCN200 can be trained for other tasks, such as identifying lane markings or traffic signals.

[0038]

[0048] DCN200 can be trained using supervised learning. During training, DCN200 may be presented with images such as image 226 of a speed limit sign, and then a forward pass may be computed to produce output 222. DCN200 may include a feature extraction section and a classification section. Upon receiving image 226, the convolutional layer 232 may apply a convolutional kernel (not shown) to image 226 to generate a first set of feature maps 218. As an example, the convolutional kernel for convolutional layer 232 may be a 5x5 kernel that generates a 28x28 feature map. In this example, four different convolutional kernels were applied to image 226 in convolutional layer 232, since four different feature maps are generated in the first set of feature maps 218. Convolutional kernels are also sometimes called filters or convolutional filters.

[0039]

[0049] A first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218; that is, the size of the second set of feature maps 220, such as 14x14, is smaller than the size of the first set of feature maps 218, such as 28x28. The smaller size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved through one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0040]

[0050] In the example in Figure 2D, a second set of feature maps 220 is convolved to generate a first feature vector 224. Furthermore, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature in the second feature vector 228 may contain a digit corresponding to a possible feature of image 226, such as "label", "60", and "100". A softmax function (not shown) can be used to convert the digits in the second feature vector 228 into probabilities. Thus, the output 222 of DCN200 may be the probability that image 226 contains one or more features.

[0041]

[0051] In this example, the probabilities in output 222 for "label" and "60" are higher than the probabilities for other outputs in output 222 such as "30", "40", "50", "70", "80", "90", and "100". Before training, the outputs 222 generated by DCN200 are likely to be inaccurate. Therefore, an error can be calculated between output 222 and the target output. The target output is the ground truth of image 226 (e.g., "label" and "60"). The weights of DCN200 can then be adjusted so that the output 222 of DCN200 is closer to the target output.

[0042]

[0052] To adjust the weights, the learning algorithm may compute a gradient vector for the weights. The gradient may indicate the amount by which the error will increase or decrease if the weights are adjusted. In the top layer, the gradient may directly correspond to the weight values ​​connecting the activated neurons in the second-to-last layer to the neurons in the output layer. In lower layers, the gradient may depend on the weight values ​​and the computed error gradients in the upper layers. The weights can then be adjusted to minimize the error. This method of weight adjustment is sometimes called "backpropagation" because it involves a "backward path" through the neural network.

[0043]

[0053] In practice, the error gradient of the weights can be calculated over a small number of examples so that the calculated gradient approximates the true error gradient. This approximation method is sometimes called stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate for the entire system stops decreasing or until the error rate reaches a target level. After training, DCN200 may be presented with a new image (e.g., the speed limit sign in image 226), and a forward pass through DCN200 may yield an output 222 which can be considered an inference or prediction of DCN200.

[0044]

[0054] Deep belief networks (DBNs) are probabilistic models with multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of training datasets. DBNs can be obtained by stacking layers of restricted Boltzmann machines (RBMs). RBMs are a type of artificial neural network that can learn probability distributions over a set of inputs. Because RBMs can learn probability distributions without information about which class each input should belong to, they are frequently used in unsupervised learning. Using a hybrid paradigm of unsupervised and supervised learning, the lowest-level RBM of a DBN can be trained unsupervised and can function as a feature extractor, while the highest-level RBM can be trained supervised (on a combined distribution of inputs from previous layers and target classes) and can function as a classifier.

[0045]

[0055] DCNs are convolutional networks composed of additional pooling and normalization layers. DCNs achieve state-of-the-art performance for a wide range of tasks. DCNs can be trained using supervised learning, where both the input and output targets are known in a large number of cases, and the network weights are modified using gradient descent.

[0046]

[0056] A DCN can be a feedforward network. In addition, as mentioned above, connections from neurons in the first layer of a DCN to groups of neurons in the next higher layer are shared across the neurons in the first layer. The feedforward and shared connections of a DCN can be leveraged for high-speed processing. The computational burden of a DCN can be significantly less than that of a similarly sized neural network, for example, one that includes recursive or feedback connections.

[0047]

[0057] The processing of each layer of a convolutional network can be considered as a spatially invariant template or basis projection. If the input is initially decomposed into multiple channels, such as the red, green, and blue channels of a color image, the convolutional network trained on that input can be considered three-dimensional, having two spatial dimensions along the image axes and a third dimension that captures color information. The output of the convolutional connections can be considered as forming a feature map in subsequent layers, where each element of the feature map (e.g., 220) receives input from a range of neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values ​​in the feature map can be further processed using rectification, max(0,x), and other nonlinearities. Values ​​from neighboring neurons can be further pooled, which corresponds to downsampling and can provide additional local invariance and dimensionality reduction. Normalization, corresponding to whitening, can be applied through lateral inhibition between neurons in the feature map.

[0048]

[0058] Figure 3 is a block diagram of DCN350. DCN350 may include several different types of layers based on coupling and weight sharing. As shown in Figure 3, DCN350 includes convolutional blocks 354A and 354B. Each of the convolutional blocks 354A and 354B may consist of a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a max pooling layer (MAX POOL) 360.

[0049]

[0059] Although only two of the convolutional blocks 354A and 354B are shown, this disclosure is not limited in that way, and instead, any number of convolutional blocks 354A and 354B may be included in the DCN350 according to design preferences.

[0050]

[0060] The convolutional layer 356 may include one or more convolutional filters that can be applied to the input data to generate a feature map. The normalization layer 358 may normalize the output of the convolutional filters. For example, the normalization layer 358 may provide whitening or lateral suppression. The maximal pooling layer 360 may provide downsampling aggregation across space for local invariance and dimensionality reduction.

[0051]

[0061] For example, a parallel filter bank of a deep convolutional network may be loaded onto the CPU 102 or GPU 104 of the SOC 100 (e.g., Figure 4) to achieve high performance and low power consumption. In an alternative embodiment, the parallel filter bank may be loaded onto the DSP 106 or ISP 116 of the SOC 100. In addition, the DCN350 may have access to other processing blocks that may reside on the SOC 100, such as the sensor processor 114 and the navigation module 120, which are dedicated to sensors and navigation, respectively.

[0052]

[0062] DCN350 may also include one or more fully connected layers 362 (FC1 and FC2). DCN350 may further include a logistic regression (LR) layer 364. Between each layer 356, 358, 360, 362, 364 of DCN350 there are weights (not shown) to be updated. The output of each layer (e.g., 356, 358, 360, 362, 364) may serve as input to one of the subsequent layers (e.g., 356, 358, 360, 362, 364) in DCN350 to learn a hierarchical feature representation from the input data 352 (e.g., images, audio, video, sensor data, and / or other input data) supplied in the first of the convolutional blocks 354A. The output of DCN350 is a classification score 366 for the input data 352. The classification score 366 can be a set of probabilities, where each probability is the probability of the input data containing the feature from the set of features.

[0053]

[0063] Figure 4 is a block diagram showing an exemplary software architecture 400 in which artificial intelligence (AI) functions can be modularized. Using this architecture, an application can be designed in which various processing blocks of the SOC 420 (e.g., CPU 422, DSP 424, GPU 426, and / or NPU 428) can support quantized cognitive federation training for an AI application 402, according to aspects of this disclosure. Architecture 400 may be included in a computing device such as a smartphone.

[0054]

[0064] The AI ​​application 402 may be configured to invoke functions defined in user space 404, for example, which can provide scene detection and recognition indicating the location where a computing device including architecture 400 is currently operating. The AI ​​application 402 may configure its microphone and camera differently depending on whether the scene to be recognized is an office, an auditorium, a restaurant, or an outdoor setting such as a lake. The AI ​​application 402 may make requests to compiled program code associated with a library defined in the AI ​​function application programming interface (API) 406. These requests may ultimately rely on the output of a deep neural network configured to provide inference responses based on video and positioning data, for example.

[0055]

[0065] The AI ​​application 402 may also have further access to a runtime engine 408, which may be compiled code for a runtime framework. The AI ​​application 402 may prompt the runtime engine to perform inference, for example, at specific time intervals or triggered by events detected by the application's user interface. Once an inference response is provided, the runtime engine may then signal an operating system in the operating system (OS) space 410, such as a Linux kernel 412 running on the SOC 420. The operating system may then perform a continuous relaxation of quantization on the CPU 422, DSP 424, GPU 426, NPU 428, or any combination thereof. The CPU 422 may be accessed directly by the operating system, while the other processing blocks may be accessed through drivers, such as drivers 414, 416, or 418 for the DSP 424, GPU 426, or NPU 428, respectively. In an exemplary example, the deep neural network may be configured to operate on a combination of processing blocks such as CPU422, DSP424, and GPU426, or it may operate on NPU428.

[0056]

[0066] According to some aspects of this disclosure, each of the fully connected layers 362 may be configured to determine parameters of a model based on one or more desired functional features of the model, and to develop one or more functional features toward the desired functional features so that the determined parameters are further adapted, adjusted and updated.

[0057]

[0067] As described above, Figures 1 to 4 are provided as examples. Other examples may differ from those described with respect to Figures 1 to 4.

[0058]

[0068] Figure 5 is a high-level block diagram showing an exemplary system 500 for hardware cognitive federative learning according to an aspect of the present disclosure. Referring to Figure 5, system 500 includes a server 502 for managing a federative learning model. System 500 also includes a plurality of end devices 504a-z. Each of the end devices 504a-z may include, for example, a mobile communication device such as a smartphone, tablet, electric vehicle, or Internet of Things (IoT) device. Each of the end devices (e.g., 504a-z) may have a different hardware configuration, which may include dynamic hardware capabilities. For example, some end devices (e.g., 504a) may consist of graphics processing units (GPUs), neural processing units (NPUs), digital signal processors (DSPs), or may have different memory configurations. Thus, each of the end devices (e.g., 504a-z) may have a different capability for operating a federative learning model or for performing on-device training of a federative learning model. According to aspects of this disclosure, each of the end devices (e.g., 504a-z) may be configured to evaluate the device's current hardware capabilities. The evaluation of current hardware capabilities may be based, for example, on the physical hardware configuration (e.g., GPU, NPU, etc.) and processing power. In some aspects, current hardware capabilities may be evaluated or determined based on the current workload or other performance metrics of the end devices (e.g., 504a-z). The end devices 504a-z may transmit their current hardware capability indications. Server 502 may then adapt initial or full-featured federated learning models for each end device (e.g., 504a-z) according to the current hardware capabilities. For example, Server 502 may compress the full-featured federated learning model (e.g., via one or more of the following model compression techniques: pruning, quantization, or other model compression techniques). The compressed model may be transmitted to a particular end device (e.g., 504a-z).

[0059]

[0069] End devices (e.g., 504a-z) can continuously monitor their current hardware capabilities and update server 502 to ensure that it continues to provide models commensurate with those capabilities. In this way, server 502 can provide the best possible level of federated learning model that the end devices (e.g., 504a-z) can adapt to. That is, server 502 can configure a federated learning model that can run on the end device with reduced, or in some aspects, no, degradation of model performance. Therefore, model latency and power consumption can be beneficially reduced. Consequently, the user experience and enjoyment while the model is running in the background can also be improved.

[0060]

[0070] In addition, the federative learning model can be improved as more end devices may be able to participate in the federative learning process. Each end device (e.g., 504a-z) can individually retrain the model on-device based on locally collected data. Each end device (e.g., 504a-z) decides on model updates (e.g., weight updates) and sends such updates to the server 502. In some embodiments, the end devices may choose how often model updates (e.g., weight updates) are provided to the server. The frequency may be based on the current hardware capabilities. For example, a powerful smartphone may provide weight updates when it is charged or when it does not have a heavy workload. In another example, for end devices with lower hardware capabilities, such as IoT devices or other battery-powered devices where battery resources are a greater concern, the update frequency may be adapted to keep the device running longer. The server 502 then jointly trains the federative learning model based on the updates received from the end devices (e.g., 504a-z).

[0061]

[0071] Figures 6A and 6B are flowcharts illustrating exemplary processes 600 and 650 for hardware-cognitive federated learning according to aspects of the present disclosure. Referring to Figure 6A, in block 602, a server (e.g., 502 shown in Figure 5) may transmit a federated learning model to a set of participating end devices (e.g., 504a-z shown in Figure 5). The federated learning model may be, for example, an artificial neural network (e.g., 350 shown in Figure 3). The federated learning model may be a top-level hierarchical model (e.g., a full-features model). In various embodiments, the federated learning model may be generated based on the highest level of hardware capability of the participating end devices. For example, the server may investigate the participating end devices and determine the highest level of hardware capability. The server may then generate a top-level hierarchical model based on the highest level of hardware capability.

[0062]

[0072] Each participating end device can assess its current hardware capabilities. In block 604, a participating end device can determine whether its current hardware capabilities are suitable for on-device training. In various embodiments, suitability for on-device training may be assessed based on several key performance indicators (KPIs). KPIs may include, for example, inferences per second (IPS), double data-rate read / write bandwidth, power consumption, memory footprint, or other performance indicators. In the first example, a threshold may be applied to determine whether the current hardware capabilities are suitable for on-device training (e.g., >50,000 IPS). In the second example, a server may advertise a set of models and a set of hardware specifications for running each model. Thus, if the end device's current hardware capabilities conform to or meet the specifications of the advertised models, in some embodiments, it may determine which model is best suited to its current hardware capabilities.

[0063]

[0073] An end device may determine its current hardware capability based on its physical hardware configuration. In addition, in some embodiments, the current hardware capability may be determined based on the current workload, estimated time to completion, or other performance metrics. If the current hardware capability is suitable for on-device training, in block 606, the device holds a model (e.g., a top-level model). The device may run the model on locally collected data. In addition, the device may perform on-device training based on locally collected data. The device may then send weight updates calculated during on-device training to a server (not shown).

[0064]

[0074] If a device (e.g., 504b) determines that its current hardware capabilities may not be suitable for on-device training, in block 608, the device may send a notification to the server. The notification may include an indication of the device's current hardware capabilities. Alternatively, in some embodiments, the end device may also indicate that its current hardware capabilities are suitable for a more complex model than the advertised model.

[0065]

[0075] In response to the notification, in block 610, the server may adapt the model to adjust its complexity. For example, in some embodiments, the server may compress the top-level model. The server may compress the top-level model using one or more of the following techniques: pruning, quantization, or other compression or model personalization techniques. The server may transmit the adapted model to the end device.

[0066]

[0076] Process 600 then returns to block 604 to evaluate whether the current hardware capabilities can be adapted for on-device training based on the adapted model.

[0067]

[0077] In this way, process 600 can be applied iteratively until each device can successfully train a federative learning model on-device.

[0068]

[0078] However, since current hardware capabilities can change, for example, based on changes in hardware configuration or workload, in some embodiments the process may be repeated continuously or periodically. In this way, model complexity may be updated and, in some embodiments, optimized based on the current hardware capabilities of each device.

[0069]

[0079] In other embodiments (not shown), the server sends a feature description of the model to a set of participating end devices instead of the entire model. In these embodiments, the end devices can determine, based on the feature description, whether they are eligible to participate in a training round for the model. If so, the end device sends a message to the server accordingly, and the server then sends the initial model to the capable end devices.

[0070]

[0080] Referring to Figure 6B, in block 652, process 650 specifies that the server may generate multiple classes or hierarchies of federated learning models. Multiple classes or hierarchies of federated learning models may have different levels of model complexity. Multiple classes or hierarchies of federated learning models may be based on different hardware capabilities of participating end devices. For example, multiple classes of federated learning models may be based on different dimensions such as hardware processors (e.g., GPU, NPU, DSP, etc.), floating-point weights, fixed-point weight quantization, and implemented edge pruning.

[0071]

[0081] In block 654, an end device may determine its current hardware capability. For example, an end device may determine whether its current hardware capability is suitable for on-device training. In various embodiments, the current hardware capability may be determined based on the physical hardware configuration. In addition, in some embodiments, the current hardware capability may also be determined based on, for example, workload (e.g., applications being run), estimated workload completion, or other performance metrics. In yet another embodiment, the server may send an evaluation function to participating devices (e.g., end device 504z) to discover their hardware capability. The evaluation function may be a program executed on the end device. The output of the program captures the hardware capability of the end device at a given time or over a period of time. The end device reports its hardware capability to the server. The end device may use the evaluation function from time to time (periodically or event-driven) to notify the server to negotiate a new model based on its current hardware capability.

[0072]

[0082] In block 656, an end device (e.g., 504z) may inform the server of its current hardware capabilities. For example, the end device may report its hardware capabilities to the server based on the output of an evaluation function. In block 658, the server may select a class or hierarchy of federated learning models for each end device based on its current hardware capabilities. The selected class or model may be an estimate of a model that the end device's current hardware capabilities are suitable for on-device training. In block 660, the server may send the selected class of models to the end devices.

[0073]

[0083] Each end device can then perform on-device training based on the received model. Thus, an end device can collect data and run its local model, and each participating device can be retrained on-device (e.g., according to a loss function) to generate local model updates (e.g., weight updates). The device can then send the weight updates determined during on-device training to the server. Furthermore, the server can update each of the classes or hierarchies of the federated learning model based on the weight updates of the end devices (e.g., 504a-z). For example, the server can update the weights of each class of the federated learning model using a weight update method (e.g., weight averaging). The server can also send the updated model classes to the respective devices.

[0074]

[0084] The process may return to block 654 and repeat the evaluation of the current hardware capabilities. Thus, the server may provide the end device with a class of models in response to any change in the current hardware capabilities. In some embodiments, the end device may initiate a model query to the server. For example, if there is a change in its current hardware capabilities (e.g., a change in physical hardware configuration or a change in workload), the end device may request that the server select a new model based on its current hardware capabilities, taking the change into account. In one example, the end device may be able to handle a more complex model because any pending processes previously running on it have completed. On the other hand, the device may have new processes competing for hardware resources and therefore may be able to adapt to a less complex model.

[0075]

[0085] In various aspects, an end device may also train multiple classes or hierarchies of a jointly trained ANN. For example, if an end device has substantial processing power (e.g., a hardware configuration with numerous processing resources) and the current workload is below a threshold (e.g., less than 10 percent of processing power), the end device may train multiple classes or hierarchies of a jointly trained ANN, similar to a server. In addition, an end device may also provide such classes or hierarchies of a jointly trained ANN to, for example, a server or other end device. For example, when a device is charging overnight and other applications are not competing for end device resources with little or no competition, the end device may perform multiple model training without affecting device performance and user experience.

[0076]

[0086] Therefore, the described dynamic approach may enable end devices (e.g., 504a-z in Figure 5) to continue participating in and benefiting from the federative learning framework, regardless of processes competing for hardware resources on the device. Furthermore, federative learning training may benefit by increasing the number of devices that can contribute to weight updates, and thus improving the federative learning model.

[0077]

[0087] The aspects of this disclosure relate to quantized cognitive federation training. Weight updates may be generated to utilize hardware cognitive federation machine learning. In various aspects of this disclosure, a server may perform quantized cognitive federation training. In some aspects, a device and a server may perform quantized cognitive federation training jointly.

[0078]

[0088] In various aspects of this disclosure, one or more subnetwork models may be generated by using techniques such as once over the entire network, or by manually creating subnetwork models through pruning to suit the hardware capabilities of a set of devices participating in federated machine learning.

[0079]

[0089] Parameter S m , {m=1...K} is a set of subnetwork models sorted from the minimum size to the maximum size. Smaller subnetwork models can be assigned to devices with less hardware capability according to the following criteria, for example, (1) a device may be assigned the largest subnetwork model that can fit in the device's memory footprint and is based on the device's processing capability, or (2) a device may be assigned the largest subnetwork model that defines an upper limit on training time on the device. Furthermore, a smaller subnetwork model may be included within a larger subnetwork model. For example, S1 may be a subset of S2, S2 may be a subset of S3, and so on.

[0080]

[0090] Given a server and a set of N devices i∈[1,N], the server may have collected data D, and each device i has a local dataset D i The server may train the aggregated global model θ independently, or may train it collaboratively with the devices. The local model θ of the device i can be trained with a fixed quantization level q by minimizing the total loss using the following formula, where q i is given i represents the quantization level.

[0081]

Equation

[0082] In the formula, L is a loss function, Q is a model quantization function, and the local model θ i can have different levels of quantization q i . The quantization function Q may quantize the vector θ element-wise, and return the sign of θ and ||θ||2 rounded to the endpoints of the inclusion interval.

[0083]

[0091] In Equation 3, the first term [Σ i∈N L(Q(θ i , qi ), D i )] is the loss across the device, and the second term L(θ,D) is the server data θ=G({θ i This is the loss of the aggregated global model D for}).

[0084]

[0092] Figure 7 is a block diagram showing an exemplary process 700 for quantized cognitive federation training according to various aspects of the present disclosure. Referring to Figure 7, the exemplary process 700 includes a server 702, a first device (device 1 704a), and a second device (device 2 704b). Two devices are shown for ease of explanation. However, the present disclosure is not limited in this way and may include any number of devices.

[0085]

[0093] Server 702 may select one or more subnetwork models (e.g., 706a, 706b) to train. In each training iteration (e.g., training round t), Server 702 may transmit the quantized subnetwork model to each device (e.g., 704a, 704b). That is, Server 702 may generate a quantized global model for each device (e.g., 704a, 704b), and the global model is quantized at the quantization level according to the subnetwork model. Thus, Server 702 (in block 710a) the quantized global model

[0086]

number

[0087] Subnetwork Model 1 and quantized global model assigned to device 1 704a

[0088]

number

[0089] This can be distributed to subnetwork model 2 assigned to device 2 704b.

[0090]

[0094] Device 1 704a and Device 2 704b each use a local quantization subnetwork model (for example,

[0091]

number

[0092] The model can be trained for subnetwork 1 (in block 706a) and for subnetwork 2 (in block 706b). Device 1 704a and device 2 704b can independently train and update local subnetwork models using their respective local data by minimizing the loss function given by the following equation.

[0093]

number

[0094]

[0095] Device 1 704a and Device 2 704b may send their respective updated subnetwork models (e.g., weight updates) to Server 702. In the example in Figure 7, Device 1 704a and Device 2 704b may not perform quantization on the updated subnetwork models.

[0095]

[0096] Next, server 702 (in block 712) may aggregate the updated subnetwork models (e.g., weight updates) from device 1 704a and device 2 704b. Server 702 may then generate a global model update based on the aggregated weight updates. For example, server 702 may perform a federated averaging process as given by the following equation.

[0096]

number

[0097]

[0097] In some embodiments, in equation 5, instead of simply dividing by the number of devices N, the mask

[0098]

number

[0099] This may apply (for example, a mask)

[0100]

number

[0101] (This can replace N). Mask

[0102]

number

[0103] It can have the same size as the global model θ. Each value of the mask

[0104]

number

[0105] This can correspond to the number of devices that update a particular parameter during iteration t.

[0106]

[0098] After updating the global model θ, in some embodiments, the server 702 may fine-tune the updated global model (in block 714). For example, the server 702 may further train the updated global model using the server data D by minimizing the loss function given by:

[0107]

number

[0108]

[0099] In various embodiments, fine-tuning may be performed using a set of public data, such as data from a backup device. Fine-tuning the updated global model can improve, and in some embodiments optimize, the trade-off between the usefulness of the updated global model on server data D and the local data on each device (e.g., 702a, 702b). Furthermore, fine-tuning may enable knowledge sharing between server 702 and the devices (e.g., 702a, 702b).

[0109]

[0100] The exemplary process 700 may then be repeated. For example, in block 710b, the finely tuned updated global model (θ t+1 ) can be quantized according to the quantization level of different subnetwork models. The exemplary process 700 can be repeated according to design preferences or until model convergence is achieved.

[0110]

[0101] Therefore, in the server quantization recognition federation training method of Figure 7, server 702 independently performs model quantization (e.g., floating-point quantization or fixed-point quantization). That is, server 702 can perform model quantization, but the devices (e.g., 704a, 704b) do not. Therefore, the computational cost of each device (e.g., processor workload, memory consumption, or power consumption) can be reduced.

[0111]

[0102] One potential drawback of the server-quantized cognitive association training method is that (for example, server data D and device (e.g., 704a, 704b) data D j If the distribution shift between the two is large, the quantized subnetwork model generated by server 702 (e.g., S i The problem is that the data can be biased toward server data D. Since server 702 performs model quantization independently, only server data D can be considered in the quantization process (quantization recognition training).

[0112]

[0103] Therefore, in order to address potential bias, in some embodiments, regularization may be performed to reduce data bias in favor of the server (e.g., 702) and to improve the performance (e.g., accuracy) and usefulness of the subnetwork model. For example, regularization term |θ i -Q(θ i t ,q i )| is as follows, local subnetwork model θ i and the quantized subnetwork model Q(θ) delivered by server 702 i t ,q i ) can be added between them.

[0113]

number

[0114] In the formula, γ is the regularization parameter.

[0115]

[0104] By applying the regularization parameter γ, data shifts between the local model fine-tuning on the device (e.g., 704a, 704b) and the optimized quantized subnetwork model D on the server data can be reduced. Thus, knowledge sharing between the server 702 and the devices (e.g., 704a, 704b) can be facilitated.

[0116]

[0105] Figure 8 shows exemplary pseudocode 800 for server quantized cognitive federation training according to various aspects of the present disclosure. As shown in Figure 8, in line 2, global model parameters may be initialized on the server (e.g., 702). In line 3, each of the devices (e.g., 704a, 704b) may be assigned a subnetwork model from among a plurality of subnetwork models. For example, the devices (e.g., 704a, 704b) may be assigned a subnetwork model based on their device hardware capabilities (e.g., memory capacity and / or processing power).

[0117]

[0106] In line 4, the quantized cognitive federation training may be repeated T times by the server (e.g., 702). One or more subnetwork models may be selected for training. As shown in line 5, the subnetwork models may be selected randomly, for example. The server (e.g., 702) may send the quantized subnetwork model to each device (e.g., 704a, 704b) that has been assigned the selected subnetwork model. In line 8, each device (e.g., 704a, 704b) may then train a local model (as shown in line 11) and send the updated subnetwork model (e.g., weight updates) to the server (e.g., 702). In line 9, the server (e.g., 702) may aggregate the received updated subnetwork models to generate an updated global model. For example, the updated subnetwork models may be aggregated and averaged (e.g., via a federation averaging process) to generate an updated global model. In some embodiments, the updated global model may be fine-tuned (at line 10). The process then returns to line 4 and can be repeated.

[0118]

[0107] Figure 9 is a block diagram showing an exemplary process 900 for collaborative quantized cognitive federation training according to various aspects of the present disclosure. Referring to Figure 9, the server 902 may select one or more subnetwork models to train. In each training iteration (e.g., training round t), the server 902 may transmit the quantized subnetwork model to each device (e.g., 904a, 904b). That is, the server 902 may generate a quantized global model for each device, in which the global model is quantized at the quantization level according to the subnetwork model. Thus, the server 902 (in block 910a) transmits the quantized global model

[0119]

number

[0120] The data is distributed to the first device (device 1 904a) assigned to the first subnetwork model (subnetwork model 1), and the quantized global model is then generated.

[0121]

number

[0122] This can be distributed to a second device (device 2 904b) assigned to a second subnetwork model (subnetwork model 2).

[0123]

[0108] Device 1 904a and Device 2 904b each have a local quantization subnetwork model (for example,

[0124]

number

[0125] The model can be trained for subnetwork 1 (in block 906a) and for subnetwork 2 (in block 906b). Device 1 904a and device 2 904b can independently train and update their respective local subnetwork models using their local data by minimizing the loss function as shown in Equation 4.

[0126]

[0109] Device 1 904a and Device 2 904b each have a local model (for example,

[0127]

number

[0128] The model can be trained for subnetwork 1 (in block 906a) and subnetwork 2 (in block 906b). Device 1 904a and device 2 904b can each quantize their respective model parameters (e.g., weights) (in blocks 908a and 908b). In some embodiments, quantization by the devices (e.g., 904a, 904b) may not be performed in all training rounds in order to reduce device computation costs, given the resource constraints of the devices (e.g., 904a, 904b).

[0129]

[0110] The devices (e.g., 904a and 904b) may send back the quantized subnetwork model to the server 902. However, if quantization is not performed by the devices (904a, 904b) for a given training round, the devices (904a, 904b) may send back the trained subnetwork model to the server 902.

[0130]

[0111] Server 902 may receive quantized updated subnetwork models generated by devices (904a, 904b) for Server 902. Next, Server 902 may aggregate the quantized updated subnetwork models (in block 912) and generate an updated global model, for example using a federated averaging process. The updated global model may also be fine-tuned (in block 914) to further improve the updated global model (for example, to increase accuracy). The server may then perform model quantization, and the process may be repeated for subsequent training rounds in block 910b.

[0131]

[0112] Figure 10 shows exemplary pseudocode 1000 for collaborative quantized cognitive federation training according to an aspect of the present disclosure. The exemplary pseudocode 1000 in Figure 10 is similar to the exemplary pseudocode 800 shown in Figure 8. However, in the exemplary pseudocode 1000, both the server (e.g., 902) and the devices (e.g., 904a, 904b) can perform model quantization. For example, in line 7, the server (e.g., 902) may send a quantized subnetwork model to each of the devices (e.g., 904a, 904b). In addition, the devices (e.g., 904a, 904b) may also perform quantization in one or more training rounds (in line 11). The devices (904a, 904b) may send back the quantized updated subnetwork model to the server (e.g., 902), as shown in line 15.

[0132]

[0113] In some embodiments, regularization is performed on server data D or device data D i This may be done to reduce data bias toward the server (e.g., 902) and the devices (e.g., 904a, 904b), so the bias may favor either the server (e.g., 902) or the devices (e.g., 904a, 904b). Therefore, regularization may be added to local and global training as follows:

[0133]

number

[0134] In the equation, Δ is the device regularization parameter.

[0135]

[0114] Figure 11 is a flowchart illustrating an exemplary processor implementation method 1100 for server-based quantized cognitive federation training according to an aspect of the present disclosure. The processor implementation method 1100 may be executed by one or more processors such as a CPU (e.g., 102, 422), a GPU (e.g., 104, 426), a DSP (e.g., 106, 424), and / or an NPU (e.g., 108, 428).

[0136]

[0115] As shown in Figure 11, in block 1102, the server quantizes the global model. The global model is quantized at multiple different quantization levels for each of the one or more subnetwork models in order to generate one or more quantized subnetwork models. The one or more subnetwork models are assigned to one or more of the multiple devices according to the device processing capacity. For example, as illustrated with reference to Figure 7, in each training iteration (e.g., training round t), the server 702 may send the quantized subnetwork models to the respective devices (e.g., 704a, 704b). The server 702 may generate a quantized global model for each device (e.g., 704a, 704b), and the global model is quantized at the quantization level according to the subnetwork model.

[0137]

[0116] In block 1104, the server distributes the quantized subnetwork model to at least one of the multiple devices. As described with reference to Figure 7, the server 702 (in block 710a) distributes the quantized global model

[0138]

number

[0139] Subnetwork Model 1 and quantized global model assigned to device 1 704a

[0140]

number

[0141] This can be distributed to subnetwork model 2 assigned to device 2 704b.

[0142]

[0117] In block 1106, the server receives model updates from at least one device based on local data. For example, as illustrated with reference to Figure 7, device 1 704a and device 2 704b receive local quantized subnetwork models (e.g., for subnetwork 1 in block 706a and for subnetwork 2 in block 706b) respectively.

[0143]

number

[0144] ) can be trained. Next, device 1 704a and device 2 704b may send their respective updated subnetwork models (e.g., weight updates) to server 702.

[0145]

[0118] In block 1108, the server generates an updated global model according to an aggregate function based on model updates from each of at least one device. For example, as illustrated with reference to Figure 7, the server 702 (in block 712) may aggregate updated subnetwork models (e.g., weight updates) from device 1 704a and device 2 704b. The server 702 may then generate a global model update based on the aggregated weight updates. In one example, the server 702 may generate the update by performing a federated averaging process with respect to the aggregated model updates. In some embodiments, the updated global model may be fine-tuned using public data from, for example, a backup device having the same device type and processing power.

[0146]

[0119] Figure 12 is a flowchart illustrating an exemplary processor implementation method 1200 for collaborative quantized cognitive federation training according to an aspect of the present disclosure. The processor implementation method 1200 may be executed by one or more processors such as a CPU (e.g., 102, 422), a GPU (e.g., 104, 426), a DSP (e.g., 106, 424), and / or an NPU (e.g., 108, 428).

[0147]

[0120] Referring to Figure 12, in block 1202, the device receives a quantized subnetwork model from the server. The quantized subnetwork model corresponds to a global model. The global model is quantized at multiple different quantization levels for each of the one or more subnetwork models in order to generate one or more quantized subnetwork models. The one or more subnetwork models are assigned to one or more of the multiple devices according to the device processing capacity. For example, as described with reference to Figure 9, the server 902 (in block 910a) receives the quantized global model

[0148]

number

[0149] The data is distributed to the first device (device 1 904a) assigned to the first subnetwork model (subnetwork model 1), and the quantized global model is then generated.

[0150]

number

[0151] This can be distributed to a second device (device 2 904b) assigned to a second subnetwork model (subnetwork model 2). A quantized global model for a first device (device 1 904a) assigned to a first subnetwork model (subnetwork model 1)

[0152]

number

[0153] and quantized global models

[0154]

number

[0155] This is generated for each device assigned to each subnetwork model, and the global model is quantized at the quantization level according to the subnetwork model.

[0156]

[0121] In block 1204, the device generates model updates for the quantized subnetwork model based on the local data. For example, as illustrated with reference to Figure 9, device 1 904a and device 2 904b can independently train and update their respective local subnetwork models using their local data by minimizing the loss function as shown in Equation 4.

[0157]

[0122] In block 1206, the device sends a model update to the server. The server generates an updated global model based on the model update. For example, as illustrated with reference to Figure 9, the devices (e.g., 904a and 904b) may send back a quantized subnetwork model to the server 902.

[0158]

[0123] Implementation examples are provided in the following numbered clauses. Clause 1. A processor implementation method that is performed by one or more processors, wherein the processor implementation method is The server quantizes the global model, wherein the global model is quantized at multiple different quantization levels for each of the one or more subnetwork models in order to generate one or more quantized subnetwork models, and the one or more subnetwork models are assigned to one or more of multiple devices according to the device processing capacity. The server delivers the quantized subnetwork model to at least one of several devices. The server receives model updates from at least one device based on local data, A processor implementation method comprising: a server generating a global model updated according to an aggregate function based on model updates from each of at least one device. Clause 2. The processor implementation method described in Clause 1, further including fine-tuning the updated global model using publicly available data from a backup device. Clause 3. The processor implementation method described in Clause 1 or 2, further comprising the server quantizing a global model updated at multiple different quantization levels for each of the one or more subnetwork models in order to generate one or more quantized updated subnetwork models. Clause 4. A processor implementation method described in any one of Clauses 1 to 3, further comprising applying a regularization process to an updated global model in order to reduce bias toward server data. Clause 5. A processor implementation method according to any one of Clauses 1 to 3, wherein a model update from at least one device is a quantized model update based on quantization by at least one device. Clause 6. A processor implementation method performed by one or more processors, wherein the processor implementation method is The device receives a quantized subnetwork model from a server, the quantized subnetwork model corresponding to a global model, the global model being quantized at multiple different quantization levels for each of the one or more subnetwork models to generate one or more quantized subnetwork models, and the one or more subnetwork models being allocated to one or more of multiple devices according to the device's processing capacity. Depending on the device, it can generate model updates for the quantized subnetwork model based on local data, A processor implementation method comprising: a device sending a model update to a server, the server generating and sending an updated global model based on the model update. Clause 7. The updated global model is generated using an aggregate function based on the model update by the device, as described in the processor implementation method in Clause 6. The processor implementation method described in Clause 8, further comprising quantizing a model update by a device to generate a quantized model update, the quantized model update being used to generate an updated global model. Clause 9. A processor implementation method according to any one of Clauses 6 to 8, further comprising applying a first regularization process to an updated global model to reduce a first bias toward server data, or applying a second regularization process to a quantization model update to reduce a second bias toward device data. Clause 10. A processor implementation method according to any one of Clauses 6 to 9, further comprising receiving, generating and transmitting repeatedly for multiple training rounds, and quantization being performed in a subset of training rounds based on the device processing capability. Clause 11. Apparatus, At least one memory, At least one processor coupled to at least one memory, It is equipped with at least one processor, The server quantizes the global model, and the global model is quantized at multiple different quantization levels for each of the one or more subnetwork models to generate one or more quantized subnetwork models, and the one or more subnetwork models are assigned to one or more of multiple devices according to the device processing capacity. The server distributes the quantized subnetwork model to at least one of several devices. The server receives model updates from at least one device based on local data. A device configured by a server to generate an updated global model according to an aggregate function based on model updates from each of at least one device. Clause 12. The apparatus described in Clause 11, wherein at least one processor is further configured to fine-tune the updated global model using the publicly available data of the backup device. Clause 13. The apparatus according to Clause 11 or 12, wherein at least one processor is further configured by the server to quantize a global model updated at multiple different quantization levels for each of the one or more subnetwork models in order to generate one or more quantized updated subnetwork models. Clause 14. The apparatus described in any one of Clauses 11 to 13, wherein at least one processor is further configured to apply a regularization process to an updated global model in order to reduce bias toward server data. Clause 15. An apparatus according to any one of Clauses 11 to 14, wherein a model update from at least one device is a quantized model update based on quantization by at least one device. Clause 16. Apparatus, At least one memory, At least one processor coupled to at least one memory, It is equipped with at least one processor, Depending on the device, a quantized subnetwork model is received from the server, the quantized subnetwork model corresponds to a global model, the global model is quantized at multiple different quantization levels for each of the one or more subnetwork models to generate one or more quantized subnetwork models, and one or more subnetwork models are assigned to one or more of multiple devices according to the device's processing capacity. Depending on the device, model updates for the quantized subnetwork model are generated based on local data. A device configured to send model updates to a server, which then generates an updated global model based on those updates. Clause 17. The updated global model is generated using an aggregate function based on the model update by the device as described in Clause 16. Clause 18. The apparatus according to Clause 16 or 17, wherein at least one processor is further configured by the device to perform quantization of a model update to generate a quantized model update, and the quantized model update is used to generate an updated global model. Clause 19. The apparatus according to any one of Clauses 16 to 18, wherein at least one processor is further configured to apply a first regularization process to an updated global model in order to reduce a first bias toward server data, or to apply a second regularization process to a quantization model update in order to reduce a second bias toward device data. Clause 20. The apparatus described in any one of Clauses 16 to 19, wherein at least one processor is further configured to receive, generate and transmit repeatedly for multiple training rounds, and the device performs quantization in a subset of training rounds based on the device's processing capability.

[0159]

[0124] In one embodiment, the quantization means, distribution means, receiving means, generation means, determination means, and / or transmission means may be a CPU 102, a program memory associated with the CPU 102, a dedicated memory block 118, a fully connected layer 362, and / or a routing connection processing unit 216 configured to perform the listed functions. In another configuration, the means described above may be any module or any device configured to perform the functions listed by the means described above.

[0160]

[0125] The various operations of the above-described method may be carried out by any preferred means capable of performing the corresponding function. The means may include, but are not limited to, various hardware and / or software components and / or modules, including circuits, application-specific integrated circuits (ASICs), or processors. Generally, where there are operations shown in the figures, those operations may have corresponding relative means-plus-function components that are similarly numbered.

[0161]

[0126] When used, the term “determining” encompasses a wide variety of actions. For example, “determining” may include calculating, calculating, processing, deriving, investigating, searching (e.g., searching a table, database, or other data structure), and confirming. Additionally, “determining” may include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and furthermore, “determining” may include resolving, selecting, choosing, and establishing.

[0162]

[0127] When used, the phrase referring to “at least one of” an enumeration of items refers to any combination of those items that includes a single member. For example, “at least one of a, b, or c” is intended to include a, b, c, ab, ac, bc, and abc.

[0163]

[0128] Various exemplary logic blocks, modules, and circuits described in this disclosure may be implemented or carried out using general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices (PLDs), discrete gates or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described. A general-purpose processor may be a microprocessor; alternatively, a processor may be any commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0164]

[0129] Steps of methods or processes described in connection with this disclosure may be embodied in hardware directly, in software modules executed by a processor, or in a combination of the two. Software modules may reside in any form of storage medium known in the art. Some examples of storage mediums that may be used include random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs, and the like. Software modules may comprise one or more instructions and may be distributed across several different code segments, between different programs, and across multiple storage mediums. Storage mediums may be coupled to a processor so that the processor can read information from and write information to the storage medium. Alternatively, storage mediums may be integrated with the processor.

[0165]

[0130] The disclosed methods include one or more steps or actions for achieving the described method. The steps and / or actions of those methods may be interchanged with one another without departing from the claims. In other words, unless a particular order of steps or actions is specified, the order of any particular steps and / or actions, and / or the use of those steps and / or actions, may be modified without departing from the claims.

[0166]

[0131] The functions described may be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, an exemplary hardware configuration may include a processing system within the device. The processing system may be implemented using a bus architecture. The bus may include any number of interconnection buses and bridges, depending on the specific application of the processing system and the overall design constraints. The bus may connect various circuits, including processors, machine-readable media, and bus interfaces. The bus interface may, among other things, be used to connect a network adapter to the processing system via the bus. The network adapter may be used to implement signal processing functions. In some embodiments, a user interface (e.g., a keypad, display, mouse, joystick, etc.) may also be connected to the bus. The bus may also connect various other circuits, such as timing sources, peripherals, voltage regulators, and power management circuits, but these circuits are well known in the art and are therefore not described further.

[0167]

[0132] The processor may be responsible for managing the bus and general processing, including executing software stored on a machine-readable medium. The processor may be implemented using one or more general-purpose processors and / or dedicated processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuits capable of executing software. Software is broadly interpreted to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or by any other name. Machine-readable medium may include, for example, random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, magnetic disks, optical disks, hard drives, or any other suitable storage medium, or any combination thereof. The machine-readable medium may be embodied in a computer program product. The computer program product may include packaging materials.

[0168]

[0133] In hardware implementations, machine-readable media may be part of a processing system separate from the processor. However, as will be readily apparent to those skilled in the art, machine-readable media or any part thereof may be outside the processing system. For example, machine-readable media may include transmission lines, data-modulated carriers, and / or computer products separate from the device, all of which may be accessed by the processor through a bus interface. Alternatively, or additionally, machine-readable media or any part thereof may be integrated into the processor, as may be the case with caches and / or general-purpose register files. The various components discussed may be described as having a specific location, such as local components, but they may also be configured in various ways, such as several components configured as part of a distributed computing system.

[0169]

[0134] The processing system may be configured as a general-purpose processing system having one or more microprocessors providing processor functions and external memory providing at least a portion of a machine-readable medium, all connected together with other auxiliary circuits through an external bus architecture. Alternatively, the processing system may comprise one or more neuromorphological processors for implementing the described neuron model and neural system model. Another alternative is that the processing system may be implemented using an application-specific integrated circuit (ASIC) having a processor, a bus interface, a user interface, auxiliary circuits, and at least a portion of a machine-readable medium integrated on a single chip, or using one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic, discrete hardware components, or any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this disclosure. Those skilled in the art will recognize how to best implement the functions described for the processing system depending on the specific application and the overall design constraints imposed on the entire system.

[0170]

[0135] A machine-readable medium may comprise several software modules. A software module contains instructions that, when executed by a processor, cause a processing system to perform various functions. A software module may include a transmit module and a receive module. Each software module may reside in a single storage device or may be distributed across multiple storage devices. For example, a software module may be loaded from a hard drive into RAM when a trigger event occurs. During the execution of a software module, the processor may load some of the instructions into a cache to increase access speed. One or more cache lines may then be loaded into a general-purpose register file for execution by the processor. When referring to the functions of a software module below, it will be understood that such functions are implemented by the processor when it executes instructions from that software module. Furthermore, it should be understood that aspects of this disclosure result in improvements to the functionality of processors, computers, machines, or other systems that implement such aspects.

[0171]

[0136] When implemented in software, functions may be stored on or transmitted via computer-readable media as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any media that facilitates the transfer of computer programs from one place to another. Storage media may be any available media that can be accessed by a computer. Such computer-readable media may include, but not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other media that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. In addition, any connection is appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. When used, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray® disc, where disk typically reproduces data magnetically and disc optically using a laser. Thus, in some embodiments, computer-readable medium may include non-temporary computer-readable medium (e.g., tangible medium). In addition, in other embodiments, the computer-readable medium may include a temporary computer-readable medium (e.g., a signal).The above combinations should also be included within the scope of computer-readable media.

[0172]

[0137] Accordingly, some embodiments may comprise a computer program product that performs the operations described. For example, such a computer program product may comprise a computer-readable medium storing (and / or encoding) instructions, the instructions being executable by one or more processors to perform the operations described. In some embodiments, the computer program product may comprise packaging material.

[0173]

[0138] Furthermore, it should be understood that modules and / or other suitable means for carrying out the described methods and techniques may, as appropriate, be downloaded and / or otherwise obtained by user terminals and / or base stations. For example, such devices may be coupled to a server to facilitate the transfer of means for carrying out the described methods. Alternatively, the various described methods may be provided via storage means so that user terminals and / or base stations can obtain the various methods by coupling or providing storage means (e.g., physical storage media such as RAM, ROM, compact disks (CDs) or floppy disks) to the device. Moreover, any other suitable techniques for providing the described methods and techniques to the device may be utilized.

[0174]

[0139] It should be understood that the claims are not limited to the exact configurations and components illustrated above. Various modifications, changes, and variations may be made to the arrangement, operation, and details of the methods and apparatus described above without departing from the claims.

Claims

1. A processor implementation method that is performed by one or more processors, wherein the processor implementation method is Quantizing a global model by a server, wherein the global model is quantized at multiple different quantization levels for each of the one or more subnetwork models in order to generate one or more quantized subnetwork models, and the one or more subnetwork models are assigned to one or more of a plurality of devices according to the device processing capacity. The server distributes the quantized subnetwork model to at least one of the multiple devices. The server receives model updates from at least one device based on local data, A processor implementation method comprising: the server generating a global model updated according to an aggregate function based on the model updates from each of the at least one device.

2. The processor implementation method according to claim 1, further comprising fine-tuning the updated global model using publicly available data from a backup device.

3. The processor implementation method according to claim 1, further comprising the server quantizing the updated global model at one or more different quantization levels for each of the one or more subnetwork models to generate one or more quantized updated subnetwork models.

4. The processor implementation method according to claim 1, further comprising applying a regularization process to the updated global model in order to reduce bias toward server data.

5. The processor implementation method according to claim 1, wherein the model update from the at least one device is a quantization model update based on quantization by the at least one device.

6. A processor implementation method that is performed by one or more processors, wherein the processor implementation method is The device receives a quantized subnetwork model from a server, wherein the quantized subnetwork model corresponds to a global model, the global model is quantized at multiple different quantization levels for each of the one or more subnetwork models in order to generate one or more quantized subnetwork models, and the one or more subnetwork models are assigned to one or more of a plurality of devices according to the device's processing capacity. The device generates model updates for the quantized subnetwork model based on local data, A processor implementation method comprising: the device transmitting the model update to the server, the server generating and transmitting a global model updated based on the model update.

7. The processor implementation method according to claim 6, wherein the updated global model is generated using an aggregate function based on the model update by the device.

8. The processor implementation method according to claim 6, further comprising quantizing the model update to generate a quantized model update using the device, wherein the quantized model update is used to generate the updated global model.

9. The processor implementation method according to claim 8, further comprising applying a first regularization process to the updated global model to reduce a first bias toward server data, or applying a second regularization process to the quantization model update to reduce a second bias toward device data.

10. The processor implementation method according to claim 8, further comprising repeating the receiving, generating, and transmitting for a plurality of training rounds, wherein the quantization is performed in a subset of the training rounds based on the device processing capability.

11. It is a device, At least one memory, At least one processor coupled to the at least one memory, The at least one processor is The server quantizes the global model, and the global model is quantized at multiple different quantization levels for each of the one or more subnetwork models in order to generate one or more quantized subnetwork models, and the one or more subnetwork models are assigned to one or more of a plurality of devices according to the device processing capacity. The server distributes the quantization subnetwork model to at least one of the multiple devices. The server receives model updates from at least one device based on local data. A device configured such that the server generates a global model updated according to an aggregate function based on the model updates from each of the at least one of the devices.

12. The apparatus according to claim 11, wherein the at least one processor is further configured to fine-tune the updated global model using publicly available data from a backup device.

13. The apparatus according to claim 11, wherein the at least one processor is further configured by the server to quantize the updated global model at a plurality of different quantization levels for each of the one or more subnetwork models to generate one or more quantized updated subnetwork models.

14. The apparatus according to claim 11, wherein the at least one processor is further configured to apply a regularization process to the updated global model in order to reduce bias toward server data.

15. The apparatus according to claim 11, wherein the model update from the at least one device is a quantization model update based on quantization by the at least one device.

16. It is a device, At least one memory, At least one processor coupled to the at least one memory, The at least one processor is A device receives a quantized subnetwork model from a server, the quantized subnetwork model corresponds to a global model, the global model is quantized at multiple different quantization levels for each of the one or more subnetwork models in order to generate one or more quantized subnetwork models, and the one or more subnetwork models are assigned to one or more of a plurality of devices according to the device's processing capacity. The aforementioned device generates a model update of the quantized subnetwork model based on local data, An apparatus configured such that the device transmits the model update to the server, and the server generates an updated global model based on the model update.

17. The apparatus according to claim 16, wherein the updated global model is generated using an aggregate function based on the model update by the device.

18. The apparatus according to claim 16, wherein the at least one processor is further configured by the device to perform quantization of the model update in order to generate the quantized model update, and the quantized model update is used to generate the updated global model.

19. The apparatus according to claim 18, wherein the at least one processor is further configured to apply a first regularization process to the updated global model to reduce a first bias toward server data, or to apply a second regularization process to the quantization model update to reduce a second bias toward device data.

20. The apparatus according to claim 18, wherein the at least one processor is further configured to repeat the receiving, generating, and transmitting for a plurality of training rounds, and the device performs the quantization in a subset of the training rounds based on the device processing capability.

Images