Activation-based machine learning model parameter quantization

Abstract

Examples described herein relate to quantization of machine learning model parameters. A set of candidate quantization configurations is identified for a component of a trained machine learning model. Each candidate quantization configuration is applied to the component. The trained machine learning model is executed on an input dataset to obtain a candidate output value for each candidate quantization configuration. A loss is determined for each candidate quantization configuration based on a comparison between the candidate output value and a reference output value for the candidate quantization configuration of the component. One of the candidate quantization configurations is selected for the component based on the determined losses associated with the set of candidate quantization configurations. At least a portion of the trained machine learning model is quantized using the selected candidate quantization configuration for the component.

Description

Priority requirements

[0001] This application claims the benefit of priority to U.S. Patent Application Serial No. 18 / 587,614, filed February 26, 2024, which is incorporated herein by reference in its entirety. Technical Field

[0002] The topics disclosed in this article generally relate to machine learning. More specifically, but not exclusively, the topic relates to the quantization of machine learning model parameters. Background Technology

[0003] Quantization is a technique used to reduce the precision of the numerical representation of parameters in a machine learning model, such as weights in a neural network. For example, by converting weights from a floating-point representation (e.g., 32-bit floating-point) to a lower-precision format (e.g., 8-bit integers), quantization reduces the memory footprint and computational requirements associated with the machine learning model. This allows machine learning models to be deployed more efficiently in resource-constrained environments, such as mobile devices, embedded systems, edge devices, or wearable devices (e.g., extended reality (XR) devices). However, quantization also has certain drawbacks, such as a potential loss of model accuracy and a reduction in inference performance. Attached Figure Description

[0004] In accompanying drawings that are not necessarily drawn to scale, similar reference numerals can describe similar parts in different views. To facilitate identification of any particular element or action being discussed, one or more of the highest-order digits in the reference numerals indicate the drawing number in which that element was first introduced. Some non-limiting examples are shown in the figures in the accompanying drawings:

[0005] Figure 1 It is a diagrammatic representation of a networked environment in which the content of this disclosure can be deployed, based on some examples.

[0006] Figure 2 This is a block diagram illustrating some components of a model quantization system based on some examples.

[0007] Figure 3 This is a flowchart illustrating the operation of a method suitable for selecting one or more quantization configurations for a trained machine learning model, based on some examples.

[0008] Figure 4 A quantization pipeline based on some examples is illustrated in a graphical manner.

[0009] Figure 5 It is a Unified Modeling Language (UML) diagram that outlines the structure and relationships of “Activation-Aware Quantization Classes” and association functions based on some examples.

[0010] Figure 6 The processing system is illustrated using diagrams based on some examples.

[0011] Figure 7 An XR device is illustrated using diagrams based on some examples.

[0012] Figure 8 The machine learning pipeline is illustrated using diagrams based on some examples.

[0013] Figure 9 The training and use of machine learning programs are illustrated using diagrams based on some examples.

[0014] Figure 10 It is a graphical representation of a machine in the form of a computer system, based on some examples, within which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein.

[0015] Figure 11 It is a block diagram illustrating an example of a software architecture that can be implemented therein. Detailed Implementation

[0016] Parameter quantization of machine learning models can be performed in various scenarios. For example, when deploying a machine learning model to hardware with limited resources (e.g., limited memory or battery resources), its parameters can be quantized before deployment to reduce the machine learning model's memory footprint and / or computational requirements. This could be the case, for example, in edge devices or wearable devices where parameters need to be stored in limited on-chip memory to provide satisfactory inference performance.

[0017] For example, in a typical machine learning model, the (unquantized) parameters developed during training can be represented using 32-bit floating-point numbers. Quantization can then involve converting these numbers into a lower-precision format, such as 16-bit floating-point numbers, 8-bit integers, or even lower (e.g., converting to 4-bit or 6-bit formats) for storage. This conversion reduces the overall size of the machine learning model and the amount of memory resources and / or computation required for inference.

[0018] Converting the parameters of a machine learning model to a lower-precision representation may result in reduced parameter fidelity, which can negatively impact the performance of the machine learning model (e.g., it may degrade the model's ability to capture or accurately identify subtle patterns in the data during inference). Therefore, it may be desirable to select an optimal or near-optimal quantization configuration (e.g., bit settings) within the environmental and constraints of the relevant machine learning model, hardware, and inference application scenario.

[0019] For example, when a weight tensor is to be quantized to 8 bits, the bit settings for the exponent and mantissa (e, m) can use different combinations of values, such as (0, 7), (2, 5), (3, 4), or (4, 3) (7 bits in total, excluding a single fixed sign bit). Choosing the combination that has the least adverse impact on the performance or accuracy of a machine learning model when compared to its performance or accuracy when running with unquantized parameters is technically challenging.

[0020] The examples described in this paper utilize a relatively small amount of input data (e.g., mini-batch samples) to ensure that quantization configurations can be allocated efficiently, resulting in higher accuracy when the machine learning model is eventually deployed on the relevant hardware. Instead of referencing the parameters (e.g., weights) themselves, the examples allow for the automatic evaluation of the loss associated with a given quantization configuration when applied to machine learning model parameters, by referencing the output generated using that configuration (e.g., feature map values ​​or activations). In other words, instead of selecting, for example, bit settings based on the error introduced by parameter values ​​compared to their original values, the examples described in this paper make such selections based on errors or biases in intermediate outputs generated within the machine learning model. Therefore, the examples described in this paper utilize "activation-based" or "activation-aware" quantization.

[0021] In some examples of this disclosure, various candidate quantization configurations (e.g., different possible bit settings) are automatically evaluated to select one or more optimal or near-optimal candidate quantization configurations. Such evaluation and selection can be performed for each of multiple components of a machine learning model (e.g., layers or channels within a neural network). The examples described herein allow for the automatic selection of an appropriate number of exponent and mantissa bits for the weights of a given component of a neural network, such that the loss compared to a reference output is minimized while adapting to the selected bit precision.

[0022] Example methods include identifying a set of candidate quantization configurations for components of a trained machine learning model. As used herein, the term "quantization configuration" refers to one or more rules, parameters, or settings that control or determine how quantization should be performed for a machine learning model or one or more specific components thereof. For example, a quantization configuration may indicate the number of bits to which a parameter of a machine learning model will be reduced (e.g., a level of precision), the allocation of these bits in various aspects of the parameter (e.g., the exponent and mantissa in a floating-point representation), the quantization scheme or algorithm, the quantization granularity (e.g., per-layer or per-channel configuration), and / or the method of rounding or truncating values ​​to fit specified constraints.

[0023] The candidate quantization configurations identified by the system as described herein may include, for example, different possible bit settings for components. As used herein in the context of machine learning models, the term "component" refers to a part, region, or substructure within a machine learning model that performs one or more specific functions or represents one or more specific aspects of the model's architecture. Components may include, for example, layers (e.g., convolutional layers, fully connected layers, or activation layers), nodes, or neurons within those layers in a neural network. For example, in a convolutional neural network (CNN) for image recognition, a component may include a convolutional layer that detects edges or texture within an input image. A component may also be a sub-element of a layer, such as a channel.

[0024] This method may include: for each candidate quantization configuration in a set of candidate quantization configurations, applying the candidate quantization configuration to a component, and after applying the candidate quantization configuration, executing a trained machine learning model on an input dataset (e.g., a small set of input samples) to obtain candidate output values ​​for the candidate quantization configuration. The input dataset may include unlabeled sample data. In the context of image processing, the input dataset may include unlabeled sample images. In some examples, the input dataset is a relatively small sample, such as less than 100 images, less than 50 images, or even less than 40 images. Therefore, the examples described herein enable the evaluation of candidate quantization configurations without the need to process large datasets.

[0025] In some examples, the input dataset includes data representing the data that the trained machine learning model is expected to process during inference. For example, in the case where the machine learning model is trained to identify different breeds of cats in images, the input dataset may include images of cats.

[0026] The method may also include determining a loss associated with a candidate quantization configuration based on a comparison between a reference output value and a candidate output value for the component. Applying a candidate quantization configuration to the component allows a trained machine learning model to be executed with the component having quantized parameters to obtain candidate output values, while a reference output value is obtained by executing the trained machine learning model with the component having unquantized parameters (e.g., the original parameters before quantization). In this way, candidate output values ​​associated with quantized parameters can be compared with reference output values ​​associated with unquantized parameters. In this context, "parameters" can be internal variables learned, adjusted, or refined from training data. For example, in a neural network, parameters include weights connecting neurons across different layers or biases that adjust the activation level of each neuron. The techniques described herein involve transforming or converting an unquantized version of such parameters into its quantized version.

[0027] In the context of comparing "candidate" values ​​with "reference" values, the term "output" refers to data generated as a result of processing by components within a trained machine learning model. Such "output" is an intermediate output within the machine learning model, as opposed to the "final" output (e.g., the final inference result). Output values ​​can include activation or feature map values ​​within the machine learning model. Output values ​​can also refer to values ​​to which a threshold function has been applied.

[0028] The method may further include: selecting candidate quantization configurations from the set of candidate quantization configurations based on a determined loss associated with the set of candidate quantization configurations, for components of the trained machine learning model. For example, the loss associated with a particular candidate quantization configuration may be determined based on a loss function, such that selecting a candidate quantization configuration from the set of candidate quantization configurations includes: detecting that the selected candidate quantization configuration produces the minimum value of the loss function for the components of the trained machine learning model.

[0029] For example, a loss function compares a value matrix produced by a neural network layer using quantized weights with a value matrix produced by the same neural network layer using unquantized weights to determine how or to what extent quantization “distorts” the output (compared to the original output or baseline output). Examples of loss functions include L1 loss (also known as mean absolute error or MAE) and L2 loss (also known as mean squared error or MSE), as described in more detail elsewhere in this disclosure. However, losses can be determined in various ways because various techniques can be used to compute the error or difference between the outputs (e.g., reference values ​​versus candidate values).

[0030] In some examples, at least a portion of a trained machine learning model is quantized using a selected candidate quantization configuration for a component. Quantization of a trained machine learning model can include quantizing its parameters. Therefore, the examples described herein specifically relate to post-training quantization, which involves quantizing parameters after the model has been trained.

[0031] The above description pertains to selecting a quantization configuration for a single component of a trained machine learning model. Note that a similar approach can be used to select a quantization configuration for multiple components or even the entire machine learning model. In some examples, this approach allows for the selection of different quantization configurations for different components. Therefore, a trained machine learning model can ultimately be quantized using one quantization configuration for one or more components, and one or more other quantization configurations for one or more other components.

[0032] In some examples, the method includes determining a first quantization configuration and a second quantization configuration, the first quantization configuration including a first bit setting for quantizing the parameters of a first layer of the neural network, and the second quantization configuration including a second bit setting for quantizing the parameters of a second layer of the neural network. The bit settings for different layers can be the same or different, depending on the results of the candidate output value evaluation.

[0033] The example system considers a threshold function within a trained machine learning model. For example, in the case where the component being evaluated includes a layer of a neural network associated with a threshold function, the threshold function can be applied to obtain at least a subset of candidate output values ​​and at least a subset of reference output values. In other words, the system can evaluate the output after applying a threshold function or, for example, another function designed to introduce nonlinearity into the output.

[0034] As mentioned above, the parameters to be quantized can include weights. Each weight can be quantized as a combination of exponent and mantissa. In some cases, the number of exponents or mantissas is zero. Therefore, as used herein, the phrase "combination of exponent and mantissa" can include combinations where the number of exponents or mantissas is zero. For example, where 7 bits are available for both exponent and mantissa, the combination (0, 7) is still "combination of exponent and mantissa" even if the number of exponents is zero.

[0035] The method may include receiving a quantization request from a user equipment, the quantization request including a selected bit precision for quantization components. A set of candidate quantization configurations may then be identified based on the selected bit precision, wherein the set of candidate quantization configurations includes different combinations of exponent bits and mantissa bits that satisfy the selected bit precision.

[0036] Quantization of a trained machine learning model can include generating a new instance of the trained machine learning model, which includes a selected candidate quantization configuration for the components. In some examples, the new instance is generated (and in some examples, returned) in response to receiving a quantization request from a user device. The quantization request can include input parameters, such as the selected bit precision.

[0037] In some examples, the method includes storing the quantized parameters in the on-chip memory of the processing device. For example, the processing device may have on-chip static random access memory (SRAM) in which the quantized parameters are stored. On-chip storage of parameters quantized using the techniques described herein can present several benefits. For example, on-chip storage can result in faster retrieval of model parameters during inference, higher energy efficiency, and / or lower latency. The techniques described herein can make fitting model parameters within the limited space of on-chip memory feasible or possible, while balancing memory savings with model accuracy.

[0038] This method may include performing batch normalization folding before quantizing the trained machine learning model. For example, the system may automatically perform batch normalization folding before obtaining candidate output values ​​and reference output values ​​for evaluation.

[0039] In some examples, the method includes generating an output that includes the selected candidate quantization configuration and transmitting the output to a user device. In this way, the quantization configuration determined to be optimal or near-optimal can be transmitted for downstream use.

[0040] The techniques described in this paper can be implemented in or benefit different types of processing systems. In some examples, trained machine learning models are quantized for deployment on event-based neural processors with multiple processing clusters. In other examples, trained machine learning models are quantized for deployment on computing devices that include such event-based neural processors (e.g., XR devices or other wearable devices). XR devices can be augmented reality (AR) or virtual reality (VR) devices. Therefore, quantization can be performed to allow the deployment of trained machine learning models to devices with memory constraints or devices designed for low power consumption.

[0041] Based on some examples, the systems or methods described herein improve the functionality of computing devices by providing enhanced techniques for selecting and applying quantized configurations. Due to the implementation of these methods, computing resources can be utilized more efficiently or at a lower cost. Examples of such computing resources include: processor cycles, network traffic, memory usage, data storage capacity, power consumption, network bandwidth, and cooling capacity.

[0042] The examples described in this paper provide technical solutions to technical problems. For example, a technical problem with current machine learning model deployments may be the need for significant computational resources to operate large, complex models, especially in resource-constrained environments such as mobile devices, wearables, or edge computing platforms. These models (typically configured with high-precision parameters) can have relatively high requirements in terms of memory and processing power, leading to inefficiency and practical limitations. The examples described in this paper provide a technical solution to this problem by systematically selecting optimal or near-optimal quantization configurations for each of one or more components of the model. By doing so, parameter precision can be reduced in a controlled and / or selective manner, thereby reducing model size and computational requirements without substantial or unacceptable loss of accuracy, enabling efficient deployment on hardware with limited resources.

[0043] Other technical problems may involve focusing on minimizing errors in parameter values ​​without adequately considering their impact on model output, potentially leading to suboptimal performance, especially in deep learning models where some functions introduce nonlinearities. The examples described in this paper address or mitigate this problem via an "activation-aware" approach, where quantization is guided by a loss associated with feature map values ​​(e.g., activations of a neural network). This approach can be specifically designed to account for nonlinearities when comparing candidate and reference outputs. This ensures that quantization substantially preserves the model's functional behavior, maintaining reliable predictive performance even after parameter precision is reduced. By focusing on activation, the examples described in this paper provide a technical solution that more closely aligns quantization with the operational characteristics of the model.

[0044] Other technical challenges may involve automating quantization processes while allowing for user input or interaction as desired. Quantization pipelines can be complex and time-consuming, requiring expertise in both machine learning and hardware optimization. The examples described in this paper automate various aspects of a quantization pipeline and provide user-interactive components that make it easy for users to specify preferences and goals or monitor progress. For example, a user-friendly interface can be provided to guide the selection of quantization configurations based on user-defined criteria.

[0045] Figure 1This is a block diagram illustrating an example interactive system 100 for facilitating interactions over a network, such as exchanging text messages, making text, audio, and video calls, or playing games. The interactive system 100 includes multiple user systems 102, each of which hosts multiple applications including an interactive client 104 (as an example of an interactive application) and other applications 106. Each interactive client 104 is communicatively coupled to other instances of the interactive client 104 (e.g., hosted on corresponding other user systems 102), server systems 110, and third-party servers 112 via one or more communication networks including a network 108 (e.g., the Internet). The interactive client 104 can also communicate with the locally hosted applications 106 using an application programming interface (API).

[0046] Each user system 102 may include one or more user devices, such as mobile devices 114, head-mounted wearable devices 116, and computer client devices 118, which are communicatively connected to exchange data and messages.

[0047] Interactive client 104 interacts with other interactive clients 104 and with server system 110 via network 108. The data exchanged between interactive clients 104 (e.g., interaction 120) and between interactive client 104 and server system 110 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0048] Server system 110 provides server-side functionality to interactive client 104 via network 108. While some functions of interactive system 100 are described herein as being performed by interactive client 104 or by server system 110, the location of certain functions—whether within interactive client 104 or server system 110—may be a design choice. For example, it may be technically preferred that certain technologies and functions are initially deployed within server system 110, but later migrated to interactive client 104, where user system 102 has sufficient processing power.

[0049] Server system 110 supports various services and operations provided to interactive client 104. Such operations include sending data to and receiving data from interactive client 104, and processing data generated by interactive client 104. This data may include message content, device information, geolocation information, content enhancements (e.g., filters or overlays), message content persistence conditions, entity relationship information, machine learning model data, and live event information. Data exchange within interactive system 100 can be activated and controlled via functions available through the user interface of interactive client 104.

[0050] Specifically, turning to server system 110, API server 122 is coupled to interactive system server 124 and provides it with a programming interface, making the functionality of interactive system server 124 accessible to interactive client 104, other applications 106, and third-party server 112. Interactive system server 124 is communicatively coupled to database server 126, thereby facilitating access to database 128, which stores data associated with the interactions processed and other functions performed by interactive system server 124. Similarly, web server 130 is coupled to interactive system server 124 and provides a web-based interface to interactive system server 124. To this end, web server 130 handles incoming network requests via Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0051] API server 122 receives and sends interactive data (e.g., command and message payloads) between interactive system server 124 and user system 102 (as well as interactive client 104 and other applications 106) and third-party server 112. Specifically, API server 122 provides a set of interfaces (e.g., routines and protocols) that can be invoked or queried by interactive client 104 and other applications 106 to activate the functionality of interactive system server 124. API server 122 exposes various functions supported by interactive system server 124, including account registration; login functionality; sending interactive data from one interactive client 104 to another interactive client 104 via interactive system server 124; transferring media files (e.g., images or videos) from interactive client 104 to interactive system server 124; setting media data sets (e.g., stories); retrieving the friend list of users in user system 102; retrieving messages and content; adding and deleting entities (e.g., friends) in entity relationship graphs; locating friends within entity relationship graphs; opening application events (e.g., related to interactive client 104); and utilizing artificial intelligence and machine learning capabilities.

[0052] Interactive system server 124 hosts multiple systems and subsystems. For example, interactive system server 124 provides multiple interactive facilitation systems 132, which may include, for example, an image processing system, an enhancement system, an enhancement creation system, a communication system, and / or a user management system. The image processing system can provide various functions that enable users to capture and enhance (e.g., annotate or otherwise modify or edit) media content. The enhancement system can provide functions related to the generation and publication of enhancements (e.g., filters or media overlays)—also known as augmented reality (AR) effects—for images or videos captured in real time by the camera device of user system 102 or retrieved from the memory of user system 102.

[0053] The augmented creation system can support AR developer platforms and includes applications that enable content creators (e.g., artists and developers) to create and publish interactive clients 104, such as AR experiences. The augmented creation system can provide content creators with a library of built-in features and tools, including, for example, custom shaders, tracking technologies, and templates.

[0054] The communication system can be responsible for enabling and processing various forms of communication and interaction within the interactive system 100, and may include a messaging system, an audio communication system, and / or a video communication system. The user management system can be operationally responsible for the management of user data and profiles, as well as the maintenance of entity information (e.g., stored in entity tables or entity diagrams) related to users and relationships between users of the interactive system 100.

[0055] The interaction system server 124 also provides an artificial intelligence and machine learning system 134. The artificial intelligence and machine learning system 134 provides various services to different subsystems within the interaction system 100, including one or more interaction facilitation systems 132. For example, the artificial intelligence and machine learning system 134 operates in conjunction with the interaction facilitation system 132 to analyze images and extract information such as objects, text, or faces. This information can then be used to enhance, filter, or manipulate (e.g., apply AR effects) the images. As another example, the artificial intelligence and machine learning system 134 can be used to perform object tracking or detection (e.g., pose recognition) to facilitate the operation of the user system 102 in the form of a head-mounted wearable device 116.

[0056] Artificial intelligence and machine learning system 134 can be used by interaction facilitation system 132 to generate augmented content and AR experiences, such as adding virtual objects or animations to real-world images. The communication system of interaction facilitation system 132 can use artificial intelligence and machine learning system 134 to analyze communication patterns and provide insights into how users interact with each other, as well as provide intelligent message classification and tagging, such as classifying messages based on sentiment or topic. Artificial intelligence and machine learning system 134 can also provide chatbot functionality to interactions 120 between user systems 102 and between user systems 102 and interaction server system 110. Artificial intelligence and machine learning system 134 can provide various generative capabilities (e.g., allowing users to generate text, image, or video content based on prompts). Artificial intelligence and machine learning system 134 can work with the communication system of interaction facilitation system 132 to provide speech recognition and natural language processing capabilities.

[0057] In some examples, the artificial intelligence and machine learning system 134 is used for training machine learning models, as well as for deploying and managing machine learning models. For this purpose, the artificial intelligence and machine learning system 134 includes a model management system 136 and a model quantization system 138.

[0058] The model management system 136 serves as the central hub within the server system 110, overseeing the lifecycle of various machine learning models. The model management system 136 can be responsible for model storage, version control, distribution, and / or monitoring. The model management system 136 can communicate with the model quantization system 138 to receive quantized models. The model management system 136 can be designed to ensure that up-to-date and efficient versions of models are readily available for deployment. The model management system 136 can also be responsible for testing, for example, comparing the performance of different model iterations or instances.

[0059] The machine learning model can be used in the context of the interactive client 104, or more generally by the user system 102. In some examples, after successful quantization and validation, the model management system 136 coordinates with the interactive client 104 running on the user system 102 to deploy the quantized model for inference tasks. The model can be accessed via network 108 or run on the user system 102. The model management system 136 can send updates to the interactive client 104, which can be part of various applications, such as image recognition, natural language processing, or other AI-driven functions.

[0060] In some examples, the model management system 136 is designed to facilitate the direct deployment of machine learning models to user system 102, where the machine learning models can be executed on-chip. This approach may be advantageous for applications requiring real-time processing, such as AR, digital assistants, real-time object detection, or real-time audio monitoring. In some examples, user system 102 employs an event-driven processing system, such as those described elsewhere in this disclosure. Figure 6 The processing system 600 is used to process neural networks.

[0061] By pushing the quantized and optimized model to user system 102, model management system 136 can ensure that computational workloads are processed locally, thereby minimizing latency and reliance on persistent connectivity. On-chip execution of the model also saves bandwidth and reduces server load, which can be critical when scaling to a large number of users. Furthermore, running the model on-chip enables enhanced user privacy and security, as sensitive data can be processed locally without being transferred to an external server.

[0062] Model quantization system 138 is responsible for processing and / or tuning model parameters, for example, to optimize or adapt model parameters for storage and computational efficiency. This may involve reducing the predictions of the machine learning model's parameters. Model quantization system 138 can identify possible quantization configurations, select an appropriate quantization configuration to apply to the model, evaluate the impact of quantization on model performance, and perform quantization. In some examples, model quantization system 138 receives quantization requests from users (e.g., via network 108) and returns a new instance of a machine learning model with quantized parameters (e.g., to model management system 136 or to user system 102). Model quantization system 138 can be designed to handle different types of machine learning models (e.g., different types of neural networks) and can be customized to support a variety of deployment environments, from high-performance servers to resource-constrained edge devices. Model quantization system 138 can provide compiler features to facilitate the compilation of machine learning models into a format suitable for running in a specific hardware environment.

[0063] In some examples, to efficiently store model weights on the hardware associated with user system 102, model quantization system 138 may utilize a quantization method that involves representing values ​​in the weight tensor using a sign bit, a number of exponent bits (e), and a number of mantissa bits (m). This can be referred to as a “bit setup” associated with the model or a portion of the model. An example of a numerical representation format is the “AdaptifFloat” representation format described by Tamble et al. in “AdapttivFloat: A Floating-Point Based DataType for Resilient Deep Learning Inference” (arXiv:1909.13271v3). Note that in some cases, the number of exponent bits or the number of mantissa bits can be zero.

[0064] One quantization method automatically selects bit settings based on the error introduced by the weight values ​​compared to their original values. However, instead of following this approach, the examples described herein provide a technical solution involving minimizing the error of the output (e.g., feature map) associated with the quantized tensor compared to the original output. For example, in the context of a neural network, model quantization system 138 is designed to minimize the difference between the values ​​of the feature maps of layers (including activations or thresholds, if present) with and without quantized weights. Quantizing the model using such bit settings can produce increased accuracy.

[0065] Referring again more broadly to interactive system 100, interactive system 100 can therefore embody multiple subsystems, which are supported on the client side by interactive client 104 and on the server side by interactive system server 124. In some examples, one or more of these subsystems are implemented as microservices. A microservice subsystem (e.g., a microservice application) can have components that enable it to operate independently and communicate with other services. Example components of a microservice subsystem may include:

[0066] Functional logic: Functional logic implements the functions of the microservice subsystem and represents the specific capabilities or functions provided by the microservice.

[0067] API Interfaces: Microservices can use lightweight protocols such as Representational State Transfer (REST) ​​or messaging to communicate with other components through well-defined APIs or interfaces. API interfaces define the inputs and outputs of a microservice subsystem, and how a microservice subsystem interacts with other microservice subsystems of the interactive system 100.

[0068] Data storage: The microservice subsystem can be responsible for its own data storage, which can be in the form of a database, cache, or other storage mechanisms (e.g., using database server 126 and database 128). This allows the microservice subsystem to operate independently of other microservices in the interactive system 100.

[0069] Service discovery: Microservice subsystems can find and communicate with other microservice subsystems in the interacting system 100. The service discovery mechanism enables microservice subsystems to locate and communicate with other microservice subsystems in a scalable and efficient manner.

[0070] Monitoring and logging: Microservice subsystems may need to be monitored and logged to ensure availability and performance. Monitoring and logging mechanisms enable the tracking of the health and performance of microservice subsystems.

[0071] In some examples, the interactive system 100 adopts a monolithic architecture, a service-oriented architecture (SOA), a function-as-a-service (FaaS) architecture, or a modular architecture.

[0072] Figure 2 This is shown based on some examples. Figure 1 A block diagram of some components of the model quantization system 138. In Figure 2In the figure, the model quantization system 138 is shown as including a user interface component 202, a batch normalization folding component 204, a sample data loading component 206, a quantization configuration identification component 208, a quantization configuration evaluation component 210, a quantization configuration selection component 212, a model quantization component 214, and an export component 216.

[0073] User interface component 202 enables communication between a user (e.g., a user of user system 102) and model quantization system 138. User interface component 202 may provide one or more interfaces that allow a user to input specific quantization requirements, select a model for quantization, provide sample data, and / or configure quantization parameters. For example, a user may specify the desired bit precision of model parameters (or multiple different bit precisions for different components of a machine learning model), or choose between different quantization schemes provided by model quantization system 138. Model quantization system 138 may also provide an interface to present the user with options for reviewing and adjusting quantization configurations based on system recommendations or performance metrics.

[0074] In some examples, the user interface component 202 provides one or more APIs that serve as channels for transmitting user input, such as quantization parameters and model selection criteria, to other components of the model quantization system 138. For example, such APIs may be designed to handle a variety of user-specified parameters, including but not limited to the number of exponent and mantissa bits for floating-point quantization, the selection of layers, channels, or graphs within the neural network model for targeted quantization, and / or the selection of a loss function for evaluating quantization effectiveness.

[0075] It also allows users to upload input datasets for use as sample data to evaluate various quantization configurations. The user interface component 202 can also allow users to activate certain components of the model quantization system 138, such as the batch normalization folding component 204, which integrates batch normalization parameters into the model's weights before quantization. The user interface component 202 can provide a graphical user interface that allows users to track the status, for example, via a real-time progress indicator and receive notifications upon completion of quantization. The user interface component 202 can also allow users to download or export new instances of the model or its quantization parameters for downstream use.

[0076] In some examples, a user can explicitly choose to perform "activated-aware" quantization when sending a quantization request to the model quantization system 138. The model quantization system 138 can then automatically utilize the techniques described herein to select one or more optimal or recommended quantization configurations.

[0077] The model quantization system 138 can implement a batch normalization folding component 204 to handle, for example, integrating batch normalization layers into adjacent layers of a neural network. The model quantization system 138 can analyze the network architecture to identify batch normalization operations and compute the necessary adjustments to the weights and biases of surrounding layers to directly incorporate these operations. Therefore, the batch normalization folding component 204 can automatically reduce the computational complexity of the machine learning model and prepare it for more efficient quantization. For example, in a deep learning model, the batch normalization folding component 204 merges normalization parameters with convolutional filters, thereby eliminating the need for at least some separate normalization computations during inference.

[0078] The sample data loading component 206 is responsible for importing and / or managing the input datasets (e.g., sample data) required for quantization processing. In some examples, the sample data loading component 206 communicates with the user interface component 202 to obtain user-provided samples and loads them for analysis. In other examples, the sample data loading component 206 retrieves data from a storage component (e.g., ...). Figure 1 Sample data was retrieved from database 128.

[0079] In some examples, sample data is used to evaluate the performance impact of different quantization configurations. The sample data loading component 206 can handle the preprocessing and formatting of the sample data. For example, the sample data loading component 206 can automatically resize images, normalize values, and / or batch the input to facilitate efficient processing.

[0080] The quantization configuration identification component 208 is responsible for identifying or generating a set of potential quantization configurations for the parameters of a machine learning model. These potential quantization configurations are referred to herein as candidate quantization configurations. The quantization configuration identification component 208 may employ algorithms to explore various combinations of bit assignments for different parameters, such as the number of exponent bits and mantissa bits in a floating-point representation or the granularity of a fixed-point representation. For example, the quantization configuration identification component 208 may identify all possible or feasible bit arrangement permutations that satisfy certain constraints (e.g., user-selected bit precision or layer-specific constraints).

[0081] The quantization configuration identification component 208 can be configured to consider any user-provided specifications (e.g., selected bit precision), model architecture, and / or computational constraints of the target deployment environment to propose a range of quantization strategies. For example, the quantization configuration identification component 208 can identify configurations with different bit widths for the weights of convolutional layers, each configuration having its own trade-off between model size and expected accuracy.

[0082] The quantization configuration evaluation component 210 evaluates the performance of each candidate quantization configuration generated or identified by the quantization configuration identification component 208. This evaluation can be performed automatically by applying candidate configurations to a model and measuring the impact on accuracy using sample data provided or accessed by the sample data loading component 206. The quantization configuration evaluation component 210 can run or execute both a machine learning model with quantized parameters and a machine learning model without quantized parameters on the sample data to obtain a reference output and candidate outputs for comparison with the reference output.

[0083] The quantization configuration evaluation component 210 can utilize one or more metrics to quantify the performance deviation from the original unquantized model. For example, the quantization configuration evaluation component 210 can automatically apply a loss function to determine the effectiveness of each quantized configuration. The quantization configuration evaluation component 210 can evaluate each candidate quantization configuration by evaluating the output within the model (e.g., feature map values ​​(e.g., activations)) rather than evaluating the parameters (e.g., weights) themselves. Where the model includes nonlinear transformations (e.g., nonlinear transformations caused by activation functions and / or threshold functions), the quantization configuration evaluation component 210 can consider the resulting values ​​after applying such nonlinear transformations.

[0084] The quantization configuration selection component 212 can be configured to select the most suitable quantization configuration (or set of quantization configurations, e.g., when different configurations are to be applied to different components) based on the evaluation of candidate quantization configurations by the quantization configuration evaluation component 210. For example, the quantization configuration selection component 212 can employ one or more decision criteria, such as minimizing the loss function, maintaining a threshold level of accuracy, or adhering to specific resource constraints. The quantization configuration selection component 212 can consider user objectives and the operating environment to select a configuration that optimally balances, for example, model performance with the practical requirements of the deployment environment.

[0085] In some examples, the model quantization system 138 is configured to assign bit settings (from multiple candidates) associated with a quantized tensor that produces the lowest overall error (e.g., L1 or L2 loss) on the output of a particular layer (e.g., feature map values) compared to the output of that layer when run using an unquantized version of that tensor. In some examples, the same sample data is used to obtain the output of both the quantized and unquantized (e.g., raw) scenes. As mentioned above, in some cases, if a thresholding layer follows a layer, the output of the thresholding layer can be examined and compared instead of the outputs of previous layers.

[0086] Model quantization component 214 can operate as an execution engine within model quantization system 138, applying the selected quantization configuration to a machine learning model. For example, model quantization component 214 can transform the parameters (e.g., weights and biases) of a trained machine learning model from their original high-precision format to a lower-precision format specified by the selected configuration (e.g., chosen from multiple candidate quantization configurations). Model quantization component 214 can handle various aspects of quantization, such as bit setting implementation, rounding, truncation, or re-encoding of parameter values.

[0087] Model quantization component 214 can compile a final version of a machine learning model, which is a new instance of the original trained machine learning model (as a result of its quantization parameters). In some examples, model quantization component 214 is provided by a dedicated compiler component that transforms the machine learning model into a version that can be executed on a specific hardware device. Example features of such a compiler component are described elsewhere in this document.

[0088] Export component 216 can be used to facilitate the distribution of quantized models or their parameters. Export component 216 can package model parameters and associated metadata into a format suitable for the target platform (e.g., target hardware). Export component 216 can handle tasks such as saving the quantized model to a file, transferring the model to a specified location, or integrating the model into an application or service. Export component 216 can facilitate the on-chip deployment of trained and quantized machine learning models, for example, on edge devices or wearable devices (e.g., [missing information]). Figure 1 The head-mounted wearable device 116. As mentioned elsewhere in this disclosure, models quantified in this way can be deployed to event-driven processing systems (e.g., Figure 6 The user system 102 of the processing system 600. The exported parameters can also be used for other downstream operations, such as "activation-aware" training of one or more other machine learning models.

[0089] Therefore, the example described in this paper differs from techniques that only focus on minimizing the error between the quantized parameter values ​​and the original parameter values. Instead, the model quantization system 138 can evaluate the impact of quantization on the model output (e.g., intermediate outputs such as feature map values). This approach can be beneficial in complex models such as deep neural networks, where the relationship between parameters and output may be non-linear, and where maintaining the integrity of activations may be important for maintaining performance.

[0090] In some examples, Figure 2At least some of the components shown are configured to communicate with each other to implement the aspects described herein. One or more of the components described herein may be implemented using hardware (e.g., one or more processors of one or more machines) or a combination of hardware and software. For example, a component described herein may be implemented by a processor configured to perform the operations described herein for that component. Furthermore, two or more of these components may be combined into a single component, or the functionality described herein for a single component may be further divided among multiple components. Additionally, according to various examples, the components described herein may be implemented using a single machine, database, or device, or may be distributed across multiple machines, databases, or devices.

[0091] Figure 3 This is a flowchart illustrating the operation of method 300, suitable for selecting one or more quantization configurations for a trained machine learning model, based on some examples. The aspects of method 300 can be described by way of example rather than limitation. Figure 1 and Figure 2 The model quantization system 138 is executed by one or more components. Therefore, the model quantization system 138 is as follows: Figure 3 The description uses it as an example. However, the model quantization system 138 is a non-limiting example of a system that can perform method 300, and it will be understood that method 300 can also be performed using one or more other systems, components, devices, or architectures.

[0092] exist Figure 3 In method 300, the parameters of a trained machine learning model (presented as an example of a trained neural network) are quantized to reduce the overall size of the trained neural network before deployment. The individual components of the trained neural network analyzed for quantization are the corresponding layers of the trained neural network. Each layer has its own parameters (e.g., weights generated by the training process).

[0093] Method 300 begins at the start loop element 302 and proceeds to operation 304, where the model quantization system 138 obtains a reference output (e.g., using the quantization configuration evaluation component 210). In this example, the reference output is a reference feature map value for each selected quantized layer of the trained neural network.

[0094] Typically, feature map values ​​are outputs generated by applying a set of learned parameters (such as the weights and biases of a neural network) to a given input within a specific component of the model. For example, in the context of CNNs, feature maps are the result of filters convolving over input data, capturing patterns or features at different locations in the input. In image processing, for instance, feature map values ​​can represent the presence of certain textures or shapes at different locations in an image. These values ​​are crucial for the model to understand and abstract various aspects of the data as it propagates through the network. Feature map values ​​can represent activations, or the network's activation response to specific features or patterns detected in the input.

[0095] To obtain reference feature map values ​​in method 300, model quantization system 138 can execute a trained neural network on an input dataset (e.g., sample data) using unquantized parameters (e.g., raw weights determined for each layer during training). This produces outputs based on the unquantized parameters. These outputs are used as baselines or reference points and can therefore be referred to as reference feature map values. In some examples, where a layer is associated with a nonlinear transformation (e.g., a threshold function) or where a nonlinear transformation follows a layer, the reference feature map values ​​used for the layer are values ​​obtained after the nonlinear transformation.

[0096] At operation 306, model quantization system 138 identifies candidate quantization configurations (e.g., using quantization configuration to identify component 208). For example, for each component of a trained neural network with parameters to be quantized (in this case, each layer), model quantization system 138 identifies multiple possible bit settings that can be achieved while satisfying the selected bit precision.

[0097] Method 300 proceeds to operation 308, where model quantization system 138 obtains candidate feature map values ​​for each candidate quantization configuration (e.g., using quantization configuration evaluation component 210). For example, model quantization system 138 applies the relevant candidate quantization configuration to a trained neural network and then executes the trained neural network on the same input dataset used to obtain reference feature map values ​​to obtain outputs. Since these outputs are associated with a specific candidate quantization configuration and will be used for comparison with the reference output, these outputs are referred to as candidate feature map values.

[0098] In some examples, where a layer is associated with a nonlinear transformation (e.g., a threshold function) or where a nonlinear transformation follows a layer, the candidate feature map values ​​for the layer are values ​​obtained after the nonlinear transformation. In some examples, given that the same input dataset is used and nonlinearity is considered in both cases, the reference feature map values ​​and candidate feature map values ​​for a particular component (e.g., a layer) can be compared in an equivalent manner.

[0099] At operation 310, model quantization system 138 (e.g., using an appropriate loss function) determines the loss associated with each candidate quantization configuration. For example, for a specific layer of a trained neural network, model quantization system 138 uses quantization configuration evaluation component 210 to compute the deviation of its candidate feature map values ​​(generated using a specific candidate quantization configuration) from reference feature map values ​​generated for that same layer. This operation can be performed for each candidate quantization configuration relative to the relevant reference feature map values.

[0100] Then, at operation 312, the model quantization system 138 selects at least one candidate quantization configuration. For example, the model quantization system 138 uses quantization configuration selection component 212 to select a candidate quantization configuration for a layer based on the determined loss associated with the candidate quantization configurations identified for each layer under consideration. The model quantization system 138 can select the candidate quantization configuration that produces the minimum value of the selected loss function in each case.

[0101] Therefore, in some examples, different quantization configurations can be selected to quantize the parameters of different components (e.g., layers). However, in other examples, a single quantization configuration can be selected for use across multiple components of the neural network or for all components. In the latter case, the loss can be determined by comparing the overall, aggregated, or average loss between the reference output and the candidate outputs across multiple components.

[0102] Method 300 proceeds to operation 314, where model quantization system 138 uses one or more selected candidate quantization configurations to quantize the trained neural network (or a portion thereof). For example, the model quantization component 214 automatically quantizes the parameters (e.g., weights and biases) of the neural network using bit settings selected for each component (e.g., layer), which reduces the precision of the parameters and thus their storage size. The quantized parameters can then be converted into a format suitable for storage on the target hardware. This may involve packaging the parameters into a binary format consistent with the hardware's memory architecture.

[0103] The converted, quantized parameters can then be transferred to hardware (e.g., to a device such as...). Figure 6 The processing system 600 is then used to perform inference tasks. Furthermore, the resulting configuration can be exported using the exported component 216 and used in downstream quantization methods. Method 300 ends at the end loop element 316.

[0104] Figure 4 Figure 400 illustrates a quantization pipeline 402 according to some examples. Various aspects of the quantization pipeline 402 can be used by way of example and not limitation. Figure 1 and Figure 2The model quantization system 138 is implemented by one or more components. Therefore, the model quantization system 138 is described below. Figure 4 The description is used as an example. However, the model quantization system 138 is a non-limiting example of a system that can be used to implement the quantization pipeline 402, and it will be understood that one or more other systems, components, devices, or architectures may also be used.

[0105] The quantization pipeline 402 is used to automatically transform a trained machine learning model with unquantized parameters (trained model 404) into a new instance or version with quantized parameters (trained and quantized model 416). Figure 4 The quantization pipeline 402 is shown to include a batch normalization fold 406, a quantization configuration selection 408, and a parametric quantization 414. In some examples, instead of parametric quantization 414, the quantization pipeline 402 includes a configuration store 418, which is followed by a compilation operation 420, as described below.

[0106] In some examples, batch normalization folding 406 is a technique performed by the model quantization system 138 to integrate batch normalized layers of a neural network into adjacent layers, simplifying the network architecture and enhancing quantization processing. Batch normalization can stabilize and / or accelerate the training of deep neural networks by normalizing the input of each layer. However, for inference purposes, these normalization computations can be incorporated into the weights and biases of the preceding or following layer, effectively “folding” them into the network. This folding process reduces the number of operations and parameters, promoting more efficient quantization by minimizing computational overhead and potential quantization errors.

[0107] After performing batch normalization folding 406 relative to the trained model 404, the model quantization system 138 performs quantization configuration selection 408. In some examples, batch normalization may be irrelevant, and therefore the quantization pipeline 402 may begin with quantization configuration selection 408.

[0108] The quantization configuration option 408 can be used to automatically determine an efficient quantization strategy for the machine learning model in question. (See reference...) Figure 3 As described, the model quantization system 138 can evaluate various potential quantization configurations (referred to as candidate quantization configurations) and how each specified parameter in these candidate quantization configurations should be quantized. The model quantization system 138 can perform this evaluation with reference to certain constraints. For example, the overall bit precision can be pre-selected and fixed, wherein the model quantization system 138 is delegated the task of determining how to allocate bit types (e.g., exponent and mantissa bits) while satisfying the overall bit precision.

[0109] During quantization configuration selection 408, model quantization system 138 can use input dataset 410 to evaluate the impact of each candidate quantization configuration on model performance. For example, model quantization system 138 measures the loss or accuracy degradation when the trained model 404 is quantized according to each configuration. As mentioned, this can involve, for a specific component (e.g., a layer of a neural network), comparing candidate output values ​​(e.g., candidate feature map values) generated when applying quantized parameters according to a specific candidate quantization configuration with reference output values ​​(e.g., reference feature map values) generated when applying unquantized parameters (e.g., using a trained model 404 with its original parameter values).

[0110] As explained elsewhere in this disclosure, the input dataset 410 used by the model quantization system 138 may be a relatively small set of samples (e.g., approximately 100 unlabeled sample images in the case of image processing) used to obtain both reference output values ​​and corresponding candidate output values. The model quantization system 138 may select a configuration (e.g., a localized setting) that minimizes such loss or degradation for a particular component or set of components.

[0111] Based on this evaluation, the model quantization system 138 selects a candidate quantization configuration or a set of candidate quantization configurations (e.g., one configuration per layer or other component). For example, and as... Figure 4 As shown, the output of quantization configuration selection 408 can be the selected configuration 412. As an example, the selected configuration 412 can specify the number of exponent bits and mantissa bits to be used during parameter quantization 414 for each quantizable layer of the trained model 404. Therefore, the selected configuration 412 determines how parameter quantization 414 should be performed.

[0112] The model quantization system 138 can convert parameters (e.g., weights of a neural network) from a high-precision representation to a lower-precision format. Parameter quantization 414 can be performed differently for different components of the trained model 404, for example, by utilizing appropriate quantization configurations for individual layers and / or channels. For instance, key parameters that significantly affect the output of the trained model 404 can be quantized with higher precision, while less influential parameters can be represented with fewer bits to save memory and computational resources. The model quantization system 138 can automatically adjust bit settings (e.g., bit precision for a specific layer) based on such determinations, and / or can use user input to guide parameter quantization 414. In some examples, once parameter quantization 414 is complete, the quantization pipeline 402 terminates with the trained and quantized model 416.

[0113] In some examples, instead of performing parameter quantization 414 directly after determining the selected configuration 412, the model quantization system 138 performs configuration storage 418 to store the selected configuration 412. In subsequent stages, this can then be... Figure 4 The compilation operation 420 shown compiles the trained and quantized model 416. The model quantization system 138 may include a compiler component configured to convert the trained model 404 into a model that can be deployed on specific hardware (e.g., as shown in reference 1). Figure 6 The implemented neural processing unit is compatible with a format. Using the configuration provided by the quantization pipeline 402, the compiler component can correctly quantize the parameters of the relevant model.

[0114] In use, the user (e.g., using) Figure 1 User system 102) can call appropriate functions to trigger Figure 4 The quantization pipeline 402 is shown. For example, a user can invoke the "ActivationAwareQuantization" module hosted at the model quantization system 138 to specify model details and the total number of bits (n_bits). The model quantization system 138 then computes the bit settings, for example, a dictionary with the (quantizable) layer names as keys and their respective recommended bit settings (for quantizing relevant layers) as values. The bit settings can include tuples containing the number of exponent bits and the number of mantissa bits. For each quantizable layer, the output feature map can be compared with the original (reference) feature map of the unquantized model via L1 loss for each bit setting (e, m). For example, the bit setting corresponding to the lowest L1 loss can be stored in the configuration and passed to the "quantize_model" function.

[0115] L1 loss and L2 loss are non-restricted examples of loss functions that can be used to evaluate candidate outputs relative to a reference output. L1 loss is the average of the absolute differences between a candidate value and a reference value (for a specific candidate quantization configuration). L2 loss is the average of the squared differences between a candidate value and a reference value (for a specific candidate quantization configuration).

[0116] Therefore, users can use their user devices (e.g., Figure 1 The user system 102 submits a quantization request to the model quantization system 138 (e.g., a function call as described above). In response, the model quantization system 138 may generate and / or return suggested bit settings, or a new instance or version of the trained model 404 with quantized parameters (e.g., a trained and quantized model 416).

[0117] Figure 5This is a UML diagram 500 outlining the structure and relationships of the activation-aware quantization class 502 and the correlation function, based on some examples. The activation-aware quantization class 502 can be used as part of a system designed to perform quantization of machine learning models (e.g., model quantization system 138), specifically designed for Keras using TensorFlow. TM Customization of neural networks implemented via API. For example, the activation-aware quantization class 502 can be made accessible via the API to assist users (e.g., Figure 1 Efficient quantization is performed on user system 102 (users). Note that TensorFlow... TM The use of frameworks is an unrestricted example, and includes PyTorch, etc. TM Other frameworks can also be used to implement activation-aware quantization.

[0118] The Activation-Aware Quantization class 502 is associated with functions that facilitate the quantization of neural network parameters, thereby ensuring that the computational accuracy of the model is tuned to balance performance and computational efficiency. In some examples, accessing the API of the Activation-Aware Quantization class 502 provides users with the necessary tools to apply quantization strategies to their neural networks, thus simplifying the process of optimizing models for deployment in resource-constrained environments.

[0119] like Figure 5 As can be seen, the attributes of the ActivationAwareQuantization class 502 include: an integer (n_bits) specifying the number of bits to be used for quantization, and a boolean flag (per_channel) indicating whether quantization should be performed per channel (e.g., whether the quantization setting is shared across the entire layer or per input channel). The methods of the ActivationAwareQuantization class 502 include the class's constructor, which initializes the "ActivationAwareQuantization" object using the specified number of bits and the per-channel flag.

[0120] The methods for activating perceptual quantization class 502 also include:

[0121] "quantize_model", which will use TensorFlow Keras TM The model and dataset are taken as input, and a dictionary of quantized configurations is returned.

[0122] "get_bit_settings" calculates and returns the bit settings for the quantization model based on the provided dataset (as described elsewhere in this document, quantization can be performed downstream using compiler components, and in this case, the compiler components can use these bit settings).

[0123] "get_layer_bit_settings", which computes and returns the bit settings for a specific layer within the quantization model; and

[0124] The term "_l1_loss" calculates the L1 loss. This is the absolute difference between the true value (y_true) (also referred to as the reference output in this paper) and the predicted value (y_pred) (also referred to as the candidate output in this paper). In this example, the L1 loss is used as a metric to evaluate the quantization error.

[0125] The associated functions include model quantization function 504 and layer quantization function 506. Model quantization function 504 is responsible for applying the quantization configuration to the model. It receives additional parameters, such as the number of bits and whether to perform quantization per channel. In some examples, the "bit_settings" parameter is a dictionary where the layer name is the key and its bit settings (tuples containing the number of exponent bits and the number of mantissa bits, respectively) are the values. It returns a dictionary or "None" depending on whether the quantization was successful.

[0126] The layer quantization function 506 is used to quantize the individual layers of the model. It receives the following parameters: for example, the layer to be quantized, the number of bits to be used for quantization, and the number of exponent bits (e_bits) and mantissa bits (m_bits) if floating-point representation is used for quantization. The “per_channel” flag indicates whether quantization should be applied per channel.

[0127] Some inputs may be redundant or optional. For example, if the number of bits and the number of “e_bits” have already been specified, the number of “m_bits” may have only one value, and therefore its input may be redundant or optional.

[0128] Figure 6 A processing system 600 is illustrated according to some examples. The processing system 600 is configured for event-based processing tasks. In some examples, components of the processing system 600 are integrated into a single processing unit (e.g., a neural processing unit). For example, the processing system 600 may be implemented as a neural processing unit within a dedicated instruction processor (ASIP) designed to facilitate inference on cloud edge devices.

[0129] The processing system 600 includes multiple processing clusters 602 interconnected via a network 604. The network 604 serves as a message exchange network for exchanging messages, including event messages, instruction messages, configuration messages, or other messages, depending on the implementation. Therefore, messages may include instructions to perform computations, configuration instructions, or other data.

[0130] Network 604 includes nodes 606 that form interfaces with the various processing clusters 602, and links 608 between nodes 606. One or more other types of processing units, such as... Figure 6 One or more other processing units 610 shown may also be included in the processing system 600 and coupled to the network 604. For example, one or more other processing units 610 may include a digital signal processor, a general-purpose processor (e.g., a central processing unit (CPU)), a host processor, or a graphics processing unit (GPU).

[0131] In some examples, each processing cluster 602 has a message receiving facility for receiving event messages via network 604, and a message sending facility for sending event messages via network 604. Each of the processing clusters 602 may include one or more processing elements (not shown). Each processing element may be a neural processing element that (at least to some extent) mimics the behavior of biological neurons in a neural network processing environment, as further described below.

[0132] Each of the processing clusters 602 may include its own local memory or cache, thereby allowing for fast data access. For example, a neuromorphic state memory may store values ​​representing neuromorphic states associated with one or more processing elements. A processing element may have its own corresponding memory for storing its state or other information, or each processing cluster 602 may have memory for storing the state or other information of multiple processing elements.

[0133] In some examples, each processing cluster 602 has its own static random access memory (SRAM) (e.g., 256kB of SRAM). Neuromorphic states can be computed using, for example, 32-bit or 16-bit floating-point numbers.

[0134] The processing system 600 may also include an input facility 612 configured to receive input data. The input facility 612 may also selectively map messages. Therefore, the processing cluster 602 can not only send messages directly, but also indirectly redirect and broadcast their messages via the input facility 612. For example, the input facility 612 may be configured to receive messages with message content and determine the destination of each corresponding message (e.g., using a mapping function and / or element addresses and / or data values ​​in the message).

[0135] Different processing clusters 602 can be configured for different tasks. For example, some clusters may be dedicated to performing basic arithmetic calculations, some to neuromorphic computing, and others to performing complex mathematical operations. In some examples, processing cluster 602 is configured to perform neural network processing, while one or more other processing units 610 perform other computational tasks. Alternatively or additionally, processing clusters that can be reconfigured to perform one category of various operation classes can be provided. Similarly, a processing cluster may have multiple processing elements that may have the same or different functions, or they may be reconfigured to have specific functions.

[0136] Each processing element can be designed or configured to detect and generate event messages based on specific computational rules (e.g., spikes when a threshold is exceeded). The neuromorphic state can be dynamically updated based on the received event messages and the computations performed within processing cluster 602. In some examples, if the value of the neuromorphic state approaches or exceeds a threshold potential, the corresponding processing element can issue a control signal causing a message sending facility to send one or more event messages (e.g., to other processing clusters 602 within processing system 600).

[0137] The processing system 600 can be used for a variety of applications, such as image processing, audio processing, machine learning, pattern recognition, or real-time data analysis. For example, in an image processing application, the processing cluster 602 can be used to perform convolution operations on image data, while another processing unit (e.g., another processing unit 610) can handle tasks such as image rendering or video encoding.

[0138] Processing system 600 can efficiently handle layer-by-layer processing in a neural network environment. As described in more detail below, processing system 600 can utilize processing elements in processing cluster 602 to perform convolution operations, which involve applying kernels or filters to input data (e.g., image data) to create feature maps. Processing elements can also apply other operations, such as activation functions. In some examples, different layers of the neural network can be assigned to different subsets of processing cluster 602 for efficient execution.

[0139] Deep neural networks (e.g., CNNs) comprise multiple neural network layers. Each neural network layer typically includes multiple neural network computational elements. Neural network computational elements in a layer can receive weighted inputs from neural network computational elements or input devices in the preceding layer, and can output to neural network computational elements in the following layer. The specific way a neural network layer connects to the preceding layer depends on its type. By way of example, in a fully connected layer, each neural network computational element can receive inputs from neural network computational elements in the preceding layer. In a convolutional layer, each neural network computational element can receive inputs from neural network computational elements in the preceding layer within the convolutional kernel range centered at a local address corresponding to a local address in the convolutional layer. Pooling layers are used for spatial dimensionality reduction. Each neural network computational element in a pooling layer corresponds to a corresponding set of neural network computational elements in the preceding layer. A pooling operation for each neural network element in a pooling layer involves, for example, selecting a value from the corresponding set of neural network elements in the preceding layer, such as sampling a maximum, minimum, median, or the value of a specific neural network element in the corresponding set of neural network elements. Alternatively, a pooling operation involves calculating an average from the corresponding set of neural network elements in the preceding layer.

[0140] An event-based or message-based processing system (e.g., processing system 600) can be configured as a deep neural network. In this case, at least some of the processing elements of processing cluster 602 are configured as neural network computing elements that can function as described above. In some examples, the processing elements can be provided as dedicated hardware acting as neural network computing elements. In other examples, this can be achieved by configuring processing system 600 to programmable the processing elements to act as neural network computing elements. In some examples, each processing element has a dedicated processor, while in other examples, the processing elements of processing cluster 602 share a processor. In operation, the processing elements of processing cluster 602 can therefore receive input messages and send output messages via network 604 when configured as or acting as neural network elements.

[0141] In some examples, each processing cluster 602 acts as a neuron core. Each processing cluster 602 can be configured to operate using Single Instruction Multiple Data (SIMD) processing. For example, each processing cluster 602 can be configured to execute a single instruction in parallel on four data inputs.

[0142] In some examples, because each processing cluster 602 has its own processing power and memory, it is possible to scale the neuron capacity of the processing system 600 to create a mesh network on-chip (NOC) with a neuron core of desired size, capacity, or performance.

[0143] When processing neural networks, processing system 600 implements event-based processing. For example, neuron activation propagates through network 604 only when its value constitutes an "event" (e.g., a value that is non-zero or exceeds a threshold). Therefore, processing system 600 leverages sparsity by treating only certain values ​​as "events." Since only active neurons transmit data, this reduces the amount of data that needs to be processed and transmitted compared to conventional architectures that might process all neuron values, thus improving efficiency.

[0144] Processing system 600 can benefit from quantization techniques as described herein. For example, by selecting a quantization configuration and quantizing the parameters of the neural network before deploying it to processing system 600 for inference (e.g., via on-chip storage), the memory footprint of the neural network can be reduced while still ensuring accurate or high-performance inference.

[0145] A software development kit (SDK) can be used to convert machine learning models into a format that can run on the processing system 600. For example, the SDK can be configured to analyze, compile, and map CNNs onto neural processing units implementing the processing system 600. The SDK can contain compiler components that convert machine learning models into binaries that the processing system 600 can run. Mappings containing the hardware details and mapping constraints required to run on actual hardware can be created. Layers can be grouped and assigned to specific cores (e.g., a specific processing cluster 602). Layers that are too large to fit on a single core can be divided into multiple segments, allowing the layers to be distributed across multiple cores. As mentioned, the compiler components can perform quantization operations using the selected configuration as described herein.

[0146] Now refer to Figure 7 The diagram illustrates a network environment 700 suitable for operating an XR device 710, according to some examples. The network environment 700 includes the XR device 710 and a server 712, which are communicatively coupled to each other via a network 704. The server 712 may be part of a network-based system. For example, the network-based system may be or include a cloud-based server system that provides the XR device 710 with additional information, such as virtual content (e.g., 3D models of virtual objects, or enhancements to be applied as virtual overlays onto images depicting real-world scenes).

[0147] The term "XR" stands for "Extended Reality," which encompasses Augmented Reality (AR) and / or Virtual Reality (VR). The term "AR" refers to an interactive experience of a real-world environment where physical objects or environments residing in the real world are "enhanced" or augmented by computer-generated digital content (also known as virtual or synthetic content). AR devices allow users to observe real-world scenes while simultaneously seeing virtual content that can be registered or overlaid with objects, images, or environments within the AR device's field of view. AR can also refer to systems that enable the combination of the real and virtual worlds, real-time interaction, and three-dimensional (3D) representations of virtual and real objects. Users of AR systems can perceive virtual content that appears attached to, associated with, or interact with real-world physical objects.

[0148] The term "VR" refers to a simulated experience of a virtual world environment that differs from the real world environment. Computer-generated digital content is displayed within this virtual world environment. VR devices can utilize virtual content displayed based on the VR device's positioning and orientation to obscure the user's field of vision. VR also refers to systems that allow users to be fully immersed in a virtual world environment and interact with virtual objects presented within it. AR and VR devices are often referred to as XR devices. Another type of device is based on Mixed Reality ("MR"), which typically represents a hybrid of AR and VR, where a camera facing the world captures images merged with virtual content for display on a VR device. AR devices are typically transparent or see-through, while VR and MR devices are opaque or non-see-through. Therefore, the term "XR" can also refer to MR.

[0149] Refer again Figure 7 User 706 operates XR device 710. User 706 can be a human user (e.g., a human), a machine user (e.g., a computer configured by software programs to interact with XR device 710), or any suitable combination thereof (e.g., a machine-assisted human or a machine supervised by a human). User 706 is not part of network environment 700, but is associated with XR device 710.

[0150] XR device 710 can be a computing device with a display, such as a smartphone, tablet computer, or wearable computing device (e.g., a watch or glasses). The computing device can be handheld or removably mounted to the head of user 706. XR device 710 includes various components, including a processing unit 714 and a camera device 716. In some examples, the display can be a screen that displays content captured by the camera device 716 of XR device 710. In other examples, the device's display can be transparent or semi-transparent, such as within the lenses of wearable computing glasses. In other examples, the display can be a transparent display, such as the windshield of a car, airplane, or truck (e.g., as part of a head-up display system). In yet another example, the display can be opaque and can be worn by the user to cover the user's field of view.

[0151] User 706 operates an application on XR device 710. The application may include an AR application configured to provide user 706 with an experience triggered or enhanced by a physical object 708, such as a two-dimensional physical object (e.g., an image or navigation cues), a three-dimensional physical object (e.g., a statue), a location (e.g., a factory), or a reference in the real-world environment 702 (e.g., a perceived corner of furniture or a wall, or a quick-response (QR) code). For example, user 706 may point at the camera device 716 of XR device 710 to capture an image of the physical object 708, and a virtual overlay may be presented on the physical object 708 via a display. Some experiences may also be triggered, enhanced, or controlled by user 706's hand. Therefore, it should be understood that the physical object 708 or real-world object that XR device 710 is tracking or detecting may be user 706's hand.

[0152] To allow user 706 to engage in AR experiences and / or interact with virtual objects, XR device 710 can detect the location and movement of objects (including, for example, one or both hands of user 706). XR device 710 can use the location, shape, or movement of the hands to determine the user's intention when manipulating the virtual objects. For this purpose, XR device 710 includes a tracking component implemented using processing unit 714. The tracking component can use image sensors (e.g., camera device 716 and / or other image sensors), inertial sensors (e.g., gyroscopes, accelerometers, etc.), and wireless sensors (e.g., Bluetooth). TM Or Wi-Fi sensors), Global Positioning System (GPS) sensors and / or audio sensors (e.g., Figure 7 The microphone 718 shown is used to track the posture (e.g., positioning and orientation) of the XR device 710 relative to the real-world environment 702.

[0153] The processing unit 714 can be used to generate tracking estimates or predictions, such as predicting the position or pose of the tracked object. For this purpose, the XR device 710 can utilize one or more object tracking machine learning models or one or more object detection machine learning models. A specific, non-limiting example of a machine learning model is a trained neural network for pose recognition.

[0154] In this context, the machine learning model may include a neural network trained on suitable training data to identify and / or track objects in one or more frames captured by the XR device 710. In some examples, Figure 6 The components of the processing system 600 are integrated into a single processing unit. The processing unit 714 of the XR device 710 may include an event-driven processing system, such as the processing system 600. Thus, the XR device 710 is a (non-limiting) example of a computing device in which the processing system 600 can be implemented. The processing system 600 may, for example, facilitate the real-time processing of sensor data captured by the XR device 710, such as image data captured using a camera device 716 or audio data captured using a microphone 718.

[0155] In some examples, the XR device 710 benefits from quantization techniques as described herein. For instance, by selecting a quantization configuration and quantizing the parameters of the machine learning model before deploying it to the XR device 710 for inference (e.g., via on-chip storage), memory footprint can be reduced while still ensuring accurate or high-performance inference. This can also lead to improved battery life or reduced latency. The XR device 710 can apply such machine learning models, for example, in the processing of image or audio data.

[0156] In some examples, server 712 can be used to perform certain detection and tracking based on sensor data (e.g., image and depth data) from XR device 710. Therefore, XR device 710 or server 712, or both XR device 710 and server 712, can perform image processing, object detection, and / or object tracking functions based on images captured by XR device 710 and one or more parameters internal or external to XR device 710. In some examples, server 712 may include a processing system or be coupled to a processing system, for example... Figure 6 The processing system is 600.

[0157] Network 704 can be any network that enables communication between or within machines (e.g., server 712), databases, and devices (e.g., XR device 710). Therefore, network 704 can be a wired network, a wireless network (e.g., a mobile or cellular network), or any suitable combination thereof. Network 704 may include one or more components constituting a private network, a public network (e.g., the Internet), or any suitable combination thereof.

[0158] Figure 8 This is a flowchart depicting a machine learning pipeline 800 based on some examples. The machine learning pipeline 800 can be used to generate trained models, for example... Figure 9 The trained machine learning program 902 is shown in Figure 900.

[0159] In a broad sense, machine learning can involve automatically learning patterns and relationships in data using computer algorithms, potentially without explicit programming. Machine learning algorithms can be categorized into three main types: supervised learning, unsupervised learning, and reinforcement learning.

[0160] Supervised learning involves training a model using labeled data to predict outputs for new, unseen inputs. Examples of supervised learning algorithms can include linear regression, decision trees, and neural networks.

[0161] Unsupervised learning involves training a model on unlabeled data to find hidden patterns and relationships within the data. Examples of unsupervised learning algorithms can include clustering, principal component analysis, and generative models (such as autoencoders).

[0162] Reinforcement learning involves training a model to make decisions in dynamic environments by receiving feedback in the form of rewards or penalties. Examples of reinforcement learning algorithms can include Q-learning and policy gradient methods.

[0163] Examples of specific machine learning algorithms that can be deployed include logistic regression, a type of supervised learning algorithm used for binary classification tasks. Logistic regression models the probability of a binary response variable based on one or more predictor variables. Another example type of machine learning algorithm is Naive Bayes, a supervised learning algorithm used for classification tasks. Naive Bayes is based on Bayes' theorem and assumes that the predictor variables are independent of each other. Random forests are another type of supervised learning algorithm used for classification, regression, and other tasks. Random forests build an ensemble of decision trees and combine their outputs to make predictions. Other examples include neural networks, which consist of layers of interconnected nodes (or neurons) that process information and make predictions based on input data. Matrix factorization is another type of machine learning algorithm used for recommender systems and other tasks. Matrix factorization decomposes a matrix into two or more matrices to reveal hidden patterns or relationships in the data. Support vector machines (SVMs) are a type of supervised learning algorithm used for classification, regression, and other tasks. SVMs find hyperplanes that separate different classes in the data. Other types of machine learning algorithms can include decision trees, k-nearest neighbors, clustering algorithms, and deep learning algorithms (such as CNNs, recurrent neural networks (RNNs), and transformer models). The choice of algorithm can depend on the nature of the data, the complexity of the problem, and the performance requirements of the application.

[0164] The performance of a machine learning model can be evaluated on a separate test dataset that was not used during training, ensuring that the model can generalize to new, unseen data. Deep learning algorithms such as CNNs, recurrent neural networks, and transformers, as well as more traditional machine learning algorithms such as decision trees, random forests, and gradient boosting, can be used in a wide variety of machine learning applications.

[0165] Two typical types of problems in machine learning are classification problems and regression problems. Classification problems (also known as categorization problems) aim to classify an item into one of several category values ​​(e.g., is the object an apple or an orange?). Regression algorithms aim to quantify some items (e.g., by providing values ​​as real numbers).

[0166] Generating a trained machine learning program 902 may include multiple stages forming part of a machine learning pipeline 800, said multiple stages including, for example... Figure 8 The following stages are shown:

[0167] Data Collection and Preprocessing 802: This stage may include acquiring and cleaning data to ensure it is suitable for use in machine learning models. This stage may also include removing duplicates, handling missing values, and converting the data to a suitable format.

[0168] Feature engineering 804: This stage may include selecting and transforming training data 906 to create features useful for predicting the target variable. Feature engineering may include (1) receiving features 908 (e.g., as structured or labeled data in supervised learning) and / or (2) identifying features 908 in training data 906 (e.g., unstructured or unlabeled data for unsupervised learning).

[0169] Model selection and training 806: This stage may include selecting an appropriate machine learning algorithm and training it on preprocessed data. This stage may also involve splitting the data into training and test sets, using cross-validation to evaluate the model, and tuning hyperparameters to improve performance.

[0170] Model Evaluation 808: This phase may include evaluating the performance of the trained model (e.g., a trained machine learning program 902) on a separate test dataset. This phase can help determine if the model is overfitting or underfitting, and whether the model is suitable for deployment.

[0171] Prediction 810: This stage involves using a trained model (e.g., a trained machine learning program 902) to generate predictions for new, unseen data.

[0172] Validation, refinement, or retraining 812: This stage may include updating the model based on feedback generated from the prediction stage (e.g., new data or user feedback).

[0173] Deployment 814: This phase may include integrating the trained model (e.g., a trained machine learning program 902) into a wider system or application (e.g., a web service, mobile application, or Internet of Things (IoT) device). This phase may involve installing the API, building the user interface, and ensuring the model is scalable and can handle large amounts of data.

[0174] In some examples, quantization can be performed before deploying the 814. For instance, it can be done via a quantization pipeline (e.g., before deployment on the target hardware). Figure 4 The quantization pipeline 402 is used to process machine learning models.

[0175] Figure 9Further details of two example phases are shown: a training phase 904 (e.g., part of model selection and training 806) and a prediction phase 910 (part of prediction 810). Prior to the training phase 904, feature engineering 804 is used to identify features 908. This may include identifying informative, distinctive, and independent features for effectively operating the trained machine learning program 902 in pattern recognition, classification, and regression. In some examples, training data 906 includes labeled data known for the pre-identified features 908 and one or more outcomes. Each of the features 908 can be a variable or attribute, such as various measurable properties of a process, item, system, or phenomenon represented by the dataset (e.g., training data 906). By way of example only, features 908 may also have different types such as numerical features, strings, and graphs, and may include one or more of content 912, concepts 914, attributes 916, historical data 918, and / or user data 920.

[0176] In training phase 904, the machine learning program can use training data 906 to find correlations among features 908 that influence prediction outcomes or prediction / inference data 922. Using training data 906 and the identified features 908, the trained machine learning program 902 is trained during training phase 904, which is also during machine learning program training 924. Machine learning program training 924 evaluates feature 908 values ​​when they correlate with training data 906. The result of the training is the trained machine learning program 902 (e.g., a trained or learned model).

[0177] Furthermore, the training phase 904 can involve machine learning, where the training data 906 is structured (e.g., labeled during preprocessing). The trained machine learning program 902 can implement a neural network 926 capable of performing operations such as classification and clustering. In other examples, the training phase 904 can involve deep learning, where the training data 906 is unstructured, and the trained machine learning program 902 implements a deep neural network 926 capable of performing both feature extraction and classification / clustering operations.

[0178] In some examples, neural network 926 can be generated during training phase 904 and implemented within a trained machine learning program 902. Neural network 926 includes a hierarchical (e.g., layered) organization of neurons, where each layer consists of multiple neurons or nodes. Neurons in the input layer receive input data, while neurons in the output layer produce the network's final output. Between the input and output layers, there may be one or more hidden layers, each consisting of multiple neurons.

[0179] Each neuron in a neural network 926 can operationally compute a function, such as an activation function, that takes as input a weighted sum of the outputs of neurons in the previous layer and a bias term. The output of this function is then passed as input to neurons in the next layer. If the output of the activation function exceeds a certain threshold, the output is passed from that neuron (e.g., a sending neuron) to a connecting neuron (e.g., a receiving neuron) in a subsequent layer. The connections between neurons have associated weights that define the influence of inputs from the sending neuron to the receiving neuron. During the training phase, these weights are adjusted by a learning algorithm to optimize the network's performance. Different types of neural networks can use different activation functions and learning algorithms, affecting their performance on different tasks. The hierarchical organization of neurons and the use of activation functions and weights enable neural networks to model complex relationships between inputs and outputs and generalize to new inputs not seen during training.

[0180] In some examples, neural network 926 can also be one of several different types of neural networks, such as a single-layer feedforward network, a multilayer perceptron (MLP), an artificial neural network (ANN), an RNN, a long short-term memory network (LSTM), a bidirectional neural network, a symmetric connection neural network, a deep belief network (DBN), a CNN, a generative adversarial network (GAN), an autoencoder neural network (AE), a restricted Boltzmann machine (RBM), a Hopfield network, a self-organizing map (SOM), a radial basis function network (RBFN), a spiking neural network (SNN), a liquid state machine (LSM), an echo state network (ESN), a neural Turing machine (NTM), or a Transformer network, for example only.

[0181] In addition to the training phase 904, a validation phase can be performed on a separate dataset called the validation dataset. The validation dataset is used to tune the model's hyperparameters, such as the learning rate and regularization parameters. Tune the hyperparameters to improve the model's performance on the validation dataset.

[0182] Once the model is fully trained and validated, the testing phase allows it to be tested on a new dataset. The test dataset is used to evaluate the model's performance and ensure that it has not overfitted the training data.

[0183] In the prediction phase 910, the trained machine learning program 902 uses features 908 to analyze the query data 928 to generate inference, results, or predictions, as examples of prediction / inference data 922. For example, during the prediction phase 910, the trained machine learning program 902 generates outputs. The query data 928 is provided as input to the trained machine learning program 902, and the trained machine learning program 902 generates prediction / inference data 922 as output in response to receiving the query data 928.

[0184] Figure 10 This is a schematic representation of machine 1000, within which instructions 1002 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause machine 1000 to perform any or more of the methods discussed herein. For example, instructions 1002 can cause machine 1000 to perform any or more of the methods described herein. Instructions 1002 transform the general, unprogrammed machine 1000 into a specific machine 1000 programmed to perform the described and illustrated functions in the described manner. Machine 1000 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 1000 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1000 may include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 1002 specifying actions to be taken by machine 1000. Furthermore, although only a single machine 1000 is shown, the term "machine" should also be considered as a collection of machines that individually or jointly execute instructions 1002 to perform any or more of the methods discussed herein. Machine 1000 may, for example, include user system 102 or any of a plurality of server devices forming part of server system 110. In some examples, machine 1000 may also include both client and server systems, wherein certain operations of a particular method or algorithm are performed on the server side, and wherein certain operations of a particular method or algorithm are performed on the client side.

[0185] Machine 1000 may include a processor 1004, a memory 1006, and an input / output (I / O) unit 1008 that can be configured to communicate with each other via a bus 1010. In the example, processor 1004 may include, for example, processors 1012 and 1014 that execute instruction 1002. Although Figure 10 Multiple processors 1004 are shown, but machine 1000 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0186] Memory 1006 includes main memory 1016, static memory 1018, and storage cells 1020, all of which are accessible by processor 1004 via bus 1010. Main memory 1006, static memory 1018, and storage cells 1020 store instructions 1002 embodying any one or more of the methods or functions described herein. Instructions 1002 may also reside wholly or partially in main memory 1016, in static memory 1018, in machine-readable medium 1022 within storage cell 1020, in at least one of the processor 1004 (e.g., in the processor's cache memory), or in any suitable combination thereof during execution by machine 1000.

[0187] I / O component 1008 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement results, etc. The specific I / O component 1008 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It will be understood that I / O component 1008 may include... Figure 10 Many other components are not shown. In various examples, I / O component 1008 may include user output component 1024 and user input component 1026. User output component 1024 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. User input component 1026 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide positioning and force for touch or touch gestures), audio input components (e.g., microphones), etc.

[0188] In other examples, I / O component 1008 may include biometric component 1028, motion component 1030, environmental component 1032, or positioning component 1034, as well as a wide range of other components. For example, biometric component 1028 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or EEG-based identification), etc.

[0189] Any biometric or other personally identifiable information (PII) collected by the biometrics component or other data capture component is captured and stored only with the user's approval and is deleted upon the user's request. Furthermore, such data may be used for very limited purposes (e.g., authentication). To ensure the restricted and authorized use of biometrics and other PIIs, access to this data is limited to authorized personnel (if applicable). The data will not be shared or sold to any third party without the user's explicit consent. Additionally, appropriate technical and organizational measures have been implemented to ensure the security and confidentiality of this sensitive information.

[0190] The moving part 1030 includes an acceleration sensor part (e.g., an accelerometer), a gravity sensor part, and a rotation sensor part (e.g., a gyroscope).

[0191] The environmental component 1032 includes, for example, one or more camera devices (with still image / photograph and video capabilities), lighting sensor components (e.g., a photometer), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., a barometer), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., an infrared sensor for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting the concentration of hazardous gases for safety purposes or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0192] Regarding the camera device, user system 102 may have a camera device system including, for example, a front-facing camera on the front surface of user system 102 and a rear-facing camera on the rear surface of user system 102. The front-facing camera may be used, for example, to capture still images and videos (e.g., “selfies”) of the user of user system 102, which can then be enhanced with the enhancement data (e.g., filters) described above. For example, the rear-facing camera may be used to capture still images and videos in a more conventional camera device mode, wherein these images are similarly enhanced with enhancement data. In addition to the front-facing and rear-facing cameras, user system 102 may also include a 360° camera device for acquiring 360° photos and videos.

[0193] Furthermore, the camera system of user system 102 may include dual rear cameras (e.g., a main camera and a depth-sensing camera), or even triple, quadruple, or quintuple rear camera configurations on the front and rear sides of user system 102. For example, these multi-camera systems may include wide-angle cameras, ultra-wide-angle cameras, telephoto cameras, macro cameras, and depth sensors.

[0194] The positioning component 1034 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure and can determine altitude based on air pressure), an orientation sensor component (e.g., a magnetometer), etc.

[0195] Communication can be implemented using a variety of technologies. I / O component 1008 also includes a communication component 1036 operable to couple machine 1000 to network 1038 or device 1040 via a suitable coupling or connection. For example, communication component 1036 may include a network interface component or another suitable device that interfaces with network 1038. In other examples, communication component 1036 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, or Bluetooth. TM Components (e.g., Bluetooth) TM Low-power, Wi-Fi components, and other communication components for providing communication via other modes. Device 1040 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0196] Furthermore, communication component 1036 may detect identifiers or include components operable to detect identifiers. For example, communication component 1036 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, QR codes such as Quick Response (QR) codes, Aztec codes, data matrices, and data symbols). TM The system can utilize optical sensors for multidimensional barcodes and other optical codes, such as MaxiCode, PDF417, UltraCode, and UCC RSS-2D barcodes, or acoustic detection components (e.g., microphones for identifying audio signals of the tags). Additionally, various information can be obtained via communication component 1036, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via detecting NFC beacon signals that can indicate a specific location.

[0197] Various memories (e.g., main memory 1016, static memory 1018, and the memory of processor 1004) and storage units 1020 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any or more of the methods or functions described herein. These instructions (e.g., instruction 1002) cause various operations to implement the disclosed examples when executed by processor 1004.

[0198] Instructions 1002 can be sent or received over network 1038 using a transmission medium via a network interface device (e.g., a network interface component included in communication component 1036) and using any of several known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 1002 can be sent or received using a transmission medium via coupling to device 1040 (e.g., peer-to-peer coupling).

[0199] Figure 11 This is a block diagram 1100 illustrating a software architecture 1102 that can be installed on any or more of the devices described herein. The software architecture 1102 is supported by hardware such as a machine 1104 including a processor 1106, memory 1108, and I / O components 1110. In this example, the software architecture 1102 can be conceptualized as a stack of layers, where each layer provides a specific function. The software architecture 1102 includes layers such as an operating system 1112, libraries 1114, frameworks 1116, and applications 1118. Operationally, application 1118 activates API call 1120 via the software stack and receives message 1122 in response to API call 1120.

[0200] Operating system 1112 manages hardware resources and provides general services. Operating system 1112 includes, for example, a kernel 1124, services 1126, and drivers 1128. Kernel 1124 serves as an abstraction layer between hardware and other software layers. For example, kernel 1124 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1126 can provide other public services to other software layers. Driver 1128 is responsible for controlling or interfacing with the underlying hardware. For example, driver 1128 may include a display driver, a camera driver, a bluetooth driver, etc. TM Or BLUETOOTH TM Low-power drives, flash drives, serial communication drives (e.g., USB drives), Wi-Fi drives, audio drives, power management drives, etc.

[0201] Library 1114 provides common low-level infrastructure used by application 1118. Library 1114 may include system library 1130 (e.g., the C standard library), which provides functions such as memory allocation, string manipulation, and mathematical functions. Additionally, library 1114 may include API library 1132, such as media libraries (e.g., libraries for supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Picture Experts Group (JPEG or JPG), or Portable Web Graphics (PNG)), graphics libraries (e.g., the OpenGL framework for rendering graphic content on a display in two-dimensional (2D) and three-dimensional (3D) formats), database libraries (e.g., SQLite, which provides various relational database functions), web libraries (e.g., WebKit, which provides web browsing functionality), and so on. Library 1114 may also include various other libraries 1134 to provide many other APIs to application 1118.

[0202] Framework 1116 provides a common high-level infrastructure for use by application 1118. For example, framework 1116 provides various graphical user interface (GUI) functions, high-level resource management, and high-level location services. Framework 1116 can provide a wide range of other APIs that can be used by application 1118, some of which may be specific to a particular operating system or platform.

[0203] In the example, application 1118 may include home application 1136, contacts application 1138, browser application 1140, book reader application 1142, location application 1144, media application 1146, messaging application 1148, game application 1150, and a wide variety of other applications such as third-party application 1152. Application 1118 is a program that performs the functions defined in the program. One or more applications 1118 can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 1152 (e.g., an application developed by an entity other than a platform-specific vendor using the Android™ or iOS™ SDK) may be mobile software that runs on mobile operating systems such as iOS™, Android™, Windows® Phone, or other mobile operating systems. In this example, a third-party application 1152 can activate API call 1120 provided by the operating system 1112 to facilitate the functionality described herein.

[0204] Example

[0205] In view of the above-mentioned implementation methods, this application discloses the following list of examples, wherein a feature of a single example or more than one feature of an example is combined together, and optionally, is combined with one or more features of one or more other examples, which are also further examples falling within the disclosure of this application.

[0206] Example 1 is a system comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations including: identifying a set of candidate quantization configurations for components of a trained machine learning model; for each candidate quantization configuration in the set of candidate quantization configurations: applying the candidate quantization configuration to the component; after applying the candidate quantization configuration, executing the trained machine learning model on an input dataset to obtain candidate output values ​​for the candidate quantization configurations, and determining a loss associated with the candidate quantization configuration based on a comparison between the candidate output values ​​for the component and a reference output value; for the component, selecting a candidate quantization configuration from the set of candidate quantization configurations based on the determined loss associated with the set of candidate quantization configurations; and using the selected candidate quantization configuration for the component to quantize at least a portion of the trained machine learning model.

[0207] In Example 2, the subject of Example 1 includes the following: applying the candidate quantization configuration to the component causes the trained machine learning model to be executed with quantized parameters in the component, and the reference output value is obtained by executing the trained machine learning model with unquantized parameters in the component.

[0208] In Example 3, the subject matter of any one of Examples 1 to 2 includes, wherein the component is one of a plurality of components of the trained machine learning model, the selected candidate quantization configuration is a first quantization configuration, and the trained machine learning model is quantized using the first quantization configuration for the component and one or more other quantization configurations selected for one or more of the plurality of components.

[0209] In Example 4, the subject of Example 3 includes the following: the trained machine learning model includes a neural network, the plurality of components include a plurality of layers of the neural network, and for each candidate quantization configuration, the candidate output value is a candidate feature map value, and the reference output value is a reference feature map value of one of the plurality of layers.

[0210] In Example 5, the subject of Example 4 includes, wherein the first quantization configuration includes a first bit setting for quantizing the parameters of a first layer among the plurality of layers, and the one or more other quantization configurations include a second bit setting for quantizing the parameters of one or more other layers of the neural network, the first bit setting being different from the second bit setting.

[0211] In Example 6, the subject matter of any one of Examples 3 to 5 includes, wherein the trained machine learning model includes a neural network, and the plurality of components include a plurality of channels within a layer of the neural network.

[0212] In Example 7, the subject matter of any one of Examples 1 to 6 includes, wherein the set of candidate quantization configurations is a first set, the component of the trained machine learning model is a first component, the candidate output value is a first candidate output value, the reference output value is a first reference output value, and the selected candidate quantization configuration is a first candidate quantization configuration, the operation further comprising: identifying a second set of candidate quantization configurations for a second component of the trained machine learning model; for each candidate quantization configuration in the second set of candidate quantization configurations, determining a loss associated with the candidate quantization configuration based on a comparison between a second candidate output value for the second component and a second reference output value, the second candidate output value being obtained by applying the candidate quantization configuration to the second component before executing the trained machine learning model; and for the second component, selecting a second candidate quantization configuration from the second set of candidate quantization configurations based on the determined loss associated with the second set of candidate quantization configurations, wherein the trained machine learning model is quantized using the first candidate quantization configuration for the first component and the second candidate quantization configuration for the second component.

[0213] In Example 8, the subject of any one of Examples 1 to 7 includes, wherein the component comprises a layer of a neural network, the layer being associated with a threshold function, and the threshold function being applied to obtain at least a subset of the candidate output values ​​and at least a subset of the reference output values.

[0214] In Example 9, the subject of any one of Examples 1 to 8 includes, wherein each candidate quantization configuration in the set of candidate quantization configurations includes bit settings for quantizing the parameters of the components of the trained machine learning model.

[0215] In Example 10, the subject of Example 9 includes, wherein the parameters include weights, and each weight is quantized to be represented by a combination of exponent and mantissa bits.

[0216] In Example 11, the subject matter of any one of Examples 1 to 10 includes, further comprising: receiving a quantization request from a user equipment, the quantization request including a bit precision for quantization of the component, wherein a set of candidate quantization configurations is identified based on the selected bit precision, the set of candidate quantization configurations including different combinations of exponent bits and mantissa bits satisfying the selected bit precision.

[0217] In Example 12, the subject of any one of Examples 1 to 11 includes, wherein the loss is determined based on a loss function, and selecting the candidate quantization configuration from the set of candidate quantization configurations includes: detecting that the selected candidate quantization configuration produces the minimum value of the loss function for the components of the trained machine learning model.

[0218] In Example 13, the subject matter of any one of Examples 1 to 12 includes, wherein quantizing the trained machine learning model includes quantizing the parameters of the trained machine learning model, and the operation further includes storing the quantized parameters in an on-chip memory of a processing device.

[0219] In Example 14, the subject matter of any one of Examples 1 to 13 includes, wherein the trained machine learning model comprises a neural network, and the operation further includes: performing batch normalization folding before obtaining the reference output value and before quantizing the trained machine learning model.

[0220] In Example 15, the subject of any one of Examples 1 to 14 includes the operation further comprising: generating an output including the selected candidate quantization configuration; and transmitting the output to a user equipment.

[0221] In Example 16, the subject matter of any one of Examples 1 to 15 includes, wherein quantizing the trained machine learning model includes generating a new instance of the trained machine learning model, the new instance of the trained machine learning model including the selected candidate quantization configuration for the component, the operation further including: receiving a quantization request from a user device; and generating the new instance of the trained machine learning model in response to receiving the quantization request.

[0222] In Example 17, the subject of any of Examples 1 to 16 includes, wherein the input dataset includes unlabeled sample data.

[0223] In Example 18, the subject of any of Examples 1 to 17 includes, wherein the input dataset comprises unlabeled sample images.

[0224] Example 19 is a method comprising: identifying a set of candidate quantization configurations for components of a trained machine learning model; for each candidate quantization configuration in the set of candidate quantization configurations: applying the candidate quantization configuration to the component; after applying the candidate quantization configuration, executing the trained machine learning model on an input dataset to obtain a candidate output value for the candidate quantization configuration; and determining a loss associated with the candidate quantization configuration based on a comparison between the candidate output value for the component and a reference output value; for the component, selecting a candidate quantization configuration from the set of candidate quantization configurations based on the determined loss associated with the set of candidate quantization configurations; and using the selected candidate quantization configuration for the component to quantize at least a portion of the trained machine learning model.

[0225] Example 20 is a non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations including: identifying a set of candidate quantization configurations for components of a trained machine learning model; for each candidate quantization configuration in the set of candidate quantization configurations: applying the candidate quantization configuration to the component; after applying the candidate quantization configuration, executing the trained machine learning model on an input dataset to obtain candidate output values ​​for the candidate quantization configurations; and determining a loss associated with the candidate quantization configuration based on a comparison between the candidate output values ​​for the component and a reference output value; for the component, selecting a candidate quantization configuration from the set of candidate quantization configurations based on the determined loss associated with the set of candidate quantization configurations; and using the selected candidate quantization configuration for the component to quantize at least a portion of the trained machine learning model.

[0226] Example 21 is at least one machine-readable medium comprising instructions that, when executed by a processing circuitry system, cause the processing circuitry system to perform an operation for implementing any one of Examples 1 to 20.

[0227] Example 22 is an apparatus that includes means for implementing any one of Examples 1 to 20.

[0228] Example 23 is a system for implementing any one of Examples 1 through 20.

[0229] Example 24 is a method for implementing any one of Examples 1 through 20.

[0230] in conclusion

[0231] Although specific examples are described herein, it will be apparent that various modifications and changes can be made to these examples without departing from the broader spirit or scope of this disclosure. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The drawings, which form part of this disclosure, illustrate specific examples of how the subject matter can be practiced by way of illustration rather than limitation. The examples shown are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other examples can be utilized and derived therefrom, allowing for structural and logical substitutions and changes without departing from the scope of this disclosure. Therefore, this specific embodiment should not be construed as restrictive, and the scope of the various examples is defined only by the appended claims and the full scope of their equivalents.

[0232] Such examples of the subject matter may be referred to herein individually or collectively by the term "example," or merely for convenience, and are not intended to voluntarily limit the scope of this application to any single example or concept—if in fact more than one example or concept is disclosed. Therefore, although specific examples have been shown and described herein, it should be understood that a particular example shown may be replaced by another arrangement calculated to achieve the same purpose. This disclosure is intended to cover any and all adaptations or variations of the various examples. Combinations of the above examples and other examples not specifically described herein will be apparent to those skilled in the art upon review of the foregoing description.

[0233] As used in this disclosure, the term "machine learning model" (or simply "model") can refer to a single, independent model or a combination of models. The term can also refer to a system, component, or module that includes a machine learning model and one or more supporting or supplementary components that do not necessarily perform machine learning tasks.

[0234] Some parts of the subject matter discussed herein can be presented based on the algorithms or symbolic representations of operations on data stored as bit or binary digital signals in machine memory (e.g., computer memory). Such algorithms or symbolic representations are examples of techniques used by those of ordinary skill in the art of data processing to convey the essence of their work to others of similar skill. As used herein, an "algorithm" is a self-consistent sequence of operations or similar processing that yields a desired result. In this context, algorithms and operations involve the physical manipulation of physical quantities. Typically, but not necessarily, such quantities can take the form of electrical, magnetic, or optical signals that can be stored, accessed, transmitted, combined, compared, or otherwise manipulated by a machine. Sometimes, primarily for general reasons, it is convenient to use terms such as "data," "content," "bit," "value," "element," "symbol," "character," "term," "quantity," "number," etc., to refer to such signals. However, these terms are merely convenient labels and are associated with appropriate physical quantities.

[0235] Unless otherwise expressly stated, the use of terms such as “processing,” “computing,” “calculating,” “determining,” “presenting,” “displaying,” etc., herein may refer to the actions or processing of a machine (e.g., a computer) whose manipulation or transformation is represented as physical (e.g., electronic, magnetic, or optical) quantities of data within one or more memories (e.g., volatile memory, non-volatile memory, or any suitable combination thereof), registers, or other machine components that receive, store, transmit, or display information. Furthermore, as is common in patent literature, unless otherwise specifically stated, the terms “a” and “an” are used herein to include one or more instances.

[0236] As used in this disclosure, phrases such as "at least one of A, B, or C," "at least one of A, B, or C," and "at least one of A, B, and C" should be interpreted as selecting at least one from the group including "A, B, and C." In this disclosure, unless explicitly stated otherwise in conjunction with specific examples, this phrasing does not imply "at least one of A, at least one of B, and at least one of C." As used in this disclosure, the example "at least one of A, B, or C" will cover any of the following selections: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, and {A, B, C}.

[0237] Unless the context explicitly requires otherwise, throughout the specification and claims, the words “comprising,” “including,” etc., shall be interpreted in an inclusive sense, rather than an exclusive or exhaustive sense, for example, in the sense of “including but not limited to.” As used herein, the terms “connection,” “coupled,” or any variation thereof mean any direct or indirect connection or coupling between two or more elements; the coupling or connection between elements may be physical, logical, or a combination thereof. Furthermore, when used in this application, the words “in this document,” “above,” “below,” and words with similar meanings refer to the entire application and not any particular part of the application. Where the context permits, the use of singular or plural terms may also include the plural or singular, respectively. When referring to a list of two or more items, the word “or” covers all the following interpretations of the word: any item in the list, all items in the list, and any combination of items in the list. Similarly, with respect to a list of two or more items, the word “and / or” covers all the following interpretations of the word: any item in the list, all items in the list, and any combination of items in the list.

[0238] The various features, steps, operations, and processes described herein can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, certain methods, processing boxes, or operations may be omitted in some implementations.

[0239] While some examples (such as those depicted in the accompanying figures) include a specific order of operations, that order may be changed without departing from the scope of this disclosure. For example, some operations in the depicted operations may be performed in parallel, or in a different order that does not substantially affect the functionality described in the examples. In other examples, different components of an example device or system implementing the example methods may perform their functions substantially simultaneously or in a specific order.

[0240] Vocabulary

[0241] For example, "carrier signal" refers to any intangible medium or other intangible medium capable of storing, encoding, or carrying machine-executable instructions and including digital or analog communication signals. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0242] "Client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablet computers, ultrabooks, netbooks, laptops, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0243] "Communications network" refers to one or more parts of a network, such as an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Common Old-Style Telephone Service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, coupling can implement any data transmission technology of various types, such as single-carrier radio transmission technology (1xRTT), evolved data optimization (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate GSM evolution (EDGE) technology, the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long-distance protocols, or other data transmission technologies.

[0244] A “component” refers to a logical or physical entity having boundaries defined by functional or subroutine calls, branch points, application programming interfaces (APIs), or other technologies that provide partitioning or modularity for a particular processing or control function. A component can be combined with other components via its interface to perform machine processing. A component can be a packaged functional hardware unit designed to be used with other components and part of a program that typically performs a related function. A component can constitute a software component (e.g., code embodied on a machine-readable medium) or a hardware component. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various examples, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups or portions of processors) of a computer system can be configured by software (e.g., an application or part of an application) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include dedicated circuitry or logic permanently configured to perform certain operations. Hardware components can be dedicated processors, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Hardware components can also include programmable logic or circuit systems that are temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific part of a machine) uniquely tailored to perform the configured function and is no longer a general-purpose processor. It will be understood that the decision to implement a hardware component mechanically in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., software-configured) circuit system may be made for cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include tangible entities, i.e., entities physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate or perform certain operations described herein. Consider the example of hardware components being temporarily configured (e.g., programmed), where each hardware component need not be configured or instantiated at any given time. For example, where the hardware components include a general-purpose processor that can be configured as a dedicated processor via software, the general-purpose processor can be configured at different times as its respective dedicated processor (e.g., including different hardware components). Thus, the software configures one or more specific processors to constitute a specific hardware component at one time and different hardware components at different times. Hardware components can provide information to and receive information from other hardware components. Therefore, the described hardware components can be considered communicatively coupled.In the presence of multiple hardware components, communication can be achieved through signal transmission between or among two or more hardware components (e.g., via appropriate circuitry and buses). In examples where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving information from said memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Another hardware component can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., collections of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily (e.g., via software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be implemented at least in part by a processor, where a particular processor (or a portion thereof) is an example of hardware. For example, at least some operations of the method can be performed by one or more processors or processor-implemented components. At least some operations can be performed by a group of computers (as an example of a machine including processors), where these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of some operations can be distributed among processors, not residing within a single machine, but deployed across multiple machines. In some examples, the processor or processor-implemented component may reside in a single geographic location (e.g., within a home environment, office environment, or server cluster). In other examples, the processor or processor-implemented component may be distributed across multiple geographic locations.

[0245] For example, "computer-readable storage medium" refers to both machine-readable storage media and transmission media. Therefore, the term includes both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" refer to the same thing and may be used interchangeably in this disclosure.

[0246] "Machine storage medium" refers to one or more storage devices and media (e.g., centralized or distributed databases and associated caches and servers) that store executable instructions, routines, and data. Therefore, this term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, by way of example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal medium."

[0247] "Non-transitory computer-readable storage medium" means, for example, a tangible medium capable of storing, encoding, or carrying instructions that can be executed by a machine.

[0248] A “processor” can refer to any one or more circuits or virtual circuits (e.g., physical circuits simulated by logic executed on an actual processor) that manipulate data values ​​according to control signals (e.g., commands, opcodes, machine codes, control words, macro instructions, etc.) and generate corresponding output signals for operating a machine. A processor can include, for example, at least one of the following: CPU, Reduced Instruction Set Computing (RISC) processor, Complex Instruction Set Computing (CISC) processor, GPU, Digital Signal Processor (DSP), Tensor Processing Unit (TPU), Neural Processing Unit (NPU), Visual Processing Unit (VPU), Machine Learning Accelerator, Artificial Intelligence Accelerator, Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), Radio Frequency Integrated Circuit (RFIC), Neuromorphic Processor, Quantum Processor, or any combination thereof. A processor can be a multi-core processor with two or more independent processors (sometimes called “cores”) that can execute instructions simultaneously. A multi-core processor can contain multiple computing cores on a single integrated circuit die, each of which can independently execute program instructions in parallel. Parallel processing on multi-core processors can be implemented via architectures such as superscalar, VLIW, vector processing, or SIMD, which enable each core to execute a separate stream of instructions simultaneously. A processor can be emulated as a virtual processor or virtual circuit in software running on a physical processor. A virtual processor can behave like an independent processor but is implemented in software rather than hardware. Therefore, unless a specific processor architecture, hardware, design, and / or architecture is specified or clearly understood from the context, the terms "processor," "processing system," etc., should be interpreted broadly herein.

[0249] "Signal medium" means, for example, any intangible medium capable of storing, encoding, or carrying machine-executable instructions and including digital or analog communication signals, or other intangible media that facilitates the transmission of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0250] "User equipment" means, for example, a device that is accessed, controlled, or owned by a user and that the user interacts with to perform actions or interactions (including interactions with other users or computer systems). A user device may be, for example, one or more of the client devices listed above.

Claims

1. A system comprising: At least one processor; as well as At least one memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations including: A set of candidate quantization configurations for component identification of a trained machine learning model; For each candidate quantization configuration in the set of candidate quantization configurations: Apply the candidate quantization configuration to the component. After applying the candidate quantization configuration, the trained machine learning model is executed on the input dataset to obtain candidate output values ​​for the candidate quantization configuration, and Based on the comparison between the candidate output value and the reference output value for the component, the loss associated with the candidate quantization configuration is determined; For the component, a candidate quantization configuration is selected from the set of candidate quantization configurations based on a determined loss associated with the set of candidate quantization configurations; and At least a portion of the trained machine learning model is quantized using a selected candidate quantization configuration for the component.

2. The system according to claim 1, wherein, The candidate quantization configuration is applied to the component so that the trained machine learning model is executed with quantized parameters in the component, and the reference output value is obtained by executing the trained machine learning model with unquantized parameters in the component.

3. The system according to claim 1, wherein, The component is one of a plurality of components of the trained machine learning model, the selected candidate quantization configuration is a first quantization configuration, and the trained machine learning model is quantized using the first quantization configuration for the component and one or more other quantization configurations selected for one or more of the plurality of components.

4. The system according to claim 3, wherein, The trained machine learning model includes a neural network, the plurality of components include a plurality of layers of the neural network, and for each candidate quantization configuration, the candidate output value is a candidate feature map value, and the reference output value is a reference feature map value of one of the plurality of layers.

5. The system according to claim 4, wherein, The first quantization configuration includes a first bit setting for quantizing the parameters of a first layer among the plurality of layers, and the one or more other quantization configurations include a second bit setting for quantizing the parameters of one or more other layers of the neural network, wherein the first bit setting is different from the second bit setting.

6. The system according to claim 3, wherein, The trained machine learning model includes a neural network, and the plurality of components include a plurality of channels within a layer of the neural network.

7. The system according to claim 1, wherein, The set of candidate quantization configurations is a first set, the component of the trained machine learning model is a first component, the candidate output value is a first candidate output value, the reference output value is a first reference output value, and the selected candidate quantization configuration is a first candidate quantization configuration. The operation further includes: A second set of candidate quantization configurations is identified for the second component of the trained machine learning model; For each candidate quantization configuration in the second set of candidate quantization configurations, a loss associated with the candidate quantization configuration is determined based on a comparison between a second candidate output value and a second reference output value for the second component, the second candidate output value being obtained by applying the candidate quantization configuration to the second component before executing the trained machine learning model; and For the second component, based on the determined loss associated with the second set of candidate quantization configurations, a second candidate quantization configuration is selected from the second set of candidate quantization configurations. The trained machine learning model is quantized using the first candidate quantization configuration for the first component and the second candidate quantization configuration for the second component.

8. The system according to claim 1, wherein, The component includes layers of a neural network, the layers being associated with a threshold function, and the threshold function being applied to obtain at least a subset of the candidate output values ​​and at least a subset of the reference output values.

9. The system according to claim 1, wherein, Each candidate quantization configuration in the set of candidate quantization configurations includes bit settings for quantizing the parameters of the components of the trained machine learning model.

10. The system according to claim 9, wherein, The parameters include weights, and each weight is quantized to be represented by a combination of exponent and mantissa bits.

11. The system according to claim 1, wherein the operation further comprises: A quantization request is received from a user equipment, the quantization request including a bit precision for quantization of the component, wherein a set of candidate quantization configurations is identified based on the selected bit precision, the set of candidate quantization configurations including different combinations of exponent bits and mantissa bits that satisfy the selected bit precision.

12. The system according to claim 1, wherein, The loss is determined based on a loss function, and selecting the candidate quantization configuration from the set of candidate quantization configurations includes: The selected candidate quantization configuration is detected to produce the minimum value of the loss function for the components of the trained machine learning model.

13. The system according to claim 1, wherein, Quantizing the trained machine learning model includes quantizing the parameters of the trained machine learning model, and the operation further includes: The quantized parameters are stored in the on-chip memory of the processing device.

14. The system according to claim 1, wherein, The trained machine learning model includes a neural network, and the operation further includes: Batch normalization folding is performed before obtaining the reference output value and before quantizing the trained machine learning model.

15. The system according to claim 1, wherein the operation further comprises: Generate an output that includes the selected candidate quantization configuration; as well as The output is then transmitted to the user equipment.

16. The system according to claim 1, wherein, Quantizing the trained machine learning model includes generating a new instance of the trained machine learning model, the new instance of the trained machine learning model including the selected candidate quantization configuration for the component, the operation further including: Receive quantization requests from user equipment; and In response to receiving the quantization request, a new instance of the trained machine learning model is generated.

17. The system according to claim 1, wherein, The input dataset includes unlabeled sample data.

18. The system according to claim 1, wherein, The input dataset includes unlabeled sample images.

19. A method comprising: A set of candidate quantization configurations for component identification of a trained machine learning model; For each candidate quantization configuration in the set of candidate quantization configurations: Apply the candidate quantization configuration to the component. After applying the candidate quantization configuration, the trained machine learning model is executed on the input dataset to obtain candidate output values ​​for the candidate quantization configuration, and Based on the comparison between the candidate output value and the reference output value for the component, the loss associated with the candidate quantization configuration is determined; For the component, a candidate quantization configuration is selected from the set of candidate quantization configurations based on the determined loss associated with the set of candidate quantization configurations; as well as At least a portion of the trained machine learning model is quantized using a selected candidate quantization configuration for the component.

20. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform an operation, the operation comprising: A set of candidate quantization configurations for component identification of a trained machine learning model; For each candidate quantization configuration in the set of candidate quantization configurations: Apply the candidate quantization configuration to the component. After applying the candidate quantization configuration, the trained machine learning model is executed on the input dataset to obtain candidate output values ​​for the candidate quantization configuration, and The loss associated with the candidate quantization configuration is determined based on a comparison between the candidate output value and the reference output value for the component. For the component, a candidate quantization configuration is selected from the set of candidate quantization configurations based on the determined loss associated with the set of candidate quantization configurations; as well as At least a portion of the trained machine learning model is quantized using a selected candidate quantization configuration for the component.

Images