Mixed precision neural network

Abstract

Embodiments of the present disclosure relate to mixed precision neural networks. Mixed precision quantization techniques for machine learning (ML) models. For a ML model (114), a target bandwidth increase is received (302), the ML model (114) including objects of a first data type represented by a first number of bits. The target bandwidth increase involves changing a first portion of the objects to a second data type represented by a second number of bits different from the first number of bits (310). The method further includes ordering the objects in the ML model based on bandwidth (304). The method further includes identifying the first portion of the objects to change from the first data type to the second data type based on the target bandwidth increase and the ordering of the plurality of objects (508). The method further includes changing the first portion of the objects from the first data type to the second data type (508).

Description

Technical Field

[0001] This disclosure relates to machine learning, and more specifically to neural networks using values ​​represented with more than one level of precision. Background Technology

[0002] Neural networks and other machine learning (ML) models can include variables that represent numerical values. These numerical values ​​are typically represented as floating-point data types. For example, floating-point values ​​are often used to represent: (1) the weights or learned values ​​of a layer in the network (such as a convolution or inner product), and (2) feature maps, also known as activations or binary large objects (BLOBs). These weights and BLOBs can be used, for example, as the output of a given layer in a neural network and as the input to subsequent layers.

[0003] However, executing neural networks with floating-point values ​​can be computationally expensive (e.g., execution can be slow and power-intensive). Many neural networks involve a large number of floating-point values, and floating-point arithmetic is generally slow and power-intensive. Furthermore, floating-point data types are typically 32 bits or larger, so loading and storing such values ​​requires storage space and time. Therefore, using floating-point values ​​may inhibit the execution of neural networks on devices with limited or power-constrained computing resources, including edge devices such as smartphones and tablets. Summary of the Invention

[0004] The embodiment includes a method. The method includes: receiving a target bandwidth increase for a machine learning (ML) model, the ML model including a plurality of objects of a first data type represented by a first digit. The target bandwidth increase involves changing a first portion of the plurality of objects to a second data type represented by a second digit different from the first digit. The method also includes sorting the plurality of objects in the ML model based on bandwidth. The method further includes: identifying a first portion of the plurality of objects to be changed from the first data type to the second data type based on the target bandwidth increase and the sorting of the plurality of objects. The method also includes changing the first portion of the plurality of objects from the first data type to the second data type.

[0005] The embodiment also includes a system comprising a processor and a memory storing instructions that, when executed by the processor, cause the processor to perform operations. The operations include: receiving a target bandwidth increase for a neural network, the neural network comprising a plurality of binary large objects (BLOBs) and weights of a first data type represented by a first bit number. The target bandwidth increase involves: changing at least some of the plurality of BLOBs and weights to a second data type represented by a second bit number different from the first bit number. The operations also include: identifying a first portion of the plurality of BLOBs and weights, based on the target bandwidth increase, to change from the first data type to the second data type. The operations also include changing the first portion of the plurality of BLOBs and weights from the first data type to the second data type.

[0006] The embodiments also include a non-transitory computer-readable medium including stored instructions that, when executed by a processor, cause the processor to perform operations. The operations include: receiving a target performance change for a machine learning (ML) model, the ML model including a plurality of objects of a first data type represented by a first digit. The target performance change involves: changing at least some of the plurality of objects to a second data type represented by a second digit different from the first digit. The operations also include sorting the plurality of objects in the ML model based on size. The operations further include: identifying a first portion of the plurality of objects to change from the first data type to the second data type based on the target performance change and the sorting of the plurality of objects. The operations also include changing the first portion of the plurality of objects from the first data type to the second data type. The operations further include: using the ML model (including changing the first portion of the plurality of objects to the second data type) to infer the result of one or more inputs. Attached Figure Description

[0007] This disclosure will be more fully understood through the detailed description given below and the accompanying drawings illustrating the examples described herein. The drawings are provided to give knowledge and understanding of the examples described herein and do not limit the scope of this disclosure to these specific examples. Furthermore, the drawings are not necessarily drawn to scale.

[0008] Figure 1 The illustration shows the use of mixed precision values ​​in an ML model according to one embodiment.

[0009] Figure 2 The illustration shows an ML training server and a mixed-precision quantization server according to one embodiment for using mixed-precision values ​​in an ML model.

[0010] Figure 3 This is a flowchart according to one embodiment for using mixed precision values ​​in an ML model.

[0011] Figure 4 This is a flowchart, according to one embodiment, for sorting objects to update mixed precision values ​​for an ML model.

[0012] Figure 5 This is a flowchart, according to one embodiment, for increasing the object size to update the blending precision value used in ML.

[0013] Figure 6 The illustration depicts an example technique for updating an ML model to mixed precision according to one embodiment.

[0014] Figure 7A The illustration shows a table of example objects in a mixed-precision ML model according to one embodiment.

[0015] Figure 7BThe illustration shows a table of example objects in a different mixed-precision ML model according to one embodiment.

[0016] Figure 8 This is a flowchart for mixed-precision quantization perception training according to one embodiment.

[0017] Figure 9 An abstract diagram of an example computer system according to one embodiment is illustrated, in which examples of this disclosure may be operated.

[0018] For ease of understanding, the same reference numerals are used where possible to denote common elements in the figures. It is contemplated that elements disclosed in one embodiment may be advantageously used in other embodiments without specific description. Detailed Implementation

[0019] By converting some or all of the floating-point values ​​to a smaller integer data type, the computational overhead of executing ML models with floating-point data types (e.g., neural networks) can be reduced. This reduces computational overhead but decreases accuracy. Neural networks can then be executed on hardware that supports the required integer data type (e.g., 12-bit integer data type), thus significantly reducing the computational cost of execution.

[0020] Furthermore, neural network hardware can support integer data types of more than one size. For example, some hardware supports 12-bit or 8-bit integer weights and BLOBs. Neural networks can store some objects as 12-bit integer data types and some as 8-bit integer data types. This can be referred to as a mixed-precision implementation.

[0021] Typically, 8-bit integer weights are loaded from memory into the multiplier register much faster than 12-bit integer weights due to their size. Writing 8-bit integer BLOBs to memory is also generally faster when a layer computes its output compared to a 12-bit integer BLOB. Loading 8-bit integer BLOBs into the multiplier register is also faster when a layer takes a previously computed BLOB as input. Furthermore, since 8 is a power of 2 while 12 is not, loading and storing 8-bit integers is generally simpler and cheaper in hardware implementations than loading and storing 12-bit integers.

[0022] However, fewer bits in the representation result in lower accuracy in representing floating-point values, which can reduce the overall inference accuracy of the neural network. Sometimes, representing all values ​​in a neural network with a smaller data type (e.g., 8-bit integer) does not provide sufficient accuracy, so some values ​​are represented using a smaller data type (e.g., 8-bit integer), while others are represented using a larger data type (e.g., 12-bit integer).

[0023] The one or more embodiments disclosed herein provide a number of techniques for selecting a mixed-precision ML model when two precision levels are possible. For example, some weights and BLOBs may be represented using a larger integer data type (e.g., a 12-bit integer data type), while the rest may be represented using a smaller integer data type (e.g., an 8-bit integer data type). The one or more techniques disclosed herein can be used to select which weights and BLOBs should be represented at different precision levels (e.g., which are represented as 8-bit integers and which are represented as 12-bit integers).

[0024] Figure 1 The illustration depicts the use of mixed-precision values ​​for an ML model according to one embodiment. In one embodiment, training data 102 is provided to an ML training server 110. The ML training server 110 includes an ML training service 112 and an initial ML model 114. In one embodiment, the ML training service 112 uses the training data 102 to train the initial ML model 114. For example, the ML training service 112 may convert the training data 102 into a feature vector. The ML training service 112 may use the feature vector and associated attribute values ​​to train the initial ML model 114. This is merely an example; any suitable training data may be used.

[0025] Furthermore, any suitable supervised ML technique can be used to train the floating-point ML model. For example, the initial ML model 114 can be a neural network (e.g., a convolutional neural network (CNN), an adversarial neural network, or any other suitable neural network), and can be trained using any suitable technique. Figure 1 This is just one example.

[0026] In one embodiment, the initial ML model 114 includes variables stored as floating-point data types. For example, the initial ML model 114 could be a neural network with a large number of weights and BLOBs stored as floating-point variables. As mentioned above, using floating-point variables to represent weights and BLOBs can increase accuracy, but can be computationally expensive and power-intensive. Floating-point is merely an example, and the initial ML model 114 can use any appropriate data type (e.g., large integer data type instead of floating-point data type) to represent numerical values.

[0027] In one embodiment, the ML training server 110 provides an initial ML model 114 to the mixed-precision quantization server 120. The ML training server 110 can communicate with the mixed-precision quantization server 120 using any suitable connection, including a local area network (LAN), a wide area network (WAN), the Internet, or a direct connection. Furthermore, as follows regarding… Figure 2Further discussed, in one embodiment, the ML training server and the mixed-precision quantization server 120 can communicate using any suitable technology, including WiFi, cellular connectivity, or wired connections. Furthermore, training data 102 can be provided to the ML training server 110 from any suitable source. For example, the training data can be stored in a remote database (e.g., a relational database), another remote storage location, or stored locally along with the ML training server 110.

[0028] The mixed-precision quantization server 120 includes a mixed-precision quantization service 122. In one embodiment, the mixed-precision quantization service 122 generates a mixed-precision ML model 130 by changing the variable types (e.g., weights and BLOBs) in the initial ML model 114 to different data types that have lower precision and are computationally cheaper. For example, the initial ML model 114 could be a neural network with weights and BLOBs stored as floating-point variables. The mixed-precision quantization service 122 can convert these floating-point variables to integers.

[0029] As an example, the following technique can be used: an integer I can be used to represent a floating-point value F via an implicit scale S using the equation F = I / S. In this example, a floating-point value in the range of 0 to 2 can be represented by a 12-bit integer range of 0 to 4095, with an implicit scale of 4095 / 2 = 2047.5. The integer value 1000 represents the floating-point value 1000 / 2047.5 = .488. The integer 2000 represents the floating-point value 2000 / 2047.5 = .977. The integer value 4095 represents the floating-point value 4095 / 2047.5 = 2.0.

[0030] In one embodiment, mixed-precision quantization service 122 converts floating-point variables in the initial ML model 114 into a hardware-backed integer data type that will be used during the inference phase (e.g., when inferring the result of one or more inputs using the ML model) to execute the mixed-precision ML model 130. For example, the hardware can be configured to support 8-bit and 12-bit integer data types in the ML model. Mixed-precision quantization service 122 can convert floating-point variables in the initial ML model 114 into a mixture of 8-bit and 12-bit integers in the mixed-precision ML model 130. This will be discussed below regarding... Figures 3-6 This will be discussed further. This is just one example. Mixed-precision quantization service 122 can transform any suitable variable, including converting larger floating-point variables to smaller floating-point variables (e.g., converting 32-bit floating-point to 16-bit or 12-bit floating-point), converting larger floating-point variables to smaller floating-point variables and a mixture of floating-point variables and integers (e.g., converting 32-bit floating-point to 16-bit floating-point and 8- or 12-bit integers), converting larger integers to smaller integers, or any other suitable combination.

[0031] Figure 2 The illustration shows an ML training server 110 and a mixed-precision quantization server 120 for applying mixed-precision values ​​to an ML model according to one embodiment. In one embodiment, the ML training server 110 and... Figure 1 This corresponds to the ML training server 110 shown. The ML training server 110 includes a processor 202, memory 210, and network components 220. The processor 202 typically retrieves and executes programming instructions stored in memory 210. The processor 202 can represent a single central processing unit (CPU), multiple CPUs, a single CPU with multiple processing cores, a graphics processing unit (GPU) with multiple execution paths, etc.

[0032] Network component 220 includes the components necessary for the ML training server 110 to interface with the communication network, as described above. Figure 1 As discussed. For example, network component 220 may include wired, WiFi, or cellular network interface components and associated software. Although memory 210 is shown as a single entity, memory 210 may include one or more memory devices having memory blocks associated with physical addresses, such as random access memory (RAM), read-only memory (ROM), flash memory, or other types of volatile and / or non-volatile memory.

[0033] Memory 210 typically includes program code for performing various functions related to the use of ML training server 110. This program code is typically described as various functional "applications" or "modules" within memory 210, but alternative implementations may have different functions and / or combinations of functions. Within memory 210, ML training service 112 facilitates the training of an initial ML model 114. In one embodiment, ML training service 112 and... Figure 1 The ML training service 112 shown corresponds to the initial ML model 114. Figure 1 This corresponds to the initial ML model 114 shown.

[0034] In one embodiment, the mixed-precision quantization server 120 corresponds to Figure 1 The mixed-precision quantization server 120 shown is included. The mixed-precision quantization server 120 includes a processor 252, memory 260, and network components 270. The processor 252 typically retrieves and executes programming instructions stored in memory 260. The processor 252 can represent a single central processing unit (CPU), multiple CPUs, a single CPU with multiple processing cores, a graphics processing unit (GPU) with multiple execution paths, etc.

[0035] Network component 270 includes components necessary for the interface between the mixed-precision quantization server 120 and the communication network, as described above. Figure 1 As discussed. For example, network component 270 may include wired, WiFi, or cellular network interface components and associated software. Although memory 260 is shown as a single entity, memory 260 may include one or more memory devices having memory blocks associated with physical addresses, such as random access memory (RAM), read-only memory (ROM), flash memory, or other types of volatile and / or non-volatile memory.

[0036] Memory 260 typically includes program code for performing various functions related to the use of the mixed-precision quantization server 120. This program code is typically described as various functional "applications" or "modules" within memory 260, but alternative implementations may have different functions and / or combinations of functions. Within memory 260, the mixed-precision quantization service 122 facilitates the conversion of the initial ML model 114 into a mixed-precision ML model 130. In one embodiment, the mixed-precision quantization service 122 corresponds to... Figure 1 The mixed-precision quantization service 122 shown corresponds to the initial ML model 114. Figure 1 The initial ML model 114 shown, and the mixed-precision ML model 130 corresponding to Figure 1 The mixed-precision ML model 130 is shown below. In one embodiment, the initial ML model 114 is converted into a mixed-precision ML model 130 as described below. Figures 3-6 This was discussed further.

[0037] Figure 3 This is a flowchart according to one embodiment for using mixed-precision values ​​in an ML model. At block 302, the mixed-precision quantization service (e.g., Figure 2 The mixed-precision quantization service 122 shown receives a bandwidth target for the ML model (e.g., for a neural network). As discussed further below, in one embodiment, the bandwidth target relates to the expected bandwidth used by the ML model for inference (e.g., the expected size of an object in the ML model, in bits), and allows users (e.g., ML model designers) to generate a preferred mixed-precision implementation by balancing the impact of the mixed-precision implementation on the accuracy of the ML model and the bandwidth required for inference using the ML model. Regarding Figure 3 One or more techniques discussed can be used to generate mixed-precision implementations for a given bandwidth target.

[0038] For example, the initial ML model (e.g., Figure 2The initial ML model 114 shown may include objects (e.g., weights and BLOBs) represented using a larger data type (e.g., floating-point data type). Users can quantize the initial ML model to operate on hardware that supports two smaller data types (e.g., 8-bit integer data type and 12-bit integer data type). If all objects are set to the smaller data type (e.g., using 8-bit integers to represent all weights and BLOBs), the baseline bandwidth required for the ML model will be achieved.

[0039] In one embodiment, at box 302, the user provides the mixed-precision quantization service with a bandwidth target relative to the baseline: for example, the user could request a mixed-precision implementation that uses 5% more bandwidth than the baseline, 10% more bandwidth than the baseline, 25% more bandwidth than the baseline, and so on. This is just one example, and the bandwidth target could also be a defined value (e.g., a size limit for weights and BLOBs in an ML model), a portion of the maximum bandwidth (e.g., a portion of the bandwidth if all weights and BLOBs are represented using a larger data type (e.g., a 12-bit integer)), or any other suitable value. See below regarding... Figure 4 Further discussion suggests that the size adopted by variables in an ML model is a proxy for bandwidth, but this is merely an example. Alternatively or additionally, bandwidth can include qualitative differences between variables (e.g., even if objects have the same size, a BLOB can be considered as having higher bandwidth than its weights). Furthermore, bandwidth is just one example of a performance objective for an ML model. In one embodiment, at box 302, the mixed-precision quantization service can receive any suitable performance objective.

[0040] At box 304, the mixed-precision quantization service sorts objects in the ML model. For example, weights and BLOBs in a neural network can be referred to as objects. At box 304, the mixed-precision quantization service sorts objects (such as weights and BLOBs) in the ML model by bandwidth. For example, the mixed-precision quantization service can sort objects by size (as a representation of bandwidth). This is discussed below regarding... Figure 4 This was discussed further.

[0041] At box 306, the mixed-precision quantization service calculates the baseline object bandwidth. For example, as described above, the mixed-precision quantization service can generate a mixed-precision ML model by quantizing the initial ML model to produce an ML model (e.g., by changing objects represented using a larger data type (e.g., floating-point data type) to one of two smaller data types (e.g., 8-bit integer data type and 12-bit integer data type), by changing objects represented using a larger data type (e.g., floating-point data type and 12-bit integer data type). Figure 2The mixed-precision ML model 130 shown is illustrated. In one embodiment, the baseline object bandwidth is the minimum possible bandwidth for representing all objects in the ML model (e.g., all weights and BLOBs) using smaller data types. For example, assuming the mixed-precision ML model uses 8-bit and 12-bit integer data types, the baseline object bandwidth would be the memory size (e.g., in bits) used to represent all objects in the initial ML model as 8-bit integers (e.g., instead of floating-point values).

[0042] At box 308, the mixed-precision quantization service calculates the target object bandwidth. In one embodiment, the target object bandwidth provides an upper limit on the bandwidth for the mixed-precision ML model. For example, assume the initial ML model uses floating-point data types to represent objects (e.g., weights and BLOBs), and assume the mixed-precision ML model uses a combination of 8-bit and 12-bit integers instead of floating-point values. The baseline object bandwidth discussed above with respect to box 306 is the bandwidth used to represent all floating-point objects as 8-bit integers (e.g., the size of these objects represented as 8-bit integers). The target object bandwidth is the bandwidth target (e.g., a percentage increase) applied to this baseline, received at box 302. For example, assume the baseline object bandwidth is 1 MB, and the bandwidth target is a 10% increase. The target object bandwidth is 1 * 1.1 = 1.1 MB.

[0043] At box 310, the mixed-precision quantization service iteratively increases the object size. For example, the mixed-precision quantization service could begin by assuming that all objects (e.g., all weights and BLOBs) should be converted to a smaller data type (e.g., 8-bit integers). The mixed-precision quantization service could then iteratively select objects, increasing from smaller data types to larger data types (e.g., from 8-bit integers to 12-bit integers), while maintaining a bandwidth below the target object. In one embodiment, the mixed-precision quantization service could use a sorted set of objects generated at box 304 to select which objects should be increased in size. For example, the mixed-precision quantization service could start with objects that consume less bandwidth and iteratively increase the object size for objects that consume increasingly more bandwidth. This is discussed below regarding... Figure 5 This was discussed further.

[0044] In one embodiment, a user can iteratively run [the program] using multiple different bandwidth targets. Figure 3 Publicly available technologies. For example, users can use these technologies to generate a series of mixed-precision ML models that meet different bandwidth objectives, and can analyze the accuracy of the resulting mixed-precision ML models (e.g., using defined test metrics). Users can then select a mixed-precision ML model that optimally balances improved bandwidth with acceptable accuracy (e.g., since a lower bandwidth model may be less accurate due to a loss of accuracy in the weights and BLOBs in the model).

[0045] Figure 4 This is a flowchart, according to one embodiment, for classifying objects to update mixed precision values ​​used in an ML model. In one embodiment, Figure 4 Corresponding to the discussion above Figure 3 Box 304 in the middle. At box 402, the mixed precision quantization service (e.g., Figure 2 The mixed-precision quantization service 122 shown is applied to the initial ML model (e.g., by bandwidth) according to bandwidth. Figure 2 The objects in the initial ML model 114 shown are sorted. As described above, in one embodiment, size is used as a representation of bandwidth. In this embodiment, the mixed-precision quantization service sorts the objects (e.g., weights and BLOBs) in the initial ML model by size.

[0046] For example, a mixed-precision quantization service can sort objects by size from smallest to largest. That is, a mixed-precision quantization service can identify the size of each object in the initial ML model (e.g., the total size of all instances of that object in the ML model, in bits) and sort the objects from smallest to largest (e.g., from the smallest total size of all instances of the object to the largest total size of all instances of the object). Size is merely a representation of bandwidth, and alternatively or additionally, a mixed-precision quantization service can sort objects based on other criteria. For example, suppose a BLOB is considered to have higher bandwidth than weights. Before sorting BLOBs and weights, the BLOB size can be multiplied by an adjustment factor, thereby increasing the bandwidth of the BLOB relative to the weights. This adjustment factor can be determined in any suitable way (e.g., through experimentation or from experience with previous models). As another example, weights or BLOBs in a given layer can be considered to have higher (or lower) bandwidth than other weights or BLOBs and can be multiplied by an adjustment factor. These are just examples; weights can be considered to have higher bandwidth than BLOBs, and other criteria can be used.

[0047] At box 404, the hybrid precision quantization service prioritizes objects with equal bandwidth that are closer to the roots of the ML model's network graph. In one embodiment, the initial ML model can be a neural network represented by a network graph that expresses the proximity of each object on the initial ML model to the root (or multiple roots). Objects farther from the roots of the network graph may depend on objects closer to the roots. In this embodiment, priority can be given to objects closer to the roots to increase the accuracy of the representation of these objects. As an example, if two objects are considered to have equal bandwidth at box 402, the object closer to the roots of the network graph can be placed first in the sorting to make that object more likely to be represented using a larger data type and more likely to be more accurate. Equal bandwidth is just one example. Alternatively or additionally, when comparing objects with similar but not exactly equal bandwidth (e.g., when objects are within each other's tolerance thresholds), the hybrid precision quantization service can prioritize objects closer to the roots of the ML model's network graph.

[0048] For example, suppose a mixed-precision quantization service converts floating-point values ​​in the initial ML model into a mixture of 12-bit and 8-bit integers. Objects closer to the root of the network graph can be prioritized for conversion to 12-bit integers (instead of 8-bit integers) to avoid mixing errors from the root of the network graph down to external nodes. By making objects closer to the root more precise, these objects become more accurate and have fewer mixing errors as they move further down the graph. Assuming two objects are considered to have the same bandwidth at box 402, at box 404, the object closer to the root of the network graph is placed first in this sorting so that it is more likely to be represented by a 12-bit integer. After sorting is complete, the mixed-precision quantization service continues to calculate the bandwidth of the baseline objects (e.g., as shown in the image). Figure 3 (As shown in box 306 in the image).

[0049] Figure 5 This is a flowchart, according to one embodiment, for increasing the object size to update the mixed precision value used in an ML model. In one embodiment, Figure 5 Corresponding to the discussion above Figure 3 Box 310 in the middle. At box 502, the mixed precision quantization service (e.g., Figure 2 The mixed-precision quantization service 122 shown in the figure is derived from the initial ML model (e.g., Figure 2 The initial ML model 114 shown selects the next object in the sorted list of objects. For example, mixed-precision quantization services can... Figure 3 The block 304 shown generates a sorted list of objects (e.g., weights and BLOBs). As described above, in one embodiment, this list is sorted from lowest bandwidth to highest bandwidth (e.g., minimum size to maximum size), and proximity to the root of the network graph is used to break deadlocks (e.g., ...). Figure 4(As described in the text).

[0050] At box 504, if the selected object increases from a smaller data type (e.g., an 8-bit integer) to a larger data type (e.g., a 12-bit integer), the mixed-precision quantization service calculates this size increase for the entire ML model. For example, suppose the selected object has a size of 1MB when represented as an 8-bit integer. If represented as a 12-bit integer, the mixed-precision quantization service can determine that the selected object will have a size of 1.5MB. This 0.5MB increase...

[0051] At box 506, the mixed-precision quantization service determines whether increasing the size makes the total size of objects in the ML model less than or equal to the target size. If not, the total size of the objects will be greater than the target total size if the size of the selected objects increases, and therefore the process ends. If yes, and the total size of the objects will be less than the target total size if the size of the selected objects increases, then the process proceeds to box 508.

[0052] At box 508, the mixed-precision quantization service increases the size of the selected object. The process then returns to box 502. At box 502, the mixed-precision quantization service selects the next object in the sorted list, and then proceeds to box 504.

[0053] For example, as mentioned above... Figure 3 As part of mixed-precision quantization, the ML model discussed here can target a 10% increase in total size. If objects in the ML model have a total size of 10MB when represented as smaller data types, the target total size is 11MB, allowing for a 1MB increase (by converting objects from smaller to larger data types). Assuming as described above, increasing the size of the selected objects from 8-bit integers to 12-bit integers will increase the size of the objects in the ML model by 0.5MB. If the size of the objects remains below the target after increasing the size of the selected objects, then the objects are increased from 8-bit integers to 12-bit integers. Otherwise, the process ends.

[0054] Figure 6 The illustration depicts an example technique for updating an ML model to mixed precision according to one embodiment. In one embodiment, Figure 6 The diagram illustrates pseudocode 600, which represents the above regarding... Figures 3-5 The techniques discussed. The "#" symbol is used to indicate explanatory comments. Figure 6In the example, the size of the object increases from a "small" 8-bit data type (e.g., an 8-bit integer) to a "large" 12-bit data type (e.g., a 12-bit integer). For a given object, the "increment" from small to large is 0.5 (12 / 8 - 1). If all objects are represented using the small data type, then "total_size" is the sum of the object sizes in the initial ML model. "expanded_size" is the total size after applying the target increase (e.g., as mentioned above regarding...). Figure 3 (As discussed in box 302).

[0055] This technique iterates through a list of objects sorted from smallest to largest from the initial ML model. (The above refers to...) Figure 3 and Figure 4 Box 304 in the document discusses this sorting. For each object, an "increment" is determined. This is the increase in the total size if the selected object is converted from a smaller data type to a larger data type. If the sum of the current total_size and the increment is less than or equal to the target expansion size, the object is converted from a smaller data type to a larger data type, total_size is increased by that increment, and the iterative loop continues to sort the next largest object in the list. If not, the technique ends.

[0056] Figure 7A Table 700 illustrates an example object in a mixed-precision ML model according to one embodiment. In one embodiment, Figure 7A The diagram illustrates objects in the example ML model resnet_152. Many ML models (including resnet_152) contain many layers with small weights or BLOBs, making it possible to represent these objects as smaller data types without significantly saving bandwidth. (The above is about...) Figures 3-6 The techniques discussed promote these smaller objects from smaller data types (e.g., 8-bit integers) to larger data types (e.g., 12-bit integers). Users can apply these techniques to a set of target bandwidth increases of his or her choice, and can select the minimum bandwidth increase that produces acceptable results (e.g., acceptable accuracy).

[0057] For example, a user can use a given target bandwidth and then use a new object to evaluate the effectiveness of an ML model (e.g., a neural network). The user can test the inference accuracy of the neural network (e.g., the accuracy of the results inferred by the neural network from one or more inputs) and compare it to an accuracy criterion. This is just one example of an effectiveness criterion, and other criteria (e.g., speed, processing load, etc.) can be used. If the evaluated neural network does not meet the effectiveness criterion, the algorithm can be run again with a modified target bandwidth increase, and the neural network can be modified and tested again. This process can be repeated until the neural network meets the effectiveness criterion (e.g., comparing the accuracy of the neural network to a threshold).

[0058] Table 700 illustrates the use of the above... Figures 3-6 The techniques discussed are converted to a mixed-precision implementation of the ResNet-152 ML model. As an example, the accuracy of ResNet-152 decreases significantly when all objects are converted from floating-point numbers to 8-bit integers. However, as shown in Table 700, the results are very close to those obtained when a 14% increase in bandwidth is allowed relative to the 8-bit integer representation (which would require a 50% increase in bandwidth).

[0059] Column 702 in Table 700 depicts the object number (e.g., label). In one embodiment, the Resnet_152 model includes 368 weights and BLOBs, the selection of which is shown in Table 700. If the object is represented as an 8-bit data type (e.g., an 8-bit integer), column 704 depicts the size of the object in a given row. If all objects are represented as 8-bit data types, column 706 depicts the sum of the object sizes up to a given row (e.g., from object 1 to the given row). Column 708 depicts the percentage of the overall object size of the model used by the objects up to the given row.

[0060] Column 710 describes the object size in the mixed-precision implementation (e.g., after updating the object selection from 8-bit to 12-bit data type). Column 712 depicts the updated object size in the mixed-precision implementation. Column 714 depicts the sum of updates to objects up to a given row. Column 716 depicts the percentage of the overall object size of the model used by the updated objects up to a given row. Column 718 depicts the total expansion of the object size in the model up to a given row after the update. Column 720 depicts the object type (e.g., BLOB or weight).

[0061] For example, as shown in Table 700, objects 1-10 (e.g., shown in rows 1-10) are relatively small and, in the mixed-precision implementation, are converted from 8-bit data types to 12-bit data types, having a relatively small impact on the overall object size. In fact, the first 231 objects are converted from 8-bit data types to 12-bit data types, while objects 232-368 remain 8-bit data types. This results in a total expansion of 1.139 (e.g., 13.9%), as shown in row 368 (i.e., the last row) of column 718.

[0062] Figure 7B Table 750 illustrates example objects in a different mixed-precision ML model according to one embodiment. In one embodiment, Figure 7B The illustration shows objects in the example ML model mobilenet_ssd, which is a scene segmentation and classification model that computes boxes (e.g., segments) around objects and classifies the contents of each box into one of 90 distinct objects. As illustrated, a 10% increase in bandwidth (e.g., size) relative to a full 8-bit representation produces results as efficient as converting all objects to a 12-bit representation.

[0063] Just like in Figure 7A Similar to Table 700 shown, column 702 in Table 750 depicts the object number (e.g., label). In one embodiment, mobilenet_ssd includes 111 weights and BLOBs, and the selection of them is illustrated in Table 750. If the object is represented as an 8-bit data type (e.g., an 8-bit integer), column 704 depicts the size of the object in a given row. If all objects are represented as 8-bit data types, column 706 depicts the sum of the object sizes up to a given row (e.g., from object 1 to the given row). Column 708 depicts the percentage of the overall object size of the model used by the objects up to the given row.

[0064] Column 710 depicts the object size in the mixed-precision implementation (e.g., after updating the object selection from 8-bit to 12-bit data type). Column 712 depicts the updated object size in the mixed-precision implementation. Column 714 depicts the sum of updates to objects up to a given row. Column 716 depicts the percentage of the overall object size of the model used by the updated objects up to a given row. Column 718 depicts the total expansion of the object size in the model up to a given row after the update. And column 720 depicts the object type (e.g., BLOB or weight).

[0065] For example, as shown in Table 750, objects 1-7 (e.g., depicted in rows 1-7) are relatively small and, in the mixed-precision implementation, are converted from 8-bit data types to 12-bit data types, having a relatively small impact on the overall object size. In fact, the first 81 objects are converted from 8-bit data types to 12-bit data types, while objects 82-111 remain 8-bit data types. This results in a total expansion of 1.095 (e.g., 9.5%), as shown in row 111 (the last row) of column 718.

[0066] In one embodiment, the target bandwidth increase varies depending on the implementation. For example, for 8-bit and 12-bit values, an increment of 5% (from 105% to 150%) could be used. This is merely one example. Furthermore, in one embodiment, after a promising result has been selected using an initial percentage increment (e.g., a 5% percentage increment), the target bandwidth increase can be varied around the result. For example, if 115% appears to be a promising result, then the above regarding... Figures 3-6 The technique shown can check 113%, 114%, 116%, and 117%.

[0067] In one embodiment, one or more of the techniques described above can be used to tune a trained neural network to increase the efficiency of the network in performing inference operations. For example, the algorithms described above can be used to tune a trained neural network to determine the bit size of various weights and BLOBs to be used in the trained neural network. Alternatively or additionally, one or more of the techniques described above can be used to tune the training of a neural network. For example, one or more of the techniques described above can be used for quantization-aware training of an ML model.

[0068] Figure 8 This is a flowchart 800 for mixed-precision quantization-aware training according to one embodiment. In one embodiment, converting an initial ML model to an updated ML model using different data types (e.g., converting floating-point objects to smaller integer objects) can be referred to as quantization. Quantization-aware training can be used to consider quantization during the training of the ML model to increase the accuracy of inference using the finally trained ML model. In one embodiment, quantization-aware training simulates low-precision inference-time computation in the forward pass of the training process. For example, quantization error can be introduced as noise during training as well as as part of the overall loss (e.g., optimization algorithms can attempt to minimize it). The ML model can then learn parameters that are more robust to quantization.

[0069] In one embodiment, the above regarding Figures 3-6 The described mixed-precision technique can be used to improve quantization-aware training. At box 802, the ML training service (e.g., Figure 2The ML training service 112 shown in the figure begins quantization-aware training. In one embodiment, any suitable technique can be used.

[0070] At box 804, the ML training service applies mixed-precision quantization during training. For example, quantization-aware training can be designed to introduce quantization error for mixed-precision (rather than single-precision) quantization during training. (The above refers to...) Figures 3-6 The described technique can be used to determine the mixed-precision implementation and appropriate error values ​​(e.g., introduced as noise during training). Quantization-aware training can then use these error values ​​to train the ML model to be robust to mixed-precision quantization. At box 806, the ML training service completes the training.

[0071] Figure 9 An example of a computer system 900 is illustrated, in which a set of instructions can be executed to cause the computer system to perform any or one of the methods discussed herein. In some implementations, the computer system may be connected (e.g., networked) to other machines or computer systems in a local area network (LAN), intranet, extranet, and / or the Internet. The computer system may operate as a server or client computer system in a client-server network environment, as a peer-to-peer computer system in a peer-to-peer (or distributed) network environment, or as a server or client computer system in a cloud computing infrastructure or environment.

[0072] A computer system can be a personal computer (PC), tablet computer, set-top box (STB), personal digital assistant (PDA), cellular phone, network device, server, network router, switch, or bridge, or any machine capable of executing a set of instructions (sequential or other instructions) that specifies the operations to be performed by the computer system. Furthermore, while a single computer system is illustrated, the term "computer system" should also be considered to include any collection of computer systems that individually or collectively execute a set (or more) of instructions to perform any or more of the methods discussed herein.

[0073] Example computer system 900 includes processing device 902, main memory 904 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM)), static memory 906 (e.g., flash memory, static random access memory (SRAM), etc.), and data storage device 918, which communicate with each other via bus 930. Main memory 904 includes or is a non-transitory computer-readable medium. Main memory 904 (e.g., a non-transitory computer-readable medium) may store one or more sets of instructions 926, which, when executed by processing device 902, cause processing device 902 to perform some or all of the operations, steps, methods, and processes described herein.

[0074] Processing device 902 represents one or more processors, such as microprocessors, central processing units, etc. More specifically, processing device 902 may be or include complex instruction set computing (CISC) microprocessors, reduced instruction set computing (RISC) microprocessors, very long instruction word (VLIW) microprocessors, processors implementing other instruction sets, or processors implementing combinations of instruction sets. Processing device 902 may also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, etc. Processing device 902 may be configured to execute instructions 926 for performing some or all of the operations, steps, methods, and processes described herein.

[0075] The computer system 900 may also include a network interface device 908 for communication via a network 920. The computer system 900 may also include a video display unit 910 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 912 (e.g., a keyboard), a cursor control device 914 (e.g., a mouse), a graphics processing unit 922, a signal generation device 916 (e.g., a speaker), a graphics processing unit 922, a video processing unit 928, and an audio processing unit 932.

[0076] Data storage device 918 may include machine-readable storage medium 924 (e.g., non-transitory computer-readable medium) storing one or more sets of instructions 926 that implement any one or more of the methods or functions described herein. During execution of instructions 926 by computer system 900, instructions 926 may also reside wholly or at least partially in main memory 904 and / or in processing device 902, which also include machine-readable storage media.

[0077] In some embodiments, instruction 926 includes instructions for performing the functions described above. Although machine-readable storage medium 924 is shown as a single medium in the example embodiments, the term "machine-readable storage medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more sets of instructions. The term "machine-readable storage medium" should also be considered to include any medium capable of storing or encoding a set of instructions for execution by a computer system and causing the computer system and processing device 902 to perform any one or more of the methods described above. Therefore, the term "machine-readable storage medium" should be considered to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0078] Some parts of the foregoing detailed description have been presented based on the algorithms and symbolic representations of operations on data bits within computer memory. These algorithmic descriptions and representations are the most effective way for those skilled in the art of data processing to communicate the essence of their work to others skilled in the art. An algorithm can be a sequence of operations that lead to a desired result. These operations are operations that require physical manipulation of physical quantities. Such quantities can take the form of electrical or magnetic signals that can be stored, combined, compared, and otherwise manipulated. Such signals can be referred to as bits, values, elements, symbols, characters, items, numbers, etc.

[0079] However, it should be remembered that all these and similar terms should be associated with appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise expressly indicated or otherwise apparent under this disclosure, it should be understood that throughout this description, some terms refer to the actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical (electronic) quantities in the registers and memories of the computer system and convert them into other data similarly represented as physical quantities within the memory or registers or other such information storage devices of the computer system.

[0080] This disclosure also relates to means for performing the operations described herein. Such means may be specifically constructed for the intended purpose, or may include a computer selectively activated or reconfigured by a computer program stored in the computer. This computer program may be stored in a computer-readable storage medium, such as, but not limited to, any type of disk (including floppy disks, optical disks, CD-ROMs, and magneto-optical disks), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic cards, or optical cards, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0081] The algorithm and display presented herein are not inherently related to any particular computer or other device. Various other systems may be used with the program based on the teachings herein, or it may be convenient to construct more specialized devices to perform the method. Furthermore, no specific programming language is referenced in describing this disclosure. It should be understood that the teachings of this disclosure as described herein can be implemented using various programming languages.

[0082] This disclosure may be provided as a computer program product or software, which may include a machine-readable medium having instructions stored thereon, the instructions being used to program a computer system (or other electronic device) to perform processes according to this disclosure. Machine-readable media include any mechanism for storing information in a machine-readable (e.g., computer-readable) form. For example, machine-readable (e.g., computer-readable) media include machine-readable (e.g., computer-readable) storage media, such as read-only memory (ROM), random access memory (RAM), disk storage media, optical storage media, flash memory devices, etc.

[0083] In the foregoing disclosure, embodiments of the present disclosure have been described with reference to specific exemplary embodiments. It will be apparent that various modifications may be made thereto without departing from the broader spirit and scope of the present disclosure as set forth in the appended claims. Where elements are referred to in the singular in this disclosure, more than one element may be depicted in the drawings, and the same elements may be labeled with the same reference numerals. Therefore, this disclosure and the drawings should be interpreted in an illustrative sense rather than a restrictive one.

[0084] In view of the foregoing, the scope of this disclosure is defined by the appended claims.

Claims

1. A method comprising: Receive a target bandwidth increase for a machine learning (ML) model, the ML model comprising a plurality of objects of a first data type represented by a first bit number, wherein the target bandwidth increase involves changing a first portion of the plurality of objects to a second data type represented by a second bit number different from the first bit number, and changing a second portion of the plurality of objects to a third data type represented by a third bit number different from both the first bit number and the second bit number. Based on bandwidth, the plurality of objects in the ML model are sorted, including: For a first object among the plurality of objects in the ML model, identify the total bandwidth of all instances of the first object in the ML model; Based on the target bandwidth increase and the sorting of the plurality of objects, the following two are identified: (i) the first portion of the plurality of objects in the ML model, changing from the first data type to the second data type, and (ii) the second portion of the plurality of objects in the ML model, changing from the first data type to the third data type, including: Based on maintaining the total bandwidth of the ML model at or below the target bandwidth increase, and using the sorted plurality of objects, selecting both the first portion and the second portion from the plurality of objects; and Based on the selection of both the first part and the second part, the processor changes the first part of the plurality of objects from the first data type to the second data type, and changes the second part of the plurality of objects from the first data type to the third data type.

2. The method of claim 1, wherein the ML model comprises a neural network, and wherein the plurality of objects comprises a plurality of weights and a plurality of binary large objects (BLOBs).

3. The method of claim 1, wherein the third data type comprises more bits than the second data type and fewer bits than the first data type, and wherein selecting both the first portion and the second portion comprises: The total bandwidth of the ML model is determined based on changing the plurality of objects to the second data type. as well as Based on the increase in the target bandwidth, one or more of the plurality of objects are iteratively identified to change to the third data type instead of the second data type.

4. The method according to claim 1, wherein the first data type includes a floating-point data type, wherein the second data type includes a first integer data type, and wherein the third data type includes a second integer data type.

5. The method of claim 1, wherein ranking the plurality of objects in the ML model based on bandwidth comprises: The objects are sorted based on their size.

6. The method of claim 5, wherein the ML model comprises a neural network, and wherein the sorting of the plurality of objects in the ML model is further based on the proximity of the respective object to the root of the network graph representing the neural network.

7. A system comprising: processor; as well as The memory stores instructions that, when executed by the processor, cause the processor to perform operations, the operations including: Receive a target bandwidth increase for a neural network, the neural network comprising a plurality of binary large objects (BLOBs) and weights of a first data type represented by a first bit number, wherein the target bandwidth increase involves changing a first portion of the plurality of BLOBs and weights to a second data type represented by a second bit number different from the first bit number, and changing a second portion of the plurality of BLOBs and weights to a third data type represented by a third bit number different from both the first bit number and the second bit number. Based on the target bandwidth increase, the following two are identified: (i) a first portion of the plurality of BLOBs and weights in the neural network, to be changed from the first data type to the second data type, and (ii) a second portion of the plurality of BLOBs and weights in the neural network, to be changed from the first data type to the third data type, including: For the first BLOB or weight and the second BLOB or weight in the neural network, identify the total bandwidth of all instances of the first BLOB or weight in the neural network; and Based on maintaining the total bandwidth of the neural network at or below the target bandwidth increase, and maintaining the identified first BLOB or weight and second BLOB or weight, both the first portion and the second portion are selected from the plurality of BLOBs and weights; and Based on the selection of both the first part and the second part, the first part of the plurality of BLOBs and weights is changed from the first data type to the second data type, and the second part of the plurality of BLOBs and weights is changed from the first data type to the third data type.

8. The system of claim 7, wherein the operation further comprises: The plurality of BLOBs and weights in the neural network are sorted based on bandwidth, wherein the sorting includes: for the first BLOB or weight in the plurality of BLOBs and weights in the neural network, the total bandwidth identifying all instances of the first BLOB or weight in the neural network, and The identification of the first portion of the plurality of BLOBs and weights to change from the first data type to the second data type, and the identification of the second portion of the plurality of BLOBs and weights to change from the first data type to the third data type, are also based on the sorting of the plurality of BLOBs and weights.

9. The system of claim 7, wherein the third data type comprises more bits than the second data type and fewer bits than the first data type, and wherein selecting both the first portion and the second portion comprises: The total bandwidth of the neural network is determined based on changing the plurality of objects to the second data type. as well as Based on the increase in the target bandwidth, one or more of the plurality of objects are iteratively identified to change to the third data type instead of the second data type.

10. The system of claim 7, wherein the first data type includes a floating-point data type, wherein the second data type includes a first integer data type, and wherein the third data type includes a second integer data type.

11. The system according to claim 7, wherein the operation further comprises: After changing the first portion of the plurality of BLOBs and weights from the first data type to the second data type, it is determined that the neural network does not meet the validity criteria, and in response: The target bandwidth has been increased. Based on the modified target bandwidth increase, a third portion of the plurality of BLOBs and weights that is different from the first portion of the plurality of BLOBs and weights is identified to change from the first data type to the second data type; as well as Change the third part of the plurality of BLOBs and weights from the first data type to the second data type.

12. A non-transitory computer-readable medium comprising stored instructions, which, when executed by a processor, cause the processor to perform operations, the operations including: Receive a target performance change for a machine learning ML model, the ML model comprising a plurality of objects of a first data type represented by a first bit number, wherein the target performance change involves changing a first portion of the plurality of objects to a second data type represented by a second bit number different from the first bit number, and changing a second portion of the plurality of objects to a third data type represented by a third bit number different from both the first bit number and the second bit number. Sort the plurality of objects in the ML model based on size, including: For a first object among the plurality of objects in the ML model, identify the total bandwidth of all instances of the first object in the ML model; Based on the target performance change and the sorting of the plurality of objects, the following two are identified: (i) the first portion of the plurality of objects in the ML model, changing from the first data type to the second data type, and (ii) the second portion of the plurality of objects in the ML model, changing from the first data type to the third data type, including: Based on maintaining the performance of the ML model to meet the target performance change and using the sorted plurality of objects, both the first part and the second part are selected from the plurality of objects; Based on the selection of both the first part and the second part, the first part of the plurality of objects is changed from the first data type to the second data type, and the second part of the plurality of objects is changed from the first data type to the third data type; and Using the ML model, including changing the first portion of the plurality of objects to the second data type and changing the second portion of the plurality of objects to the third data type, to infer results from one or more inputs.

13. The non-transitory computer-readable medium of claim 12, wherein the ML model comprises a neural network, and wherein the plurality of objects comprises a plurality of weights and a plurality of binary large objects (BLOBs).

14. The non-transitory computer-readable medium of claim 12, wherein the third data type comprises more bits than the second data type and fewer bits than the first data type, and wherein selecting both the first portion and the second portion comprises: The overall performance of the ML model is determined based on changing the plurality of objects to the second data type. as well as Based on the target performance change, one or more of the plurality of objects are iteratively identified to change to the third data type instead of the second data type.

15. The non-transitory computer-readable medium of claim 12, wherein the first data type includes a floating-point data type, wherein the second data type includes a first integer data type, and wherein the third data type includes a second integer data type.

16. The non-transitory computer-readable medium of claim 15, wherein the floating-point data type includes a 32-bit floating-point data type, the first integer data type includes a 12-bit integer data type, and the second integer data type includes an 8-bit integer data type.

17. The non-transitory computer-readable medium of claim 12, wherein the ML model comprises a neural network, and wherein the sorting of the plurality of objects in the ML model is also based on the proximity of the respective object to the root of the network graph representing the neural network.

Images