Weight Layout Transformation-Assisted Nested Loop Optimization for AI Inference

Abstract

Various embodiments include methods and apparatus for weight layout transformation of a weight tensor. Embodiments may include: accessing a first memory to retrieve the weights of a weight tensor in a transformed order different from the order used to retrieve the weights for computation at a network layer of a trained machine learning model; and loading the weights into a second memory in the transformed order. Embodiments may further include: retrieving the weights from the second memory in the transformed order; and reordering the weights into the order used to implement computation at the network layer of the trained machine learning model.

Description

[0001] background

[0002] Various types of computing hardware (such as ultra-low power processors, such as sensor digital signal processors (DSPs), modem DSPs, memory control units (MCUs), etc.) use trained neural networks to generate inferences in various applications. Executing trained neural networks by such computing hardware is expensive because computing resources (such as computing power and memory space and bandwidth) are limited. The trained neural network may require expensive nested loop execution, which can burden the computing resources of such computing hardware.

[0003] Overview

[0004] Various aspects may include methods and apparatus for weight layout transform-assisted nested loop optimization for artificial intelligence (AI) inference. Various aspects may include: accessing a first memory to retrieve weights of a weight tensor in a transformed order different from the order used to retrieve the weights for computation at a network layer of a trained machine learning model; and loading the weights into a second memory in the transformed order.

[0005] In some aspects, accessing the first memory to retrieve the weights of the weight tensor in a transformed order that is different from the order in which the weights were retrieved for computations at the network layer of the trained machine learning model may include accessing the first memory to retrieve the weights according to a memory access pattern that iterates over the slowest changing dimension of the weight tensor.

[0006] In some aspects, accessing the first memory to retrieve the weights according to a memory access pattern that iterates over a slowest varying dimension of the weight tensor may include retrieving the weights according to a memory access pattern that iterates over a height dimension of the weight tensor.

[0007] In some aspects, accessing the first memory to retrieve the weights of the weight tensor in a transformed order that is different from the order used to retrieve the weights for a computation at a network layer of the trained machine learning model may include accessing the first memory to retrieve the weights of the weight tensor in an order specified by a first counter variable and a second counter variable of a first memory access command, wherein the first counter variable and the second counter variable are configured to represent a position in the weight tensor, and wherein the first counter variable and the second counter variable are transposed relative to a second memory access command having the first counter variable and the second counter variable for the network layer of the trained machine learning model.

[0008] In some aspects, loading the weights into the second memory in the transformed order may include loading the weights into the second memory in a linear layout according to a memory access pattern that iterates over the slowest varying dimension of the weight tensor.

[0009] In some aspects, loading the weights into the second memory in a linear layout may include loading the weights into the second memory as a linear array.

[0010] Some aspects may further include: retrieving the weights from the second memory in the transformed order; and reordering the weights into an order for implementing calculations at the network layer of the trained machine learning model.

[0011] In some aspects, the first memory and the second memory may be in the same memory device.

[0012] Further aspects include a computing device having a processing device configured to perform the operations of any of the methods outlined above. Further aspects include a computing device having means for performing the functions of any of the methods outlined above. Further aspects include a non-transitory processor-readable medium having stored thereon processor-executable instructions configured to cause a processor or other component of a computing device to perform the operations of any of the methods outlined above. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated herein and constitute a part of this specification, illustrate example embodiments among the various embodiments and, together with the general description given above and the detailed description given below, serve to explain features of the claims.

[0015] Figure 1 is a component block diagram illustrating an example computing device suitable for implementing various embodiments.

[0016] Figure 2A and 2B is a block diagram illustrating an example of a weight tensor transformation suitable for implementing various embodiments.

[0017] Figure 3 is a component block diagram and flow chart illustrating execution of an example trained machine learning model using a linear weight layout of weight tensors suitable for implementing various embodiments.

[0018] Figure 4 1 is a component block diagram and flow chart illustrating an example system suitable for implementing various embodiments for implementing source code of a trained machine learning model configured for weight tensor transformation.

[0019] Figure 5 is a computer code block diagram illustrating an example of computer code for source code of a trained machine learning model configured for weight tensor transformation suitable for implementing various embodiments.

[0020] Figure 6is a process flow diagram illustrating a method for executing a trained machine learning model using weight tensor transformations according to some embodiments.

[0021] Fig. 7A and 7B is a process flow diagram illustrating a method for weight layout transformation according to some embodiments.

[0022] Figure 8 is a process flow diagram illustrating a method for weight layout transformation according to some embodiments.

[0023] Fig. 9 is a process flow diagram illustrating a method for weight correction according to some embodiments.

[0024] Fig.10 is a component block diagram illustrating an example mobile computing device suitable for implementing the various embodiments.

[0025] Fig.11 is a component block diagram illustrating an example mobile computing device suitable for implementing the various embodiments.

[0026] Fig.12 is a component block diagram illustrating an example server suitable for implementing various embodiments.

[0027] Detailed Description

[0028] Various embodiments will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. References to specific examples and implementations are for illustrative purposes and are not intended to limit the scope of the claims.

[0029] Various embodiments include methods for weight layout transformation-assisted nested loop optimization for artificial intelligence (AI) inference and computing devices implementing such methods. Various embodiments may include transforming a weight tensor into a linear format (such as a linear array) in a memory for access during execution of a trained neural network. In some embodiments, the weight transformation from a weight tensor to a linear format may be implemented by modifying source code for the trained neural network. In some embodiments, the modification to the source code for the trained neural network may include a modification to a memory access mode configured to reduce the number of iterations of a nested loop for retrieving weights for implementing the trained neural network. In some embodiments, the memory access mode may be configured to retrieve weights according to the slowest rate of change of weight values ​​organized in a weight tensor. Some embodiments may include correcting a weight retrieval mode from a linear format to enable a trained neural network to generate inferences.

[0030] The terms "computing device" and "mobile computing device" are used interchangeably herein to refer to any or all of the following: a cellular telephone, a smart phone, a personal or mobile multimedia player, a personal data assistant (PDA), a laptop computer, a tablet computer, a convertible laptop / tablet computer (2-in-1 computer), a smartbook, an ultrabook, a netbook, a palmtop computer, a wireless email receiver, an Internet-enabled multimedia cellular telephone, a mobile game console, a wireless game controller, and similar personal electronic devices that include a memory and a programmable processor. The term "computing device" may further refer to a stationary computing device, including a personal computer, a desktop computer, an all-in-one computer, a workstation, a supercomputer, a mainframe computer, an embedded computer (such as embedded in a vehicle and other larger systems), a server, a multimedia computer, and a game console.

[0031] The terms "edge processing device" and "edge processor" are used interchangeably herein to refer to a processing device that can use existing dedicated firmware tool chains and implement machine learning model processing locally on a computing device, for which the machine learning model needs to be adapted for use with the existing dedicated firmware tool chains to be implemented by the processing device. The edge processing device may have limited compiler capabilities, memory, and / or processing power. The edge processing device may refer to any or all of a low-power processor, a sensor digital signal processor, a modem digital signal processor, a memory control unit, an embedded processor, a controller, a microcontroller, and the like.

[0032] Various software vendors have developed and trained machine learning models that can be implemented on computing devices developed by computing device developers. For example, trained machine learning models may include Keras, TensorFlow, TensorFlow Lite, PyTorch, Caffe, Caffe 2, MXNet, Android Neural Network API, Snapdragon Neural Processing Engine (SNPE), etc. Such machine learning models are usually distributed with software development kit (SDK) libraries for implementation on computing devices. General-purpose processors (such as central processing units) can use various compilers that are configured to compile software developed using machine learning model SDK libraries and execute the compiled software.

[0033] However, many edge processing devices may have limited capabilities to use the machine learning model SDK, and the trained machine learning model may be converted into source code that can be compiled and implemented by the edge processing device. For example, the trained machine learning model may be converted into source code that can be compiled and implemented by the edge processing device, as described in International Patent Application No. PCT / CN2020 / 095790 filed on June 12, 2020, the entire contents of which are incorporated herein by reference as background.

[0034] The trained machine learning model source code (referred to herein as the network construction source code) can be implemented using nested loops. Loops play an important role in improving execution speed and reducing the overhead associated with the execution of the trained machine learning model. For example, nested loops can be used to implement the various layers of the trained machine learning model to traverse the tensors for input to these layers and perform calculations for these layers. However, the compiler is inefficient for nested loop optimization. Inference generation using the trained machine learning model is based on nested loops using dynamic inputs (such as feature maps) and static inputs (such as weights). The order in which the nested loops are implemented and the order in which the implementation of the nested loops accesses the inputs may result in high rates of dynamic memory allocation and writing, which consumes memory resources such as space, bandwidth, and electrical power.

[0035] In embodiments described herein, methods and computing devices implementing such methods may implement a transformation of a weight tensor for a trained neural network via a memory access pattern that is configured to reduce the number of iterations of a nested loop for retrieving weights for implementing the trained neural network. The transformation may be implemented using a modified network construction source code (referred to herein as a weight layout transformer) configured to access weights in a memory access pattern (which may be different from the memory access pattern of the original network construction source code). The memory access pattern may use a static memory allocation that requires less memory resources than multiple dynamic allocations. The memory access pattern may also be a sequential memory access that reduces the number of times the memory is written and read to retrieve the weights. For example, the memory access pattern may allow each weight to be accessed sequentially, such as using a stride 1 reference pattern (such as a major row layout) of a linear layout of a multidimensional weight tensor. In some embodiments, the memory access pattern for weights may be the same as the memory access pattern for dynamic inputs.

[0036] The memory access pattern may access the weights in a transformed order that is different from the order that may be expected to be used to implement the calculations of the layers of the trained neural network. Using weights received in an unexpected order to implement the calculations of the layers may produce incorrect calculation results. To account for the difference in memory access patterns, the weight corrector may modify the order in which the weights retrieved from the memory are provided to implement the calculations of the layers. The weight corrector may ensure that the weights are used in the expected order to implement the calculations of the layers.

[0037] The weight layout transformer and weight corrector can be written in the same high-level programming language as the original network construction source code, such as C, C++, Java, Pascal, COBOL, BASIC, etc. The high-level programming language can make the compiler for the language implementable by the edge processing device, and make the trained machine learning model implementable in the software created using the existing dedicated firmware tool chain of the edge processing device without adapting the machine learning model SDK and the edge processing device hardware. The weight layout transformer and weight corrector can be used for software created using the existing dedicated firmware tool chain of the edge processing device without using the machine learning model SDK library. Using the disclosed embodiments can reduce the time to market of the edge processing device capable of implementing the trained machine learning model. Using a high-level programming language can make it faster and easier to test and debug the weight layout transformer and weight corrector, as well as the software that implements the trained machine learning model generated using the weight layout transformer and weight corrector and the existing dedicated firmware tool chain of the edge processing device. In addition, the weight layout transformer and weight corrector are portable to any edge processing device configured to compile and implement the programming language of the weight layout transformer and weight corrector.

[0038] Figure 1 A system including a computing device 100 suitable for use with various embodiments is illustrated. The computing device 100 may include a SoC 102 having a processor 104, a memory 106, a communication interface 108, a memory interface 110, a peripheral device interface 120, and an edge processor 124. The computing device 100 may further include a communication component 112 (such as a wired or wireless modem), a memory 114, an antenna 116 for establishing a wireless communication link, and / or a peripheral device 122. The processor 104 may include any of a variety of processing devices (e.g., a number of processing cores).

[0039] The term "system on chip" or "SoC" is used herein to refer to a group of interconnected electronic circuits, typically but not exclusively including a processing device, a memory, and a communication interface. The processing device may include a variety of different types of processors 104 and / or processor cores, such as a general-purpose processor, a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), an accelerated processing unit (APU), a security processing unit (SPU), a subsystem processor for a specific component of the computing device (such as an image processor for a camera subsystem or a display processor for a display), an auxiliary processor, a single-core processor, a multi-core processor, a controller, and / or a microcontroller. The processing device may further implement other hardware and hardware combinations, such as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), other programmable logic devices, discrete gate logic, transistor logic, performance monitoring hardware, watchdog hardware, and / or a time reference. The integrated circuit may be configured so that the components of the integrated circuit reside on a single piece of semiconductor material (such as silicon).

[0040] SoC 102 may include one or more processors 104. Computing device 100 may include more than one SoC 102, thereby increasing the number of processors 104 and processor cores. Computing device 100 may also include processors 104 that are not associated with SoC 102. Individual processors 104 may be multi-core processors. Processors 104 may each be configured for a specific purpose that may be the same or different from other processors 104 of computing device 100. One or more of processors 104 and processor cores of the same or different configurations may be grouped together. A group of processors 104 or processor cores may be referred to as a multi-processor cluster.

[0041] The memory 106 of the SoC 102 may be a volatile or non-volatile memory configured to store data and processor executable code for access by the processor 104 or other components of the SoC 102, including the edge processor 124. The computing device 100 and / or the SoC 102 may include one or more memories 106 configured for various purposes. One or more memories 106 may include volatile memory, such as random access memory (RAM) or main memory, or cache memory. These memories 106 may be configured to temporarily save a limited amount of data received from a data sensor or subsystem, data and / or processor executable code instructions requested from non-volatile memory, loaded from non-volatile memory to the memory 106 in anticipation of future access based on various factors, and / or intermediate processing data and / or processor executable code instructions generated by the processor 104 and / or edge processor 124 and temporarily stored for future quick access without being stored in non-volatile memory. In some embodiments, any number and combination of memories 106 may include one-time programmable or read-only memory.

[0042] The memory 106 may be configured to at least temporarily store data and processor-executable code loaded to the memory 106 from another memory device, such as another memory 106 or memory 114, for access by one or more processors 104 or by other components of the SoC 102, including the edge processor 124. The data or processor-executable code loaded to the memory 106 may be loaded in response to execution of a function by the processor 104 or by other components of the SoC 102, including the edge processor 124. The loading of the data or processor-executable code to the memory 106 in response to the execution of the function may result from an unsuccessful memory access request (or “miss”) to the memory 106 because the requested data or processor-executable code is not located in the memory 106. In response to the miss, a memory access request may be made to another memory 106 or memory 114 to load the requested data or processor-executable code from the another memory 106 or memory 114 to the memory 106. Loading data or processor-executable code to memory 106 in response to execution of a function may originate from a memory access request to another memory 106 or memory 114, and the data or processor-executable code may be loaded to memory 106 for later access.

[0043] The memory interface 110 and the memory 114 may work together to allow the computing device 100 to store data and processor executable code on volatile and / or non-volatile storage media and retrieve data and processor executable code from the volatile and / or non-volatile storage media. The memory 114 may be configured very similarly to the embodiment of the memory 106, where the memory 114 may store data or processor executable code for access by one or more processors 104 or by other components of the SoC 102 (including the edge processor 124). In some embodiments, the non-volatile memory 114 may retain information after the computing device 100 has been powered off. When power is restored and the computing device 100 is rebooted, the information stored on the memory 114 is available to the computing device 100. In some embodiments, the volatile memory 114 may not retain information after the computing device 100 has been powered off. The memory interface 110 may control access to the memory 114 and allow the processor 104 or other components of the SoC 102 , including the edge processor 124 , to read data from and write data to the memory 114 .

[0044] The SoC 102 may also include any number of edge processors 124. The edge processor 124 may be a processing device that may use existing dedicated firmware toolchains, for which the machine learning model needs to be adapted to work with these existing dedicated firmware toolchains to be implemented by the edge processor 124. The edge processor may implement the machine learning model processing locally on the computing device 100. The edge processor 124 may have limited compiler capabilities, memory, and / or processing power compared to non-low-power processors (such as non-low-power CPUs, GPUs, etc.).

[0045] The edge processor 124 may include any of a low-power processor, a sensor DSP, a modem DSP, a memory control unit (MCU), an embedded processor, a controller, a microcontroller, etc. The edge processor(s) 124 may be individual components of the SoC 102 and / or integral components of other SoC components, such as the communication interface 108, the memory interface 110, and / or the peripheral device interface 120. The computing device 100 may also include an edge processor 124 that is not associated with the SoC 102. Such an edge processor 124 may be a stand-alone component of the computing device 100 and / or integrated into other SoCs 102 and / or other computing device components (such as the communication component 102 and the peripheral device 122).

[0046] Some or all components of computing device 100 and / or SoC 102 may be arranged and / or combined differently while still providing the functionality of various embodiments. Computing device 100 may not be limited to one of each component, and multiple instances of each component may be included in various configurations of computing device 100.

[0047] Figure 2A and 2B An example of a weight tensor transformation suitable for implementing various embodiments is described. Figure 1 , 2A 2B, the weight tensor 200 can be a multi-dimensional organization of weights. For example, Figure 2A The weight tensor 200 illustrated in is a three-dimensional organization of weights having a height, a width, and a depth. The height may correspond to the number of rows, the width may correspond to the number of columns, and the depth may correspond to the number of channels of the weight tensor 200. In this example, the weight tensor may have a height of three rows of weights, a width of three columns of weights, and a depth of three channels of weights. The weight tensor 200 may also be represented as two-dimensional channels 202a, 202b, 202c. Each channel 202a, 202b, 202c may represent the depth of the weight tensor 200, and each channel 202a, 202b, 202c may have the same height and width of the weight tensor 200. In Figure 2A In the example illustrated in , similar shaded portions of channels 202a, 202b, 202c may represent corresponding positions of weights in channels 202a, 202b, 202c.

[0048] During execution of the network construction source code for the trained machine learning model, the memory (e.g., Figure 1 The memory 106 in the network is dynamically allocated to load the weights of the weight tensor 200. The weights can be loaded into the memory according to the access mode implemented by the network construction source code. The access mode can be iterated on the fastest changing dimension of the weight tensor 200, resulting in frequent allocation, recovery and writing of the memory to load the requested weights. In this example, the fastest changing dimension of the weight tensor 200 can be the depth, and the network construction source code can implement an access mode that iterates on the depth of the weight tensor 200. As a result, the memory may have to repeatedly load weights from channels 202a, 202b, 202c for various accesses to weights at different height and width positions in different channels. Repeated dynamic allocation, recovery and writing of memory are costly in terms of memory resource consumption (such as space, bandwidth and electrical power).

[0049] By using a modified version of the network construction source code, a weight layout transformer, the execution of the trained machine learning model can achieve a more efficient memory access pattern. The weight layout transformer can be configured to implement an access pattern in which weights are loaded into memory for iterating over a slower (such as the slowest) changing dimension of the weight tensor 200. In this example, the slowest changing dimension of the weight tensor 200 can be a row, and the weight layout transformer can implement an access pattern that iterates over the rows of the weight tensor 200. An access pattern that iterates over a slower changing dimension of the weight tensor 200 can improve data locality in memory because the access pattern is configured to use more data (such as weights) loaded into memory per load.

[0050] Figure 2B An example of weights of a weight tensor 200 accessed from memory using an access pattern of a weight layout transformer is illustrated. For example, the access pattern may transform a multidimensional weight tensor 200 into a linear organization 204 of weights. The access pattern may include transforming the weight tensor 200 into a linear format 204 in memory for use during execution of a trained neural network. In some embodiments, the linear format 204 may be a linear array. In some embodiments, the linear format 204 may be a linked list, a stack, a queue, etc. The linear format 204 of weights may organize the weights in a manner that improves the locality of the weights in memory to use memory resources more efficiently. Organizing the weights in a linear format 204 may allow the memory access pattern to be a sequential memory access pattern, which may reduce the number of times the memory is written and read to retrieve the weights. For example, the memory access pattern may allow each weight to be accessed sequentially, such as using a stride 1 reference pattern (such as a major row layout) of a linear layout 204 of a multidimensional weight tensor 200.

[0051] Figure 2BThe example illustrated in FIG. 2 shows an embodiment of a potential linear layout 204 of a weight tensor 200 as a major row layout, where weights may be organized in the following manner: weights are organized per position in a row of each channel 202a, 202b, 202c. For example, the linear layout 204 may group together each row 212, 214, 216 of weights, where similar shading patterns represent weights from the same row in each channel 202a, 202b, 202c of the weight tensor 200. More specifically, the linear layout 204 may group together a first row 212 of weights, a second row 214 of weights, and a third row 216 of weights. In each weight group by row 212, 214, 216, the linear layout 204 can further group the weights by channel 206a, 206b, 206c, 208a, 208b, 208c, 210a, 210b, 210c, by position in each row, where similar shading levels represent weights from the same position in the row. More specifically, the linear layout 204 can group the weights of the first position of the first row of each channel 206a, the weights of the second position of the first row of each channel 206b, and the weights of the third position of the first row of each channel 206c. A similar pattern can be implemented for grouping the following: the weights of the first position of the second row of each channel 208a and the third row of each channel 210a, the weights of the second position of the second row of each channel 208b and the third row of each channel 210b, and the weights of the third position of the second row of each channel 208c and the third row of each channel 210c.

[0052] When a linear layout 204 (such as a major row layout) is used for weights, the last index may be the fastest changing. The memory location of a weight may be calculated from its index as follows:

[0053]

[0054] Where "N" is the linear layout dimension, "n" is the index used to access a specific element of the linear layout 204, and "d" is the feature map size. In the continued example, when a three-dimensional feature map is used, d=3, and the last dimension (depth or channel) may change the fastest and the first dimension (height or row) may change the slowest. The offset for a given height may be:

[0055] n3+N3*(n2+N2*n1)

[0056] The dimensions, sizes, and organization of the weight tensor 200 and the linear layout 204 in the foregoing examples are used for clarity and ease of explanation, and do not limit the scope of the claims and the specification. It is contemplated that various embodiments may use weight tensors of different dimensions, sizes, and / or organizations, and corresponding linear layouts of different sizes and / or organizations.

[0057] Figure 3 An example of a linear weight layout using a weight tensor is illustrated performed by training a machine learning model 300 according to various embodiments. Figure 1-3 , the trained machine learning model 300 can be processed by an edge processor (e.g., Figure 1 The edge processor(s) 124 and other edge processors described herein are implemented using modified network construction source code (such as a weight layout transformer) and a weight corrector to generate inferences. The trained machine learning model 300 may include any type of machine learning model with any number and combination of layers. For clarity and ease of explanation, the following examples are explained using a convolutional machine learning model and do not limit the scope of the claims and description.

[0058] Each layer 306a, 306b, 306c, 306d, 306e of the trained machine learning model 300 can be loaded from the dynamic buffer 302a, 302b, 302c, 302d, 302e, 302f ( Figure 3 The dynamic buffers 302a, 302b, 302c, 302d, 302e, 302f, 302g may be any number and combination of memories (e.g., Figure 1 The dynamic buffers 302a, 302b, 302c, 302d, 302e, 302f, 302g may be used to load and store feature maps because the data in these feature maps may change between implementations of the trained machine learning model 300. For example, an input feature map may be loaded into a dynamic buffer 302a, which is configured to provide the input feature map to the first layer (convolutional layer) 306a of the trained machine learning model 300. The data of the input feature map may affect the feature maps generated by each layer of 306a, 306b, 306c, 306d, 306e of the trained machine learning model 300 and loaded into each successive dynamic buffer 302b, 302c, 302d, 302e, 302f, 302g. Different input feature maps may be loaded into the dynamic buffer 302a, and may result in different feature maps being generated by each layer in 306a, 306b, 306c, 306d, 306e of the trained machine learning model 300 and loaded into each successive dynamic buffer 302b, 302c, 302d, 302e, 302f, 302g. The individual feature maps may differ in content and size.

[0059] At least some layers (convolutional layers) 306a, 306c of the trained machine learning model 300 may also be loaded from the static buffers 304a, 304b ( Figure 3 The static buffers 304a, 304b may be any number and combination of memories (e.g., Figure 1 The static buffers 304a, 304b may be used to load and store weights because these weights may be static for multiple implementations of the trained machine learning model 300. For example, regardless of the feature maps loaded into the dynamic buffers 302a, 302b, 302c, 302d, 302e, 302f, the weights for the corresponding layers 306a, 306c of the trained machine learning model 300 loaded into the static buffers 304a, 304b may not change with the implementation of the trained machine learning model 300. These weights may be constant in content and size.

[0060] Regardless of the static properties of the weights loaded into the static buffers 304a, 304b, the static buffers 304a, 304b may in some cases not be large enough to load the weights from the weight tensor (e.g., Figure 2A 200 in the weight tensor 200). The weights may be loaded into the static buffers 304a, 304b in a manner that reflects the access pattern of the static buffers 304a, 304b for implementing the corresponding layers 306a, 306c of the trained machine learning model 300. The access pattern for implementing the corresponding layers 306a, 306c may be controlled by the network construction source code for implementing the corresponding layers 306a, 306c. Depending on the access pattern for implementing the corresponding layers 306a, 306c, weights may be loaded, but not all loaded weights may be accessed for the first access to the static buffers 304a, 304b; these weights may be recycled when different weights are accessed for the second access to the static buffers 304a, 304b; and for the third access to the static buffers 304a, 304b, these weights may be reloaded and accessed. Each access to the static buffers 304a, 304b may correspond to a loop that executes the network construction source code. The more loops that are executed, the more accesses are required to retrieve the weights from the static buffers 304a, 304b. Loading weights when not required for the first access to the static buffers 304a, 304b and reloading weights when required for the third access to the static buffers 304a, 304b consumes unnecessary memory resources, such as space, bandwidth, and electrical power.

[0061] In some embodiments, rather than implementing the inefficient network construction source code to implement the corresponding layers 306a, 306c, the edge processing device may implement a weight layout transformer and / or a weight corrector. The weight layout transformer may be a modified version of the network construction source code that is configured to retrieve the same weights from the static buffers 304a, 304b for implementing the corresponding layers 306a, 306c using an access mode of the static buffers 304a, 304b that is different from the access mode of the network construction source code.

[0062] The access pattern of the weight layout transformer may prompt the static buffers 304a, 304b to load weights in an ordered manner so that more loaded weights (up to all loaded weights) are available for access to the static buffers 304a, 304b. For example, the access pattern of the static buffers 304a, 304b of the weight layout transformer may prompt the static buffers 304a, 304b to be arranged in a linear manner (e.g., Figure 2B The linear layout 204 in the weight tensor is used to load the weights. In a further example, the linear layout can be a linear array. The linear format can group weights from the same position in the weight tensor across the slowest changing dimension of the weight tensor. For example, the linear format can group weights from the same position in the row of the weight tensor for each depth and column of the weight tensor.

[0063] As the weight layout transformer iterates through the loops for executing the corresponding layers 306a, 306c, the access pattern can be sequentially iterated along the linear format, thereby successively accessing each weight loaded into the static buffers 304a, 304b. Accessing all weights when loading into the static buffers 304a, 304b through successive loops can eliminate the need to reload weights for successive loops, thereby saving memory resources that would otherwise be used for additional memory loading and recycling using the access pattern of the network construction source code. In addition, the access pattern for retrieving weights can be synchronized with the pattern for performing the calculation of the corresponding layers 306a, 306c using weights and feature map inputs. This synchronization can result in fewer memory accesses to retrieve weights for calculation. Fewer memory accesses may require fewer loops, fewer iterations, and / or fewer groupings to achieve weight retrieval of weight tensors from the static buffers 304a, 304b to execute the corresponding layers 306a, 306c.

[0064] Figure 4 An example system for implementing source code of a trained machine learning model configured for weight tensor transformation according to various embodiments is illustrated. Figure 1-4 , the weight layout transformation system 400 may be implemented by a processor (e.g., Figure 1 The processor 104 and the edge processor 124 are implemented.

[0065] In the weight layout transformation system 400, the network construction source code 402 - a high-level programming language version of the trained machine learning model - can be converted into a weight layout transformer 404. In some embodiments, the network construction source code 402 can be manually converted into the weight layout transformer 404 by a developer. In some embodiments, the processor can be configured to match and select a template 406 for converting the network construction source code 402 into the weight layout transformer 404. The processor can use the template 406 to modify the network construction source code 402 to generate the weight layout transformer 404. In some embodiments, the template 406 can be pre-configured and stored in a memory (e.g., Figure 1 The matching and / or selection of templates 406 can be based on an analysis of the network construction source code 402 (including metadata and / or source code) to identify the type of trained machine learning model and / or the type of trained machine learning model layer and identify the template 406 for the trained machine learning model type and / or layer type. The weight layout transformer 404 can be configured to use a more efficient memory access mode from the memory (e.g., Figure 1 The weights of the trained machine learning model are retrieved from the memory 106 in the network to implement the same trained machine learning model as the network construction source code 402.

[0066] A more efficient memory access pattern may retrieve weights in a transformed order that is unintended or incompatible with the execution of computations of a trained machine learning model. The weight corrector 408 may be configured to correct the order in which weights are retrieved from memory to be used for computations of a trained machine learning model. For example, the weight corrector 408 may correct the transformed order of weights retrieved from memory by the weight layout transformer 404 to the order in which the network construction source code 402 will retrieve the weights. The weight corrector 408 may be generated and / or selected (e.g., by a processor) for the weight layout transformer 404. In some embodiments, the weight corrector 408 may be manually generated by a developer. In some embodiments, the weight corrector 408 may be automatically generated by a processor analyzing the memory access patterns of the network construction source code 402 and the weight layout transformer 404. In some embodiments, the weight corrector 408 may be pre-configured and stored in a memory (e.g., Figure 1 The processor may be configured to select a weight corrector 408 based on analyzing the network construction source code 402 (including metadata and / or source code) to identify a trained machine learning model type and / or a trained machine learning model layer type and identifying a weight corrector 408 for the trained machine learning model type and / or layer type.

[0067] Software and / or firmware developers (who may also be hardware 412 (e.g., Figure 1 A hardware developer of the edge processor 124 in the embodiment of the present invention can develop software and / or firmware for execution by the hardware 412 using a hardware compatible tool chain 409 (also referred to herein as an existing dedicated firmware tool chain). The software and / or firmware can be a compilable network integration software and / or firmware 410 that incorporates the weight layout transformer 404 and the weight corrector 408 to implement the trained machine learning model by the hardware 412.

[0068] In some embodiments, the network integration software and / or firmware 410 may be compiled in an executable format and provided to the hardware 412. In some embodiments, the network integration software and / or firmware 410 may be provided to the hardware 412. The hardware 412 may compile the network integration software and / or firmware 410 into an executable format. The hardware 412 may execute the compiled network integration software and / or firmware 410. Executing the compiled network integration software and / or firmware 410 may cause the hardware 412 to implement a trained machine learning model.

[0069] Figure 5 Example computer code for source code of a trained machine learning model configured for weight tensor transformation according to various embodiments is illustrated. Figure 1-5 , code block 500 may correspond to network construction source code for a trained machine learning model and / or a trained machine learning model layer (e.g., Figure 4 The code block 502 may correspond to a weight layout transformer for the network construction source code (e.g., Figure 4 4. The code block 504 may correspond to a weight corrector for the weight layout transformer (e.g., Figure 4 For convenience, code block 500, code block 502, and code block 504 may sometimes be referred to as first code block 500, second code block 502, and third code block 504, respectively.

[0070] like Figure 5 As shown in the example illustrated in , code block 500 can use nested loops to perform weight retrieval for a trained machine learning model and / or a trained machine learning model layer. The nested loops can control the iterative execution of weight retrieval from a weight tensor (e.g., weight tensor 200 in FIG. 2 ) so as to execute the trained machine learning model and / or the trained machine learning model layer using nested loops. The lowest level loop in code block 500 iterates over the fastest changing dimension of the weight tensor.

[0071] Continuing with the previous example, the lowest level loop is over the depth or channels of the weight tensor (e.g. Figure 2B However, the memory (e.g., Figure 1 The access pattern for retrieving weights using nested loops in memory 106) may not correspond to an efficient access pattern for retrieving weights required to implement computations of a trained machine learning model and / or a trained machine learning model layer using weights and feature map inputs. In other words, the access pattern for retrieving weights may be asynchronous with the pattern for performing computations of a trained machine learning model and / or a trained machine learning model layer using weights and feature map inputs. This asynchrony may result in excess memory accesses to retrieve weights for computations. Figure 5 In the example illustrated in , code block 500 accesses memory to retrieve weights from the weight tensor at locations corresponding to ordered counter values ​​of nested loops. For example, "W[filter_y][fitler_x][c1][c2]" may retrieve the weight value "[filter_y][fitler_x][c1][c2]" of the weight tensor location corresponding to the ordered counter value from memory.

[0072] Code block 502 may be a modification of code block 500. Code block 502 may use nested loops to perform weight retrieval for a trained machine learning model and / or a trained machine learning model layer. However, the lowest level loop of code block 500 may be modified in code block 502. The lowest level loop in code block 502 may iterate over a slower (such as the slowest) changing dimension of the weight tensor.

[0073] Continuing with the previous example, the lowest level loop may iterate over the height or rows of the weight tensor. The memory access pattern of retrieving weights using nested loops may correspond to a more efficient access pattern for retrieving weights needed to implement computations of a trained machine learning model and / or a trained machine learning model layer using weights and feature map inputs. For example, the access pattern may result in the weights being laid out as a linear layout (e.g., Figure 2B The linear layout 204 in the weight tensor is loaded into memory. In a further example, the linear layout can be a linear array. The linear format can group weights from the same position in the weight tensor across the slowest changing dimension of the weight tensor. For example, the linear format can group weights from the same position in the row for each channel and column of the weight tensor.

[0074] As the weight layout transformer iterates through the loops for executing the trained machine learning model and / or the trained machine learning model layer, the access pattern can be iterated sequentially along a linear format, thereby accessing each weight loaded into the memory in sequence. Accessing all weights as they are loaded into the memory through successive loops can eliminate the need to reload the weights for the successive loops, thereby saving memory resources that would otherwise be used for additional memory loading and recycling using the access pattern of the network construction source code. In addition, the access pattern for retrieving weights can be synchronized with the pattern for performing calculations of the trained machine learning model and / or the trained machine learning model layer using weights and feature map inputs. This synchronization can result in fewer memory accesses to retrieve weights for calculations. Fewer memory accesses may require fewer loops, fewer iterations, and / or fewer groupings to achieve weight retrieval of weight tensors from memory to execute the trained machine learning model and / or the trained machine learning model layer.

[0075] exist Figure 5 In the example illustrated in , code block 502 may access memory to retrieve weights from the weight tensor at locations corresponding to ordered counter values ​​of the nested loops. For example, "W[filter_y][fitler_x][c2][c1]" may retrieve the weight value "[filter_y][fitler_x][c2][c1]" from memory for the weight tensor location corresponding to the ordered counter value. In this example, the counter values ​​[c1] and [c2] are transposed relative to code block 500, which changes the order in which code block 502 retrieves the weights from memory compared to the order in which code block 500 retrieves the weights from memory (referred to herein as the transformed order).

[0076] The more efficient access pattern for retrieving weights from memory implemented by code block 502 retrieves the weights in a transformed order that is different from the order in which computations for a machine learning model and / or a trained machine learning model layer may be expected to be performed. Using the weights in the order retrieved by code block 502 may produce incorrect computational results. Code block 504 may be configured to correct the order of weights retrieved from memory by code block 502 to provide weights for performing computations for a machine learning model and / or a trained machine learning model layer. For example, code block 504 reorganizes the order of weights retrieved from memory by code block 502 to match the order of the weights as if the weights were retrieved from memory by code block 500.

[0077] Continuing with the above example, code block 504 may reorder the weights retrieved from memory by undoing the transposition of the order of the counter values ​​used to retrieve the weights from memory by code block 502 so that the order of the counter values ​​corresponds to the order of the counter values ​​in block 500. Figure 5 In the example illustrated in , “WT[filter_y][filter_x][c2][c1]=W[filter_y][filter_x][c1][c2]” may reorder the weights retrieved from memory using the order “[filter_y][fitler_x][c2][c1]” as in block 502 to the order “[filter_y][fitler_x][c1][c2]” as in block 500 .

[0078] Figure 6 A method 600 for generating source code executed by a trained machine learning model using weight tensor transformations is illustrated in accordance with some embodiments. Figure 1-6 , the method 600 may be implemented in the following: a computing device (eg, Figure 1 computing device 100 in the example), general purpose hardware (e.g., Figure 1 processor 104 in ), dedicated hardware (e.g., see Figure 1 edge processor(s) 124 and other edge processors described herein), software executed in the processor (e.g., Figure 4 In and refer to Figure 2A-5 The method 600 may be implemented in a manner that is consistent with the weight layout transformation system 400, the weight layout transformer 404, the template 406, the weight corrector 408 described herein), or a combination of a processor configured by software and dedicated hardware. To cover alternative configurations implemented in various embodiments, the hardware implementing the method 600 is referred to herein as a "processing device."

[0079] In block 602, a processing device may receive network construction source code (e.g., Figure 4 In and refer to Figure 2A-5 4. In some embodiments, the processing device receiving the network construction source code in block 602 may be one or more general purpose processors. In some embodiments, the processing device receiving the network construction source code in block 602 may be one or more edge processing devices.

[0080] In box 604, the processing device may analyze the network construction source code received in box 602. In some embodiments, the processing device may be configured to read the metadata of the network construction source code and identify the type of the trained machine learning model and / or the trained machine learning model layer of the network construction source code. In some embodiments, the processing device may be configured to parse the network construction source code to locate and identify the trained machine learning model layer. The processing device may be configured to parse the network construction source code to locate and identify the network layer type, network layer execution, network layer flow control, and / or the memory access command for weight retrieval of the network construction source code. The processing device may be configured to locate and identify the code that matches the criteria for the format of the following items: network layer type, network layer execution, network layer flow control, and / or the memory access command for weight retrieval of the network construction source code. For example, the network layer type, network layer execution, network layer flow control, and / or the memory access command for weight retrieval of the network construction source code can be marked with a specific function call, a specific code pattern (such as a loop), a specific identifier, etc. In some embodiments, different criteria may be used by a processing device to parse the network construction source code to locate and identify network layer types of different trained machine learning models, network layer flow control, and / or memory access commands for weight retrieval of the network construction source code. In some embodiments, the processing device that analyzes the network construction source code in block 604 may be one or more general purpose processors. In some embodiments, the processing device that analyzes the network construction source code in block 604 may be one or more edge processing devices.

[0081] In optional block 606, the processing device may select a template for a weight layout transformer. The processing device may identify metadata and / or content of the network construction source code that satisfies criteria for identifying a weight layout transformer. The processing device may compare the metadata and / or content of the network construction source code with the criteria for identifying a weight layout transformer and identify the weight layout transformer from the metadata and / or content that satisfies the criteria. The processing device may be configured to select a template for the weight layout transformer based on an identified network layer type, network layer execution, network layer flow control, and / or memory access commands for weight retrieval of the network construction source code (e.g., from Figure 1106, 114 in the memory). Each template for the weight layout transformer may correspond to a type of network and / or network layer. The template for the weight layout transformer may be preconfigured to correspond to any network and / or network layer type. The template for the weight layout transformer may be configured to provide source code for execution, flow control, and memory access commands for weight retrieval for network and / or network layer types in a programming language that can be compiled and executed by the edge processing device. For example, the template for the weight layout transformer may include specific function calls configured to implement network and / or network layer execution and flow control, specific code patterns (such as loops), specific identifiers, specific memory access commands, etc.

[0082] In some embodiments, different templates for the weight layout transformer may include different codes for implementing network and / or network layer execution, flow control, and memory access commands for weight retrieval for different networks and / or layers of different trained machine learning models. In some embodiments, the processing device that selects the template for the weight layout transformer in box 606 may be one or more general purpose processors. In some embodiments, the processing device that selects the template for the weight layout transformer in box 606 may be one or more edge processing devices.

[0083] In box 608, the processing device may generate and / or select a weight layout transformer. In some embodiments, the processing device may read a selected template for the weight layout transformer. The processing device may be configured to generate weight layout transformer code for use when executing a trained machine learning model and / or a trained machine learning model layer using a selected layer template. In some embodiments, the processing device may read the code of the selected template for the weight layout transformer. Reading the selected template for the weight layout transformer may provide the processing device with source code for initialization, execution, flow control, and / or memory access commands for the trained machine learning model and / or the trained machine learning model layer. In some embodiments, the processing device may write out the code of the selected template for the weight layout transformer to a memory (e.g., Figure 1 In some embodiments, the processing device may be configured to read the data from the memory (e.g., Figure 1 The network construction source code is modified by modifying and / or replacing a memory access command for retrieving weights from a memory specified in a template for a weight layout transformer (the memory 106 in the example). For example, a processing device may change and / or replace a memory access command for retrieving weights in a lowest level loop of a nested loop. In some embodiments, the processing device that generates the weight layout transformer in box 608 may be one or more general purpose processors. In some embodiments, the processing device that generates the weight layout transformer in box 608 may be one or more edge processing devices.

[0084] In some embodiments, the weight layout transformer may be preconfigured and stored in the memory of the computing device, and the processing device may select the weight layout transformer to be used instead of the network construction source code based on the analysis of the network construction source code in box 604. The processing device may identify the metadata and / or content of the network construction source code that meets the criteria for identifying the weight layout transformer. The processing device may compare the metadata and / or content of the network construction source code with the criteria for identifying the weight layout transformer, and identify the weight layout transformer from the metadata and / or content that meet the criteria. The processing device may be configured to select the weight layout transformer based on the identified network layer type, network layer execution, network layer flow control, and / or the memory access command for weight retrieval of the network construction source code. Each weight layout transformer may correspond to the type of network and / or network layer. The weight layout transformer may be preconfigured to correspond to any network and / or network layer type. In some embodiments, the processing device that selects the weight layout transformer in box 608 may be one or more general-purpose processors. In some embodiments, the processing device that selects the weight layout transformer in box 608 may be one or more edge processing devices.

[0085] In optional box 610, the processing device may analyze the weight layout transformer. In some embodiments, the processing device may be configured to read the metadata of the weight layout transformer and identify the type of the trained machine learning model and / or the trained machine learning model layer of the weight layout transformer. The processing device may identify the difference between the weight layout transformer and the network construction source code. For example, compared with the network construction source code, the processing device may identify different memory access commands for retrieving weights from the memory in the weight layout transformer. In some embodiments, the processing device may be configured to parse the weight layout transformer to locate and identify the memory access command for retrieving weights. The processing device may be configured to locate and identify the code that matches the criteria for the memory access command for retrieving weights. In some embodiments, different criteria may be used by the processing device to parse the weight layout transformer to locate and identify the memory access command for weight retrieval of the weight layout transformer for different trained machine learning models. In some embodiments, the processing device that analyzes the weight layout transformer in box 610 may be one or more general-purpose processors. In some embodiments, the processing device that analyzes the weight layout transformer in box 610 may be one or more edge processing devices.

[0086] In box 612, the processing device may generate and / or select a weight corrector. In some embodiments, the processing device may be configured to generate a weight layout corrector for use when executing a trained machine learning model and / or a trained machine learning model layer based on an analysis of the network construction source code (in box 604) and / or the weight layout transformer (optionally in box 610). In some embodiments, the processing device may read the code of the network construction source code and / or the weight layout transformer. Reading the network construction source code and / or the weight layout transformer may provide the processing device with source code for memory access commands for retrieving weights. In some embodiments, the processing device may generate code for a weight corrector that returns the order in which weights are retrieved from memory by the weight layout transformer (referred to herein as the transformed order) to the order in which weights are retrieved from memory by the network construction source code, and store the code of the weight corrector to a memory (e.g., Figure 1 In some embodiments, the processing device that generates the weight corrector in block 612 may be one or more general purpose processors. In some embodiments, the processing device that generates the weight corrector in block 612 may be one or more edge processing devices.

[0087] In some embodiments, the weight layout corrector may be preconfigured and stored in the memory of the computing device. The processing device may select the weight layout corrector to be used based on the analysis of the network construction source code (in box 604) and / or the weight layout converter (optionally in box 610). The processing device may identify the metadata and / or content of the network construction source code and / or the weight layout converter that meet the criteria for identifying the weight corrector. The processing device may compare the metadata and / or content of the network construction source code and / or the weight layout converter with the criteria for identifying the weight corrector, and identify the weight layout converter from the metadata and / or content that meet the criteria. The processing device may be configured to select the weight corrector based on the identified network layer type, network layer execution, network layer flow control, and / or the memory access command for the weight retrieval of the network construction source code and / or the weight layout converter. Each weight corrector may correspond to the type of network and / or network layer. The weight corrector may be preconfigured to correspond to any network and / or network layer type. In some embodiments, the processing device that selects the weight corrector in box 612 may be one or more general-purpose processors. In some embodiments, the processing device that selects the weight corrector in block 612 may be one or more edge processing devices.

[0088] In some embodiments, any or all of blocks 602, 604, 606, 608, 610, 612 may be implemented for each network layer of a trained machine learning model.

[0089] Fig. 7AA method 700 for weight layout transformation according to some embodiments is illustrated. Figure 1-7A , the method 700 may be implemented in the following: a computing device (eg, Figure 1 computing device 100 in the example), general purpose hardware (e.g., Figure 1 processor 104 in ), dedicated hardware (e.g., see Figure 1 edge processor(s) 124 and other edge processors described herein), software executed in the processor (e.g., Figure 4 In and refer to Figure 2A-5 The weight layout transformation system 400, weight layout transformer 404 described herein), or a combination of a processor configured by software and dedicated hardware. To cover alternative configurations implemented in various embodiments, the hardware implementing method 700 is referred to herein as a "processing device."

[0090] In block 702, a processing device may execute a nested loop of a weight layout translator. As described herein, a weight corrector may include a nested loop configured to cause a weight layout translator to traverse a weight tensor (e.g., weight tensor 200 in FIG. 2 ) and retrieve weights of the weight tensor. The weight layout translator retrieves the weights of the weight tensor from a memory (e.g., Figure 1 The order in which the weights of the weight tensor are retrieved from the memory 106 in the network (referred to herein as the transformed order) may be different from the network construction source code (e.g., Figure 4 In and refer to Figure 2A-5 The network construction source code 402 described in the embodiment of the present invention) retrieves the weights of the weight tensor from the memory in an order. In some embodiments, the processing device that executes the nested loops of the weight layout translator in box 702 can be one or more general-purpose processors. In some embodiments, the processing device that executes the nested loops of the weight layout translator in box 702 can be one or more edge processing devices.

[0091] In box 704, the processing device may access the memory to retrieve the weights of the weight tensor in a transformed order that may be different from the order in which the network construction source will retrieve the weights of the weight tensor. At certain levels of the nested loops executed in box 702, the weight layout transformer may include a memory access command configured to retrieve the weights of the weight tensor. In some embodiments, the nested loop at this level at which the weight layout transformer may include a memory access command configured to retrieve the weights of the weight tensor may be the lowest level nested loop. The memory access command may include variables (such as counter values) that may specify the position in the weight tensor from which the weights are to be retrieved. As the nested loops iterate, the values ​​of these variables of the memory access command may change, thereby changing the position in the weight tensor from which the weights are to be retrieved. In some embodiments, the variable order of the memory access command of the weight layout transformer may be different from the variable order of the memory access command of the network construction source code. For example, the variable order of the memory access commands of the network construction source code may iterate over a fast changing dimension of the weight tensor, such as depth or channel (e.g., channels 202a, 202b, 202c in FIG. 2), while the variable order of the memory access commands of the weight layout transformer may iterate over a slower changing dimension of the weight tensor. In some embodiments, the processing device that accesses the memory in box 704 to retrieve the weights of the weight tensor in a transformed order that may be different from the order in which the network construction source would retrieve the weights of the weight tensor may be one or more general purpose processors. In some embodiments, the processing device that accesses the memory in box 704 to retrieve the weights of the weight tensor in a transformed order that may be different from the order in which the network construction source would retrieve the weights of the weight tensor may be one or more edge processing devices.

[0092] In block 706, the processing device may load the weights of the weight tensor in a transformed order that may be different from the order in which the weights of the weight tensor are loaded into the memory in response to the memory access request of the network construction source code in response to the memory access request of the weight layout transformer. The processing device may load the weights retrieved from the weight tensor into the memory. Based on the transformed order of the weight retrieval specified by the memory access command in block 704, the processing device may similarly load the weights into the memory in the transformed order. In some embodiments, the processing device that loads the weights of the weight tensor in a transformed order that may be different from the order in which the weights of the weight tensor are loaded into the memory in response to the memory access request of the network construction source code in block 706 in response to the memory access request of the weight layout transformer may be one or more general-purpose processors. In some embodiments, the processing device that loads the weights of the weight tensor in a transformed order that may be different from the order in which the weights of the weight tensor are loaded into the memory in response to the memory access request of the network construction source code in block 706 in response to the memory access request of the weight layout transformer may be one or more edge processing devices.

[0093] In box 708, the processing device may retrieve the weights of the weight tensor from the memory in response to the memory access request of the weight layout transformer in a transformed order that may be different from the order in which the weights of the weight tensor are retrieved from the memory in response to the memory access request of the network construction source code. The processing device may retrieve the weights of the weight tensor loaded into the memory. Based on the transformed order of the weight retrieval specified by the memory access command in box 704, the processing device may similarly retrieve the weights from the memory in the transformed order. In some embodiments, the processing device that retrieves the weights of the weight tensor from the memory in response to the memory access request of the weight layout transformer in box 708 in a transformed order that may be different from the order in which the weights of the weight tensor are retrieved from the memory in response to the memory access request of the network construction source code may be one or more general-purpose processors. In some embodiments, the processing device that retrieves the weights of the weight tensor from the memory in response to the memory access request of the weight layout transformer in box 708 in a transformed order that may be different from the order used to retrieve the weights of the weight tensor from the memory in response to the memory access request of the network construction source code may be one or more edge processing devices.

[0094] In some embodiments, any or all of blocks 702, 704, 706, 708 may be implemented for each layer of the trained machine learning model. In some embodiments, any or all of blocks 702, 704, 706, 708 may be implemented serially and / or in parallel. In some embodiments, any or all of blocks 702, 704, 706, 708 may be implemented repeatedly and / or continuously. For example, blocks 702, 704, 706, 708 may be implemented repeatedly and / or continuously for all iterations of a nested loop of a weight layout transformer.

[0095] Figure 7B A method 710 for weight layout transformation according to some embodiments is illustrated. Figure 1-7B , the method 710 may be implemented in the following: a computing device (eg, Figure 1 computing device 100 in the example), general purpose hardware (e.g., Figure 1 processor 104 in ), dedicated hardware (e.g., see Figure 1 edge processor(s) 124 and other edge processors described herein), software executed in the processor (e.g., Figure 4 In and refer to Figure 2A-5 The weight layout transformation system 400, weight layout transformer 404 described herein), or a combination of a processor configured by software and dedicated hardware. To cover alternative configurations implemented in various embodiments, the hardware implementing method 710 is referred to herein as a "processing device."

[0096] In block 712, the processing device may access a first memory (eg, Figure 1 2 ) to retrieve the weights of a weight tensor (e.g., weight tensor 200 in FIG. 2 ) in a transformed order different from the order used to retrieve the weights for calculations at the network layer of the trained machine learning model. Block 712 may be performed similarly to that described with reference to Fig. 7A The operations in block 704 of method 700 are described herein.

[0097] In block 714, the processing device may load the weights into a second memory (e.g., Figure 1 106). Block 714 may be executed similarly to that described with reference to Fig. 7A The operations in block 706 of the described method 700 are implemented in this manner. In some embodiments, the first memory and the second memory may be within the same memory device, such as different partitions within the same memory device.

[0098] Figure 8 A method 800 for weight layout transformation according to some embodiments is illustrated. Figure 1-8, the method 800 may be implemented in the following: a computing device (eg, Figure 1 computing device 100 in the example), general purpose hardware (e.g., Figure 1 processor 104 in ), dedicated hardware (e.g., see Figure 1 edge processor(s) 124 and other edge processors described herein), software executed in the processor (e.g., Figure 4 In and refer to Figure 2A-5 7 ), or a combination of a processor configured with software and dedicated hardware. To cover alternative configurations implemented in various embodiments, the hardware implementing method 800 is referred to herein as a "processing device." In some embodiments, method 800 may further describe method 700 described herein with reference to FIG. 7 .

[0099] In block 802, a processing device may execute a nested loop of a weight layout translator. As described herein, a weight corrector may include a nested loop configured to cause a weight layout translator to traverse a weight tensor (e.g., weight tensor 200 in FIG. 2 ) and retrieve weights of the weight tensor. The weight layout translator retrieves the weights of the weight tensor from a memory (e.g., Figure 1 The order in which the weights of the weight tensor are retrieved from the memory 106 in the network (referred to herein as the transformed order) may be different from the network construction source code (e.g., Figure 4 In and refer to Figure 2A-5 In some embodiments, the processing device that executes the nested loop of the weight layout translator in box 802 may be one or more general-purpose processors. In some embodiments, the processing device that executes the nested loop of the weight layout translator in box 802 may be one or more edge processing devices. In some embodiments, box 802 may further describe box 702 of method 700 described herein with reference to FIG. 7.

[0100] In block 804, the processing device may access the memory to retrieve the weights of the weight tensor iterated on the slowest changing dimension of the weight tensor. At certain levels of the nested loops executed in block 802, the weight layout transformer may include a memory access command configured to retrieve the weights of the weight tensor. In some embodiments, the nested loop at this level at which the weight layout transformer may include a memory access command configured to retrieve the weights of the weight tensor may be the lowest level nested loop. The memory access command may include variables (such as counter values) that may specify the position in the weight tensor from which the weight is to be retrieved. As the nested loops iterate, the values ​​of these variables of the memory access command may change, thereby changing the position in the weight tensor from which the weight is to be retrieved. In some embodiments, the variable order of the memory access command of the weight layout transformer may be iterated on the slowest changing dimension of the weight tensor. In some embodiments, the slowest changing dimension of the weight tensor may be a height or a row. In some embodiments, the variable order of the memory access command of the weight layout transformer for specifying the position in the weight tensor may be transposed relative to the variable order of the memory access command of the network construction source code for specifying the position in the weight tensor. In some embodiments, the processing device that accesses the memory in block 804 to retrieve the weights of the weight tensor iterated over the slowest varying dimension of the weight tensor may be one or more general purpose processors. In some embodiments, the processing device that accesses the memory in block 804 to retrieve the weights of the weight tensor iterated over the slowest varying dimension of the weight tensor may be one or more edge processing devices.

[0101] In box 806, the processing device may load the weights of the weight tensor into the memory in a linear layout according to the slowest changing dimension of the weight tensor. The processing device may load the weights retrieved from the weight tensor into the memory. Based on the different order of weight retrieval specified by the memory access command in box 804, the processing device may similarly load the weights into the memory in the different order, such as in a linear layout sorted according to the slowest changing dimension of the weight tensor (e.g., linear layout 204 in Figure 2). In some embodiments, the weight layout transformer may arrange the retrieved weights in the memory in a linear array. In some embodiments, the weight layout transformer may arrange the retrieved weights in the memory in a linear data structure, such as via a linked list, a stack, a queue, etc. In some embodiments, the linear layout may be a major row layout, in which the weights may be organized at each position in the row of each channel. For example, the linear layout may arrange each row of weights (e.g., Figure 2B In each weight group by row, the linear layout can be further divided by channel, by position in each row (e.g., Figure 2BThe weights are grouped by position in each row of channels 206a, 206b, 206c, 208a, 208b, 208c, 210a, 210b, 210c). In some embodiments, the processing device that loads the weights of the weight tensor into memory in a linear layout according to the slowest changing dimension of the weight tensor in block 806 may be one or more general purpose processors. In some embodiments, the processing device that loads the weights of the weight tensor into memory in a linear layout according to the slowest changing dimension of the weight tensor in block 806 may be one or more edge processing devices.

[0102] In block 808, the processing device may retrieve the weights of the weight tensor from the memory in a sequential order of the linear layout. The processing device may retrieve the weights of the weight tensor loaded into the memory. Based on the weight retrieval order specified by the memory access command in block 804, the processing device may retrieve the weights of the weight tensor from the memory in a sequential order of the linear layout. The linear layout of the weights may organize the weights in a manner that improves the locality of the weights in the memory to use memory resources more efficiently. Organizing the weights in a linear layout may allow memory access to be sequential memory access, which may reduce the number of times the memory is written and read to retrieve the weights. For example, the memory access may allow each weight to be accessed sequentially, such as using a stride 1 reference mode of the linear layout (such as using a main row layout). In some embodiments, the processing device that retrieves the weights of the weight tensor from the memory in a sequential order of the linear layout in block 808 may be one or more general purpose processors. In some embodiments, the processing device that retrieves the weights of the weight tensor from the memory in a sequential order of the linear layout in block 808 may be one or more edge processing devices.

[0103] In some embodiments, any or all of blocks 802, 804, 806, 808 may be implemented for each layer of the trained machine learning model. In some embodiments, any or all of blocks 802, 804, 806, 808 may be implemented serially and / or in parallel. In some embodiments, any or all of blocks 802, 804, 806, 808 may be implemented repeatedly and / or continuously. For example, blocks 802, 804, 806, 808 may be implemented repeatedly and / or continuously for all iterations of a nested loop of a weight layout transformer.

[0104] Fig. 9 A method 900 for weight correction according to some embodiments is illustrated. Figure 1-9 , the method 900 may be implemented in the following: a computing device (eg, Figure 1 computing device 100 in the example), general purpose hardware (e.g., Figure 1 processor 104 in ), dedicated hardware (e.g., see Figure 1edge processor(s) 124 and other edge processors described herein), software executed in the processor (e.g., Figure 4 In and refer to Figure 2A-5 7 and 8 ), and can be implemented sequentially and / or in parallel with the methods 700 and 800 described herein.

[0105] In block 902, processing devices may be arranged in a linear layout (e.g., Figure 2B 204) from memory (e.g., Figure 1 In some embodiments, the processing device may receive the weights retrieved from the memory in block 708 of the method 700 as described with reference to FIG. 7. In some embodiments, the processing device may receive the weights retrieved from the memory in block 708 of the method 700 as described with reference to FIG. Figure 8 The weights retrieved from the memory in block 808 of the described method 800. In some embodiments, the processing device that reads the weight data in block 902 may be one or more general purpose processors. In some embodiments, the processing device that reads the weight data in block 902 may be one or more edge processing devices.

[0106] In block 904, the processing device may reorder the weights into an order for performing calculations at the trained machine learning model layer. As discussed herein, a weight layout transformer (e.g., Figure 4 The weight layout transformer 404 in the weight layout transformer 404) transforms the weight tensor (e.g., Figure 2A The transformed order in which the weights of the weight tensor 200 in 200 are retrieved may be different from the network construction source code (e.g., as referenced Figure 3-5The described network construction source code 304) will retrieve the order in which the weights from the weight tensor are retrieved. The order in which the network construction source code will retrieve the weights may be the order in which the weights are calculated at the trained machine learning model layer. The order in which the weight layout transformer retrieves the weights may be unexpected and / or incompatible with the order in which the weights are calculated at the trained machine learning model layer. The processing device may instantiate variables, such as counter values, which may specify the position of the weights in the weight tensor. When accessing the weights, the weights may be expected to correspond to the position in the weight tensor according to the variable value. For weights retrieved in an order different from the expected order (such as weights corresponding to the position in the weight tensor according to the variable value in the weight layout transformer), the order of the weights may be reordered to use the expected variable value. The processing device may set the weights received by the weight layout transformer in the first order to the second order using the expected variable value. In some embodiments, the expected order of the variables used to reorder the weights may be transposed relative to the order of the variables used for the weights retrieved by the weight layout transformer. In some embodiments, the processing device that reorders the weights into the order intended for performing the computations at the trained machine learning model layer in block 904 may be one or more general purpose processors. In some embodiments, the processing device that reorders the weights into the order intended for performing the computations at the trained machine learning model layer in block 904 may be one or more edge processing devices.

[0107] In box 906, the processing device may provide reordered weights for performing calculations at the trained machine learning model layer. The processing device may receive a request for weights in the execution of the calculations at the trained machine learning model layer and respond to the request by providing the reordered weights. In some embodiments, the request may be a request from a network to construct source code for implementing the calculations at the trained machine learning model layer. In some embodiments, the processing device that provides the reordered weights for performing calculations at the trained machine learning model layer in box 906 may be one or more general-purpose processors. In some embodiments, the processing device that provides the reordered weights for performing calculations at the trained machine learning model layer in box 906 may be one or more edge processing devices.

[0108] In box 908, the processing device may use the reordered weights to perform calculations at the trained machine learning model layer. In some embodiments, the processing device may execute the network construction source code of the trained machine learning model, and may use the reordered weights to perform calculations at the trained machine learning model layer. The network construction source code may include code for performing calculations at the trained machine learning model layer. The network construction source code may be executed as a stand-alone code and / or incorporated into software and / or firmware. These calculations may be configured to provide accurate results based on receiving weights in the order in which they are calculated at the trained machine learning model layer. The reordered weights may be configured in the order in which the calculations are performed at the trained machine learning model layer, and the calculations at the trained machine learning model layer may use the reordered weights. In some embodiments, the processing device that performs the calculations at the trained machine learning model layer using the reordered weights in box 908 may be one or more general-purpose processors. In some embodiments, the processing device that performs the calculations at the trained machine learning model layer using the reordered weights in box 908 may be one or more edge processing devices.

[0109] In some embodiments, any or all of blocks 902, 904, 906, 908 may be implemented for each layer of the trained machine learning model. In some embodiments, any or all of blocks 902, 904, 906, 908 may be implemented serially and / or in parallel. In some embodiments, any or all of blocks 902, 904, 906, 908 may be implemented repeatedly and / or continuously. For example, blocks 902, 904, 906, 908 may be implemented repeatedly and / or continuously for all weights of the trained machine learning model layer.

[0110] According to various embodiments (including but not limited to the above reference Figure 1-9 The methods of the embodiments described herein and devices for implementing such methods may be implemented in a variety of computing systems including mobile computing devices. Examples of mobile computing devices suitable for use with various embodiments are Fig.10. The mobile computing device 1000 may include a processor 1002 coupled to a touch screen controller 1004 and an internal memory 1006. The processor 1002 may be one or more multi-core integrated circuits designated for general or specific processing tasks. The internal memory 1006 may be a volatile or non-volatile memory, and may also be a secure and / or encrypted memory, or an unsecure and / or unencrypted memory, or any combination thereof. Examples of memory types that may be utilized include, but are not limited to, DDR, LPDDR, GDDR, WIDEIO, RAM, SRAM, DRAM, P-RAM, R-RAM, M-RAM, STT-RAM, and embedded DRAM. The touch screen controller 1004 and the processor 1002 may also be coupled to a touch screen panel 1012, such as a resistive sensing touch screen, a capacitive sensing touch screen, an infrared sensing touch screen, etc. Additionally, the display of the mobile computing device 1000 need not have touch screen capabilities.

[0111] The mobile computing device 1000 may have one or more radio signal transceivers 1008 (e.g., Peanut, Bluetooth, Zigbee, Wi-Fi, RF radios) coupled to each other and / or to the processor 1002 and an antenna 1010 for sending and receiving communications. The transceiver 1008 and antenna 1010 may be used with the circuit systems mentioned above to implement various wireless transmission protocol stacks and interfaces. The mobile computing device 1000 may include a cellular network wireless modem chip 1016 that implements communication via a cellular network and is coupled to the processor.

[0112] The mobile computing device 1000 may include a peripheral device connection interface 1018 coupled to the processor 1002. The peripheral device connection interface 1018 may be configured solely to accept one type of connection, or may be configured to accept various types of physical and communication connections that are shared or proprietary, such as Universal Serial Bus (USB), FireWire, Thunderbolt, or PCIe. The peripheral device connection interface 1018 may also be coupled to a similarly configured peripheral device connection port (not shown).

[0113] The mobile computing device 1000 may also include a speaker 1014 for providing audio output. The mobile computing device 1000 may also include a housing 1020 for housing all or some of the components described herein, the housing 1020 being constructed of plastic, metal, or a combination of materials. The mobile computing device 1000 may include a power source 1022, such as a disposable or rechargeable battery, coupled to the processor 1002. The rechargeable battery may also be coupled to a peripheral device connection port to receive a charging current from a source external to the mobile computing device 1000. The mobile computing device 1000 may also include a physical button 1024 for receiving user input. The mobile computing device 1000 may also include a power button 1026 for turning the mobile computing device 1000 on and off.

[0114] According to various embodiments (including but not limited to the above reference Figure 1-9 The methods and devices for implementing the methods described in the various embodiments can be implemented in a variety of computing systems, including a laptop computer 1100, an example of which is shown in FIG. Fig.11 11. The laptop computer 1100 will typically include a processor 1102 coupled to a volatile memory 1112 and a large non-volatile memory such as a compact disc (CD) drive 1113 or flash memory. In addition, the computer 1100 may have one or more antennas 1108 for sending and receiving electromagnetic radiation, which may be connected to a wireless data link, and / or a cellular telephone transceiver 1116 coupled to the processor 1102. The computer 1100 may also include a floppy disk drive 1114 and a CD drive 1113 coupled to the processor 1112. In a notebook configuration, the computer housing may include a battery 115, a touchpad touch surface 1117 used as a pointing device for the computer, a keypad 1118, and a display 1119, all coupled to the processor 1102. Other configurations of computing devices may include a computer mouse or trackball coupled to the processor (e.g., via a USB input) as is well known, which may also be used in conjunction with various embodiments.

[0115] According to various embodiments (including but not limited to the above reference Figure 1-9 The methods of the various embodiments described herein and devices for implementing such methods may also be implemented in a fixed computing system, such as any of a variety of commercially available servers. Fig.12 An example server 1200 is illustrated in FIG. Such a server 1200 typically includes one or more multi-core processor assemblies 1201 coupled to volatile memory 1202 and large-capacity non-volatile memory (such as disk drives 1204). Fig.121, a multi-core processor assembly 1201 may be added to a server 1200 by inserting the multi-core processor assembly 1201 into a chassis for the assembly. The server 1200 may also include a floppy disk drive, a compact disk (CD), or a digital versatile disk (DVD) disk drive 1206 coupled to the processor 1201. The server 1200 may also include a network access port 1203 coupled to the multi-core processor assembly 1201 for establishing a network interface connection with a network 1205, such as a local area network coupled to other broadcast system computers and servers, the Internet, a public switched telephone network, and / or a cellular data network (e.g., CDMA, TDMA, GSM, PCS, 3G, 4G, LTE, 5G, or any other type of cellular data network).

[0116] Further details regarding various embodiments are described in Appendix A of this document, which is part of the present specification disclosure as if included in the numbered paragraphs.

[0117] Computer program code or "program code" for execution on a programmable processor to perform the operations of the various embodiments may be written in a high-level programming language (such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic), a structured query language (e.g., Transact-SQL), Perl, or in a variety of other programming languages. Program code or program stored on a computer-readable storage medium as used in this application may refer to machine language code (such as object code) in a format that can be understood by a processor.

[0118] The foregoing method descriptions and process flow charts are provided only as illustrative examples and are not intended to require or imply that the operations of the various embodiments must be performed in the order given. As will be appreciated by those skilled in the art, the order of operations in the foregoing embodiments may be performed in any order. Phrases such as "thereafter," "then," "next," and the like are not intended to limit the order of operations; these phrases are simply used to guide the reader through the description of the method. In addition, any reference to a claim element in the singular form (e.g., a reference using the article "a," "a," or "the") should not be interpreted as limiting the element to the singular.

[0119] The various illustrative logic blocks, modules, circuits, and algorithmic operations described in conjunction with the various embodiments may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and operations are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. A skilled person may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the claims.

[0120] The hardware for implementing the various illustrative logics, logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device designed to perform the functions described herein, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Alternatively, some operations or methods may be performed by a circuit system dedicated to a given function.

[0121] In one or more embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, each function may be stored as one or more instructions or codes on a non-transient computer-readable medium or a non-transient processor-readable medium. The operation of the method or algorithm disclosed herein may be implemented in a processor-executable software module, which may reside on a non-transient computer-readable or processor-readable storage medium. A non-transient computer-readable or processor-readable storage medium may be any storage medium that can be accessed by a computer or processor. As an example and not limitation, such non-transient computer-readable or processor-readable media may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. As used herein, disks and discs include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and blue-ray discs, wherein disks often reproduce data magnetically and discs reproduce data optically with lasers. The above combinations are also included within the scope of non-transient computer-readable and processor-readable media. Additionally, the operations of a method or algorithm may reside as a code and / or instruction or any code and / or instruction combination or set on a non-transient processor-readable medium and / or computer-readable medium that may be incorporated into a computer program product.

[0122] The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present claims. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and implementations without departing from the scope of the claims. Thus, the present disclosure is not intended to be limited to the embodiments and implementations described herein, but should be accorded the broadest scope consistent with the appended claims and the principles and novel features disclosed herein.

Claims

1. A method for weight layout transformation of a weight tensor, comprising: accessing a first memory to retrieve the weights of the weight tensor in a transposed order different from an order used to retrieve the weights for computation at a network layer of a trained machine learning model; as well as The weights are loaded into a second memory in the transposed order.

2. The method of claim 1 , wherein accessing the first memory to retrieve the weights in a transposed order different from an order used to retrieve the weights of the weight tensor for computation at the network layer of the trained machine learning model comprises: The first memory is accessed to retrieve the weights according to a memory access pattern that iterates over the slowest varying dimension of the weight tensor.

3. The method of claim 2, wherein accessing the first memory to retrieve the weights according to a memory access pattern that iterates over the slowest varying dimension of the weight tensor comprises: The weights are retrieved according to a memory access pattern that iterates over the height dimension of the weight tensor.

4. The method of claim 1 , wherein accessing the first memory to retrieve the weights in a transposed order different from an order used to retrieve the weights of the weight tensor for computation at the network layer of the trained machine learning model comprises: Accessing the first memory to retrieve weights of the weight tensor in an order specified by a first counter variable and a second counter variable of a first memory access command, wherein the first counter variable and the second counter variable are configured to represent a position in the weight tensor, and wherein the first counter variable and the second counter variable are transposed relative to a second memory access command having the first counter variable and the second counter variable of the network layer of the trained machine learning model.

5. The method of claim 1 , wherein loading the weights into the second memory in the transposed order comprises: The weights are loaded into the second memory in a linear layout according to a memory access pattern that iterates over the slowest varying dimension of the weight tensor.

6. The method of claim 5, wherein loading the weights into the second memory in the linear layout comprises: The weights are loaded into the second memory as a linear array.

7. The method of claim 1, further comprising: retrieving the weights from the second memory in the transposed order; as well as The weights are reordered into the order for implementing computations at the network layer of the trained machine learning model.

8. The method of claim 1, wherein the first memory and the second memory are in the same memory device.

9. A computing device comprising: a first memory; A second memory; as well as a processing device coupled to the first memory and the second memory and configured with processor-executable instructions for: accessing the first memory to retrieve the weights of a weight tensor in a transposed order different from an order used to retrieve the weights for computation at a network layer of a trained machine learning model; as well as The weights are loaded into the second memory in the transposed order.

10. A computing device as described in claim 9, wherein the processing device is further configured with processor-executable instructions for accessing the first memory to retrieve the weights in a transposed order different from the order used to retrieve the weights of the weight tensor for calculations at the network layer of the trained machine learning model by: accessing the first memory to retrieve the weights according to a memory access pattern that iterates over the slowest changing dimension of the weight tensor.

11. A computing device as described in claim 10, wherein the processing device is further configured with processor executable instructions for accessing the first memory to retrieve the weight according to a memory access pattern that iterates over the slowest changing dimension of the weight tensor by performing the following operations: retrieving the weight according to a memory access pattern that iterates over the height dimension of the weight tensor.

12. A computing device as described in claim 9, wherein the processing device is further configured with processor-executable instructions for accessing the first memory to retrieve the weights in a transposed order different from the order used to retrieve the weights of the weight tensor for calculations at the network layer of the trained machine learning model by the following operations: accessing the first memory to retrieve the weights of the weight tensor in an order specified by a first counter variable and a second counter variable of a first memory access command, wherein the first counter variable and the second counter variable are configured to represent a position in the weight tensor, and wherein the first counter variable and the second counter variable are transposed relative to a second memory access command having the first counter variable and the second counter variable of the network layer of the trained machine learning model.

13. A computing device as described in claim 9, wherein the processing device is further configured with processor-executable instructions for loading the weights into the second memory in the transposed order by performing the following operations: loading the weights into the second memory in a linear layout according to a memory access pattern that iterates over the slowest changing dimension of the weight tensor.

14. The computing device of claim 13, wherein the processing device is further configured with processor-executable instructions for loading the weights into the second memory in the linear layout by: loading the weights into the second memory as a linear array.

15. The computing device of claim 9, wherein the processing device is further configured with processor-executable instructions for: retrieving the weights from the second memory in the transposed order; and The weights are reordered into the order for implementing computations at the network layer of the trained machine learning model.

16. The computing device of claim 9, wherein the first memory and the second memory are in the same memory device.

17. A computing device comprising: means for accessing the first memory to retrieve the weights of the weight tensor in a transposed order different from the order used to retrieve the weights for computation at a network layer of a trained machine learning model; as well as Means for loading the weights into a second memory in the transposed order.

18. A non-transitory processor-readable medium having stored thereon processor-executable instructions configured to cause a processor of a computing device to perform operations comprising: accessing the first memory to retrieve the weights of the weight tensor in a transposed order different from an order used to retrieve the weights for computation at a network layer of the trained machine learning model; as well as The weights are loaded into a second memory in the transposed order.

19. A non-volatile processor-readable medium as described in claim 18, wherein the processor-executable instructions are configured to cause the processor of the computing device to perform operations such that: accessing the first memory to retrieve the weights in a transposed order different from the order used to retrieve the weights of the weight tensor for calculations at the network layer of the trained machine learning model includes accessing the first memory to retrieve the weights according to a memory access pattern that iterates over the slowest changing dimension of the weight tensor.

20. A non-volatile processor-readable medium as described in claim 19, wherein the processor-executable instructions are configured to cause the processor of the computing device to perform operations such that: accessing the first memory to retrieve the weight according to a memory access pattern that iterates over the slowest changing dimension of the weight tensor includes retrieving the weight according to a memory access pattern that iterates over the height dimension of the weight tensor.

21. A non-volatile processor-readable medium as described in claim 18, wherein the processor-executable instructions are configured to cause the processor of the computing device to perform operations such that: accessing the first memory to retrieve the weights in a transposed order different from the order used to retrieve the weights of the weight tensor for calculations at the network layer of the trained machine learning model includes accessing the first memory to retrieve the weights of the weight tensor in an order specified by a first counter variable and a second counter variable of a first memory access command, wherein the first counter variable and the second counter variable are configured to represent a position in the weight tensor, and wherein the first counter variable and the second counter variable are transposed relative to a second memory access command having the first counter variable and the second counter variable for the network layer of the trained machine learning model.

22. A non-volatile processor-readable medium as described in claim 18, wherein the processor-executable instructions are configured to cause the processor of the computing device to perform operations such that: loading the weights into the second memory in the transposed order includes loading the weights into the second memory in a linear layout according to a memory access pattern that iterates over the slowest changing dimension of the weight tensor.

23. A non-volatile processor-readable medium as described in claim 22, wherein the processor-executable instructions are configured to cause the processor of the computing device to perform operations such that: loading the weights into the second memory in the linear layout includes loading the weights into the second memory as a linear array.

24. The non-transitory processor-readable medium of claim 18, wherein the processor-executable instructions are configured to cause the processor of the computing device to perform operations further comprising: retrieving the weights from the second memory in the transposed order; and The weights are reordered into the order for implementing computations at the network layer of the trained machine learning model.

25. The non-transitory processor-readable medium of claim 18, wherein the processor-executable instructions are configured to cause the processor of the computing device to perform operations such that: the second memory is the first memory.

Images