System and method for matrix multiplication instructions with specified bias floating point operands

Abstract

A microprocessor system includes a matrix calculation unit and a control unit. The matrix calculation unit includes a plurality of processing elements. The control unit is configured to provide a matrix processor instruction to the matrix calculation unit. The matrix processor instruction specifies a floating point operand formatted using a first floating point representation format. The matrix calculation unit accumulates intermediate result values ​​calculated using the floating point operands. The intermediate result values ​​are in a second floating point representation format.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 16 / 403,083, filed on May 3, 2019, entitled “DATA PATH FOR EXTENDED MATRIX NODE ENGINE WITH HYBRID DATA FORMAT,” and also claims priority to U.S. Patent Application No. 16 / 421,225, filed on May 23, 2019, entitled “EXTENDED MATRIX NODE WITH CONFIGURABLE DATA FORMAT,” the disclosures of which are incorporated herein by reference in their entirety. Background Art

[0003] Machine learning training is a data and compute intensive operation. This process is tedious and time consuming, requiring a large amount of relevant training data and computing resources to process it. Moreover, data and computing resources will only increase with the complexity of the problem being solved. To train machine learning models, high-power CPUs perform complex matrix operations using training data to determine appropriate weights. In order to increase the speed of training, graphics processing units (GPUs) are used as an alternative or supplement to traditional CPUs. GPUs allow some training in training to be parallelized and help optimize certain mathematical operations. However, traditionally, GPUs are designed for processing graphics problems, such as rendering a three-dimensional world onto a two-dimensional display. When applied to machine learning, GPUs may require a lot of computing power to meet the computing power they provide. Moreover, the data formats and data pipelines used by GPUs are designed for graphics processing, not for training machine learning models. Therefore, a powerful, computationally powerful and energy-efficient machine learning training system is needed. Such a system should support high data bandwidth to significantly increase the amount of training data that can be processed. Moreover, the data format and data pipeline should be optimized for training data and the resulting machine learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0005] Figure 1 is a flow diagram illustrating an embodiment of a process for training a machine learning model.

[0006] Figure 2 is a block diagram illustrating an embodiment of a system for training a machine learning model.

[0007] Figure 3 is a block diagram illustrating an embodiment of a node engine for performing matrix calculations.

[0008] Figure 4 is a block diagram illustrating an embodiment of an 8-bit floating point format.

[0009] Figure 5is a block diagram illustrating an embodiment of a 21-bit floating point format.

[0010] Figure 6 is a flow chart illustrating an embodiment of a process for performing matrix calculations.

[0011] Figure 7 is a flow chart illustrating an embodiment of a process for performing matrix calculations.

[0012] Figure 8 is a flow chart illustrating an embodiment of a process for performing multiple interleaved matrix calculations. DETAILED DESCRIPTION

[0013] The present invention may be implemented in a variety of ways, including as a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and / or a processor, such as a processor configured to execute instructions stored on and / or provided by a memory coupled to the processor. In this specification, these implementations or any other form that the present invention may take may be referred to as techniques. In general, the order of the steps of the disclosed processes may be changed within the scope of the present invention. Unless otherwise stated, components such as processors or memories described as being configured to perform tasks may be implemented as general components that are temporarily configured to perform tasks at a given time or as specific components manufactured to perform tasks. As used herein, the term 'processor' refers to one or more devices, circuits, and / or processing cores configured to process data such as computer program instructions.

[0014] Provided below are detailed descriptions of one or more embodiments of the present invention and drawings illustrating the principles of the present invention. The present invention is described in conjunction with these embodiments, but the present invention is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention encompasses many alternatives, modifications and equivalents. In the following description, many specific details are set forth to provide a thorough understanding of the present invention. These details are provided for illustrative purposes, and the present invention can be practiced according to the claims without some or all of these specific details. For the sake of clarity, technical materials known in the technical field related to the present invention have not been described in detail to avoid unnecessary confusion of the present invention.

[0015] A scalable node engine with multiple matrix processors and configurable data formats is disclosed. As a core component of a training platform for a machine learning model, the node engine can be arranged in a network to perform training on the machine learning model. As computing and data requirements increase, the number of node engines in the network can be increased to handle additional requirements. Compared with traditional CPUs and GPUs that delegate tasks for similar workloads, the disclosed node engine is very efficient in terms of performance per square millimeter per watt. The node engine architecture achieves this performance improvement in part by optimizing the data format and data path for machine learning workloads. For example, the node engine includes multiple matrix processors, each of which can interleave multiple matrix operations. A node engine with a group of eight matrix processors can calculate the result of matrix multiplication in each cycle. When stopping waiting for data for a first set of related matrix operations, each matrix processor can interleave a second set of related matrix operations to utilize computing resources that would otherwise be idle. In some embodiments, matrix operands are stored using a low-order floating point format and intermediate results and final results are calculated using a high-order floating point format. The low-order format improves the read data bandwidth of the matrix processor, and the high-order format retains the accuracy and precision of the matrix result, for example, by preventing the accuracy of the loss quantization result. Different configurable data formats can be selected to specify different data format configurations, for example, to change the number of digits assigned to the mantissa field and the exponent field. This allows to optimize the data format based on the specific matrix operation for a specific machine learning task. Additionally, the data format can include a configurable bias for a biased exponent. This improves the scope of the exponent and allows to utilize a larger scope.

[0016] In some embodiments, the node engine is arranged in a mesh network. Each node engine includes a control unit, a memory, a register, a plurality of matrix processors, and a post-processing unit such as a vector calculation unit. The control unit can process custom instructions, which include matrix calculation instructions involving a matrix processor in a plurality of matrix processors, and are used to synchronize the results between different matrix processors and the node engine. The matrix results can be stored in a register file and processed using vector operations by the post-processing unit. The software running the node engine can perform large matrix operations and subdivide the problem. Different sub-components of the problem may be distributed to different node engines and different matrix processors of each node engine. For example, two large matrices can be sliced ​​so that each slice is optimized to the matrix size of the matrix processor. Then, the slices can be assigned to different matrix processors of different node engines, where matrix multiplication is performed on the slices. The result of each matrix multiplication can be combined to calculate the multiplication result of the original larger matrix.

[0017] In some embodiments, the microprocessor system includes a matrix calculation unit and a control unit. The matrix calculation unit includes one or more processing elements. For example, the matrix calculation unit includes a calculation unit matrix for determining the calculation results of two elements from two operands. An 8×8 matrix calculation unit includes 64 calculation units. Similarly, an M×N matrix calculation unit includes M×N calculation units. The matrix calculation unit is part of a matrix processor controlled by a control unit. In some embodiments, the control unit is configured to provide a matrix processor instruction to the matrix calculation unit. For example, the control unit provides a matrix multiplication instruction to the matrix processor for execution by the matrix calculation unit. The matrix processor instruction specifies a floating-point operand that is formatted with an exponent that has been biased using a specified configurable bias. For example, a matrix multiplication instruction specifies two floating-point matrix operands. Each element of the matrix operand is formatted using a specific floating-point format and a configurable exponent bias. Together with the matrix operand, the matrix processor instruction specifies the floating-point format used by the matrix element (such as a format that allocates 1 bit for the sign bit, 4 bits for the exponent, and 3 bits for the mantissa) and a specific exponent bias. In various embodiments, the bias can be configured by specifying a value corresponding to the exponential bias. In some embodiments, the bias is reconfigurable. For example, a matrix instruction can specify a new bias for reconfiguring a configurable bias. In some embodiments, the floating point format supports denormal numbers to increase the number of values ​​that can be represented.

[0018] In some embodiments, the matrix processor instruction specifies a floating-point operand formatted using a first floating-point representation format. For example, the instruction specifies an 8-bit floating-point format, which allocates 4 bits for the exponent, 3 bits for the mantissa, and a single sign bit. The specified format is used for the elements of the matrix operand. The format can be selected to increase the data bandwidth of the matrix calculation unit entering the matrix processor. The matrix calculation unit accumulates the intermediate result values ​​calculated using the floating-point operand, and the intermediate result values ​​adopt the second floating-point representation format. For example, the intermediate result uses different floating-point formats, such as 21-bit floating-point formats. As another example, the intermediate result can use different floating-point formats, such as 27 or another appropriate floating-point format. The number of bits dedicated to the intermediate result can be selected to prevent the accuracy loss when the result is quantized. The format of the intermediate result can be selected to use a large number of bits to prevent overflow errors and / or underflow errors that may be caused by using the first floating-point format. The matrix calculation unit outputs the accumulated intermediate result as an output formatted with a third floating-point representation format. For example, a plurality of accumulated intermediate results can be removed from the matrix processor as matrix results. The result can be output using a third format compatible with the bus to which the matrix processor is connected. For example, the node engine can utilize an internal bus with a width of 64 bytes. The intermediate accumulation result can be output from the matrix calculation unit as a 16-bit floating point value, allowing 32 elements to be moved from the matrix processor for each move instruction. The accumulation result with 64 elements can be moved from the matrix processor to the register file of the node engine using two move instructions, wherein each instruction moves 32 elements. The upward shift instruction can be used to move the high-order 32 elements (e.g., element 32 to 63), and the downward shift instruction can be used to move the low-order 32 elements (e.g., element 0 to 31). In some embodiments, the move instruction is a non-destructive instruction, and when a value is moved from the source accumulator of the matrix processor to a memory location (such as an output array or a register) outside the matrix processor, the content of the source accumulator will not be cleared.

[0019] Figure 1 is a flow chart illustrating an embodiment of a process for training a machine learning model. For example, Figure 1The process can be used to train models for autonomous driving or driver-assisted driving. Driving data can be captured when the vehicle is driven autonomously by a human driver or when a mixture of manual driving and assisted driving is used. The captured data is prepared as training data and used to train new machine learning models to improve the driving experience. The new driving experience can be improved in areas such as safety, efficiency (power, time, etc.), comfort, performance, convenience, etc. Once the new model is trained and verified, the newly trained model will be deployed to the vehicle that is used by one or more machine learning networks to achieve improved driving features and functions. New features may include autonomous driving features or assisted driving features, such as autonomous lane changes, autonomous lane merging into highways, autonomous exiting highways, improved detection of obstacles and road scenes, and autonomous navigation-based driving. In various embodiments, the machine learning model can be trained on a training platform that utilizes multiple node engines and each of which includes multiple matrix processors and configurable data formats.

[0020] At 101, data is captured for machine learning training. In some embodiments, data corresponding to vehicle driving is captured while the vehicle is driven by a human, an autonomous driving system, or both. The captured data of vehicle driving conditions may include image sensor data, vehicle operating parameters (e.g., speed, steering, etc.), vehicle type information (e.g., left-hand drive, right-hand drive, vehicle model, etc.), whether autonomous driving is enabled, when autonomous driving was last disengaged, detected obstacles, driving conditions, etc. The data may be captured passively without interfering with vehicle driving and without driver assistance.

[0021] In various embodiments, the vehicle may be equipped with sensors of different arrangements to capture different forms of data. In some embodiments, the sensor data may be visual data, ultrasonic data, LiDAR data, or other appropriate sensor data. For example, images are captured from a high dynamic range forward camera. As another example, ultrasonic data is captured from a side ultrasonic sensor. In some embodiments, the vehicle is attached with multiple sensors for capturing data. For example, in some embodiments, eight surround cameras are attached to the vehicle and provide 360-degree visibility around the vehicle, with a range of up to 250 meters. Differently arranged camera sensors may include a wide front camera, a narrow front camera, a rear-view camera, a front-view side camera, and / or a rear-view side camera. In some embodiments, additional ultrasonic sensors and / or radar sensors are used to capture surrounding details. For example, twelve ultrasonic sensors may be attached to the vehicle to detect both hard and soft objects. Additional forward radars may also be used to capture data of the surrounding environment. In various embodiments, radar sensors are able to capture surrounding details despite heavy rain, fog, dust, and other vehicles. Individual sensors are used to capture the environment around the vehicle, and the captured data is stored for consideration as training data for a deep learning network.

[0022] Once captured, the data captured from one or more vehicles is transmitted to the machine learning training platform. For example, a vehicle with wireless connectivity (such as a cellular or WiFi connection) can wirelessly transmit data to the machine learning training platform. As another option, the captured data can be downloaded from the vehicle while a technician is servicing the vehicle. In various embodiments, data captured from multiple vehicles (such as a fleet of vehicles) is aggregated at the machine learning training platform and used as at least one training data source for the training data source.

[0023] At 103, the captured data for training the machine learning model is prepared. At 101, the data captured from the vehicle is prepared as training data. In some scenarios, the data is divided into training data and validation data. Preparing the data may include selecting (or eliminating) the captured data to identify particularly good training data. In some embodiments, the data is annotated to identify features for training. For example, lane markings, traffic lights, traffic signs, vehicles, pedestrians, etc. may be annotated to enhance the usefulness of the training data as part of data preparation. As another example, the data may be converted to a different format or pre-processed as part of the preparation process. In some embodiments, the data may be converted from a source data format to a format compatible with a matrix processor. For example, data captured as fixed point data may be converted to floating point data to improve accuracy.

[0024] At 105, a machine learning model is trained. Using the training data prepared at 103, one or more machine learning models are trained. The training can utilize both a training data set and a validation data set. In some embodiments, the training utilizes a machine learning platform consisting of a plurality of node engines and each of which includes a plurality of matrix processors. By utilizing, for example, a plurality of node engines organized into a grid or another appropriate architecture, complex machine learning training problems can be parallelized and executed faster and more efficiently. Similarly, since each node engine includes a plurality of matrix processors, each node can perform a plurality of matrix operations in parallel. In some embodiments, by operating a plurality of matrix processors in parallel, a node engine can output the result of a matrix multiplication at each clock cycle. The delay for waiting for data to be read is significantly reduced, the delay between matrix multiplication results is significantly reduced, and the performance bandwidth is significantly increased.

[0025] The result of the training is one or more trained machine learning models. In some embodiments, multiple models are trained, each for a potentially different neural network. For example, one machine learning model may be trained to use sensor data from a forward-facing camera as input, while another model may be trained to use sensor data from a side-facing ultrasonic sensor as input.

[0026] At 107, the trained machine learning model is distributed. For example, the trained model is distributed to a vehicle and installed on the vehicle. The model can be installed via an over-the-air update, by a technician when servicing the vehicle, or otherwise. In some cases, the model is packaged in a data format for ease of installation on the vehicle. For example, the model can be compressed to minimize the time and bandwidth required to transmit the model to the vehicle. In some embodiments, multiple models (e.g., each model for a different neural network engine running on the vehicle) can be packaged together and transmitted to the vehicle as a single package.

[0027] At 109, the trained machine learning model is applied. For example, a convolutional neural network on the vehicle utilizes the new model to process sensor data and implement autonomous driving or driver assistance features. In some embodiments, more than one model is applied and / or more than one neural network is utilized. For example, on some vehicles, multiple neural networks are utilized to process different data from different sensors. Once the new model is utilized, data reflecting the performance of the new model can be captured and used for future training. Figure 1 The process can be used to continuously improve the performance of the machine learning network. In this way, the processing loop returns to 101 of capturing data. The data can be analyzed to identify difficult use cases for the currently deployed model, and the corresponding captured data can be utilized for future training.

[0028] Figure 2is a block diagram illustrating an embodiment of a system for training a machine learning model. Figure 2 A training system can train a machine learning model to achieve autonomous driving and / or driver assistance functions. In some embodiments, Figure 2 The training system is used to perform Figure 1 process. In the example shown, the training system utilizes certain training-related subsystems of the vehicle subsystem 201 located on the vehicle. The training-related subsystems communicate with the server side of the training system located in one or more training data centers 221. The vehicle subsystem 201 includes a sensor 203, a deep learning network 205, an AI processor 207, a vehicle control module 209, a network interface 211, a vehicle data capture system 213, and a capture data storage device 215. For example, additional vehicle subsystems may exist to perform other functions, but are not shown. One or more training data centers 221 include a training platform 223, a training data storage device 227, and a model data storage device 229. The training platform 223 includes at least one or more node engines 225. The node engines are connected (e.g., in a mesh network) to perform parallel processing on machine learning training. In some embodiments, the training platform 223, the training data storage device 227, and the model data storage device 229 are located in a single data center, but can also be distributed or replicated across multiple data centers.

[0029] In some embodiments, a vehicle (not shown) includes a vehicle subsystem 201 to implement autonomous and driver assistance functions and capture data that can be used to train one or more machine learning models to implement and / or improve functions and / or new features. In various embodiments, different vehicle subsystems can be communicatively connected. For example, sensor data from sensor 203 is fed to vehicle data capture system 213 to be stored in capture data storage device 215. The captured data is sent to training platform 223 via network interface 211. As another example, sensor data from sensor 203 is fed to deep learning network 205 running on AI processor 207. The output of deep learning network 205 running on AI processor 207 is fed to vehicle control module 209. In various embodiments, network interface 211 is a wireless network interface, such as a wireless network interface including WiFi and / or cellular network connectivity. Network interface 211 is used to communicate with a remote server, make calls, send and / or receive text messages, transmit sensor data to training platform 223, etc. In some embodiments, vehicle subsystem 201 may include additional or fewer subsystems, as appropriate. For example, in some embodiments, an image preprocessor (not shown) is used to preprocess the captured sensor data. As another example, in some embodiments, a post-processing component (not shown) is used to perform post-processing on the output of the deep learning network 205 before providing the output to the vehicle control module 209. In some embodiments, a trigger classifier component (not shown) is used to identify driving data as potential training data.

[0030] In some embodiments, sensor 203 includes one or more sensors. Sensor 203 can be attached to the vehicle at different locations of the vehicle and / or oriented in one or more different directions. For example, sensor 203 can be attached to the front, side, rear and / or roof of the vehicle in the forward, rear, lateral and other directions. In some embodiments, sensor 203 can be an image sensor such as a high dynamic range camera. In some embodiments, sensor 203 includes non-visual sensors. Sensor 203 may include radar, LiDAR and / or ultrasonic sensors, etc. In some embodiments, sensor 203 is not installed on a vehicle with vehicle control module 209. For example, sensor 203 can be installed on an adjacent vehicle and / or attached to a road or environment, and is included as part of a system for capturing sensor data.

[0031] In some embodiments, the deep learning network 205 is a deep learning network for implementing autonomous vehicle control. For example, the deep learning network 205 may be an artificial neural network such as a convolutional neural network (CNN) that is trained using sensor data and whose output is provided to the vehicle control module 209. The machine learning model used by the deep learning network 205 may be a neural network that can be used to perform autonomous vehicle control. Figure 2 system for training.

[0032] In some embodiments, the artificial intelligence (AI) processor 207 is a hardware processor for running the deep learning network 205. In some embodiments, the AI ​​processor 207 is a dedicated AI processor for performing inference on sensor data using a convolutional neural network (CNN). The AI ​​processor 207 can be optimized for the bit depth of the sensor data and / or for deep learning operations such as neural network operations including convolution, dot product, vector and / or matrix operations. In some embodiments, the AI ​​processor 207 is implemented using a graphics processing unit (GPU). In various embodiments, the AI ​​processor 207 is coupled to a memory that is configured to provide instructions to the AI ​​processor that, when executed, cause the AI ​​processor to perform deep learning analysis on the received input sensor data and determine machine learning results for at least partially autonomously operating the vehicle.

[0033] In some embodiments, the vehicle control module 209 is used to process the output of the artificial intelligence (AI) processor 207 and transform the output into a vehicle control operation. In some embodiments, the vehicle control module 209 is used to control the vehicle for autonomous driving and can adjust the speed and / or steering of the vehicle. For example, the vehicle control module 209 can be used to control the vehicle by braking, steering, changing lanes, accelerating, and merging into another lane. In some embodiments, the vehicle control module 209 is used to control vehicle lighting, such as brake lights, turn signals, headlights, etc. In some embodiments, the vehicle control module 209 is used to control vehicle audio conditions, such as the vehicle's sound system, play audio alarms, enable microphones, enable speakers, etc. In some embodiments, the vehicle control module 209 is used to control notification systems including warning systems to notify drivers and / or passengers of driving events, such as potential collisions or approaching an expected destination. In some embodiments, the vehicle control module 209 is used to adjust sensors such as sensors 203 of the vehicle. For example, the vehicle control module 209 can be used to change parameters of one or more sensors, such as modifying orientation, changing output resolution and / or format type, increasing or decreasing capture rate, adjusting captured dynamic range, adjusting the focus of the camera, enabling and / or disabling sensors, etc. In various embodiments, the vehicle control module 209 is used to implement autonomous driving and / or driver assistance control of the vehicle.

[0034] In some embodiments, the network interface 211 is a communication interface for sending and / or receiving data including captured sensor data. In various embodiments, the network interface 211 includes a cellular or wireless interface for interfacing with a remote server such as the training platform 223 to connect and make voice calls, send and / or receive text messages, transmit sensor data, receive updates to the autonomous driving system including newly trained machine learning models, etc. For example, the network interface 211 can be used to receive instructions and / or operating parameters for the sensor 203, the deep learning network 205, the AI ​​processor 207, the vehicle control module 209, and / or the vehicle data capture system 213. For example, the machine learning model of the deep learning network 205 can be updated using the network interface 211. As another example, the network interface 211 can be used to update the firmware of the sensor 203 and / or the operating parameters of the vehicle data capture system 213, such as filters and / or parameters for determining the type and amount of data to be captured.

[0035] In some embodiments, the vehicle data capture system 213 and the captured data storage device 215 are used to capture and store data associated with vehicle driving conditions. The data captured by the vehicle data capture system 213 is stored in the captured data storage device 215. The captured data storage device 215 can be implemented using any appropriate data storage device such as a hard drive, a non-volatile memory, etc. In some embodiments, the captured data storage device 215 is implemented using a database, a file system, or another data organization means. The data of the vehicle driving conditions captured may include image sensor data, vehicle operating parameters (e.g., speed, steering, etc.), vehicle type information (e.g., left-hand driving, right-hand driving, vehicle model, etc.), whether autonomous driving is enabled, the time when autonomous driving was last disengaged, the obstacles detected, driving conditions, etc. Data can be passively captured without interfering with vehicle driving and without the need for driver assistance. The data captured by the vehicle data capture system 213 includes data captured from the sensor 203.

[0036] In some embodiments, the vehicle data capture system 213 communicates with the training platform 223 via a network interface 211. The network interface 211 can be a wireless network, such as WiFi and / or a cellular network. The vehicle data capture system 213 utilizes the network interface 211 to transfer the captured data stored in the capture data storage device 215 to the training platform 223. In some embodiments, the network interface 211 is used to download a trained machine learning model to be installed in the deep learning network 205 running on the vehicle.

[0037] exist Figure 2In the example of, the server-side components of the training system are located in one or more data centers of one or more training data centers 221 and include a training platform 223, a training data storage device 227, and a model data storage device 229. The training platform 223 includes one or more computer servers for receiving captured data from the vehicle data acquisition system 213. The training platform 223 is communicatively connected to the vehicle data acquisition system 213 via a wireless network interface 211 through a computer network such as a wired network or an optical network of one or more training data centers 221. The training platform 223 also includes one or more node engines 225. For example, multiple node engines 225 can be connected in a mesh network. The training platform 223 receives the captured data from the vehicle data capture system 213, processes the data into usable training (and verification) data, and uses the node engine 225 to train one or more new machine learning models. The training data storage device 227 is used to store the captured data received from one or more vehicles. In some embodiments, the processed captured data used as training data, including annotated data, is stored in the training data storage device 227. Once the training is completed, the model data storage device 229 is used to store the trained machine learning model. For example, different versions of trained machine learning models can be stored in the model data storage device 229 and used to determine the relative functions of different models and identify areas for improvement. In some embodiments, one or more data storage devices are used to implement the training data storage device 227 and the model data storage device 229.

[0038] In some embodiments, the node engine 225 includes a plurality of connection nodes, which can be used to parallelize computing tasks. Each connection node includes at least one matrix processor, and may be more than one matrix processor. For example, a single node may include eight matrix processors, each of which is capable of determining at least one matrix multiplication result. In some embodiments, the matrix multiplication result requires a single matrix processor to calculate at least a minimum number of clock cycles. By extending each node to include multiple matrix processors, after an initial delay corresponding to the minimum number of clock cycles for calculating matrix multiplication, the node can output a matrix multiplication result for each clock cycle. For example, if matrix multiplication requires eight clock cycles to complete, after an initial delay of seven clock cycles, a node with eight matrix processors can determine the result of matrix multiplication in each clock cycle. In various embodiments, throughput is also determined by memory access, including the delay of accessing matrix operands. In various embodiments, the node engine can use a variety of digital formats to perform matrix calculations. For example, a node can utilize a fixed point format and a floating point format. For a floating point format, a node can be configured to run in a variety of formats such as an 8-bit format, a 16-bit format, and a 32-bit format. For each bit depth, one or more different formats can be selected. Depending on the computational goal, different formats can be used to represent numerical values. A format can be selected to assign higher precision to the mantissa of a floating point number, and another format can be selected to assign higher precision to the exponent of a floating point number. In some embodiments, the floating point format utilizes configurable biases to further customize computational operations. The configurability of the digital format allows the training system to target different machine learning operations, for example, based on expected input values, intermediate values, and output values. In various embodiments, the configurability of nodes including support for multiple floating point formats and floating point formats using configurable biases greatly improves the bandwidth and performance of matrix computational operations without sacrificing precision and accuracy. Similarly, power consumption and efficiency are also significantly improved.

[0039] Figure 33 is a block diagram illustrating an embodiment of a node engine for performing matrix calculations. In the example shown, the node engine 300 includes a control unit 301, a memory 303, a load register 305, a post-processing unit register file 307, multiplexers 309 and 311, a matrix processor 313 and 351 to 357, an output array 315 and a post-processing unit 317. In various embodiments, the node engine may include multiple matrix processors to calculate multiple matrix operations in parallel. In the example shown, the node engine 300 includes eight matrix processors 313 and 351 to 357. Each matrix processor includes a data input array, a weight input array, a plurality of output accumulators, and a matrix calculation unit. In the example shown, the matrix processor 313 includes a data input array 321, a weight input array 323, and two output accumulators 329 and 331. The data input array and the weight input array feed the input to the matrix calculation unit 325. For example, data in an input array (e.g., data input array 321 and / or weight input array 323) is shifted a certain number of bytes (e.g., eight bytes) over multiple cycles (e.g., eight consecutive cycles) to feed the matrix calculation unit 325. In some embodiments, each matrix processor includes a single data input array and a single weight input array. The matrix calculation unit 325 includes a matrix of calculation units such as calculation unit 327. The M×N dimensional matrix calculation unit includes M×N calculation units. The size of each input array is designed to fit the entire input matrix, and the size of each output accumulator is designed to fit the entire matrix result. In some embodiments, the node engine supports multiple floating point formats, which include Figure 4 The 8-bit floating point formats 400 and 410 and Figure 5 In some embodiments, the node engine 300 is used to perform Figure 1 , Figure 6 , Figure 7 and / or Figure 8 process.

[0040] In some embodiments, the node engine 300 may include additional components and additional control lines that are not shown. For example, the node engine 300 may include additional registers, such as scalar registers, one or more memory caches, a data formatter for formatting values ​​for a matrix processor, and additional control lines from the control unit 301 to subcomponents such as multiplexers 309 and 311 and matrix processors 351 to 357, as several examples. In some embodiments, certain registers (not shown) are dedicated to storing configurable parameters such as configurable biases of digital formats and floating point numbers. In some embodiments, the bus connecting the different components of the node engine 300 is a wide data bus. The size of the bus can be selected to optimize the transmission matrix value. For example, the width of the bus can be 64 bytes. This allows 8×8 matrices of 64 1-byte elements to be transmitted from memory to registers, matrix processors, etc. as containing units.

[0041] In the example shown, the control unit 301 is communicatively connected to one or more components of the node engine 300, including a memory 303, a matrix processor 313, an output array 315, and a post-processing unit 317. Although not shown, the control unit 301 is also communicatively connected to each of the remaining matrix processors 351 to 357. In various embodiments, the control unit 301 is used to synchronize the processing of computational operations, including matrix operations and post-processing operations (such as vector operations) and / or access to memory and registers. For example, the control unit 301 sends a signal to the matrix processor 313 to schedule matrix calculation instructions and can monitor the ready signal from the matrix processor 313 to indicate when a new instruction can be received and / or when a matrix operation is completed and when a matrix result is ready.

[0042] In some embodiments, memory 303 is a memory module for storing input operands and output results of matrix calculations and post-processing calculations. Memory 303 may include one or more caches (not shown). In the example shown, memory 303 is connected to load register 305, multiplexers 309 and 311, and post-processing unit register file 307. Additional connections or fewer connections are possible, depending on the flexibility required for storing data in memory and retrieving data from memory. As shown, data can be read from memory to load register 305 and post-processing unit register file 307 and read from load register 305 and post-processing unit register file 307 to memory. Connection with registers allows data values ​​to be quickly stored in registers, for example, as arguments (arguments) of matrix or vector calculations. Memory 303 is also connected to multiplexers 309 and 311 so that input matrices can be retrieved from memory. In some embodiments, memory access to memory 303 is controlled by a memory arbiter (not shown), such as by queuing memory requests and prioritizing certain memory reads over other memory reads to optimize memory requests. In some embodiments, memory 303 is static random access memory (SRAM).

[0043] In some embodiments, the node engine 300 includes registers such as load registers 305 and post-processing unit register files 307. These registers can be used to optimize memory access. As several examples, registers can be used to store values ​​retrieved from memory 303, to store these values ​​before writing values ​​to memory 303, to store input values ​​and output values ​​of the matrix processor, and to store input values ​​and output values ​​of the post-processing unit. In some embodiments, the post-processing unit register file 307 is a register file for the post-processing unit 317 and is compatible with different lane configurations (e.g., 64 lane configurations, 32 lane configurations, and / or 16 lane configurations) of the post-processing unit 317. For example, the registers of the post-processing unit register file 307 can be addressed using various byte formats such as 1-byte values, 2-byte values, and 4-byte values. In some embodiments, the size of each register is 64 bytes and the register can store 64 1-byte elements, 32 2-byte elements, or 16 4-byte elements. In various embodiments, the data format can be configured and includes various 8-bit floating point formats, 16-bit floating point formats, and 32-bit floating point formats.

[0044] In some embodiments, multiplexers are used to select the input operand source to the matrix processor. In the example shown, multiplexers 309 and 311 are used to select the source for the data input matrix and the weight input matrix for the matrix processor 313. According to the control signal received at each multiplexer, the data can be derived from the memory 303 or the post-processing unit register file 307. In some embodiments, the data derived from the memory 303 is retrieved via the register of the load register 305. In some embodiments, multiplexers 309 and 311 are also used to select the data input matrix and the weight input matrix for the matrix processors 351 to 357. By offsetting the processing of multiple matrix processors of the node engine, a pair of multiplexers are used to select the input of all matrix processors of the node engine. In various embodiments, multiplexers 309 and 311 are used to control which matrix processor receives which matrix operand. According to the configuration, a single matrix processor, a subset of all matrix processors, or all matrix processors receive the selected matrix operand. In various alternative embodiments, node engine 300 includes additional multiplexers (not shown) dedicated to each of matrix processors 351 - 357 .

[0045] In some embodiments, the matrix processor 313 receives matrix operation instructions and performs matrix calculations such as matrix multiplication. For each matrix instruction, the matrix processor 313 stores one or more matrix operands in one or more input arrays. For example, the data matrix is ​​stored in a data input array such as data input array 321, and the weight matrix is ​​stored in a weight input array such as weight input array 323. In various embodiments, the matrix operand is a pair of data and weight matrices, a pair of data and gradient matrices, a pair of weight and gradient matrices, or another pair of appropriate matrix operands. In various embodiments, the matrix processor 313 is used to calculate multiple related matrix calculations as part of the process for calculating matrix multiplication of matrices that are too large to fit in the input arrays 321 and 323 of the matrix processor 313. The results of the related matrix calculations are combined as part of the process of calculating matrix multiplication of larger matrices. In various embodiments, the matrix processor 313 interleaves multiple matrix operations (related or unrelated). For example, the matrix processor 313 can interleave one or more related matrix operations on the first pair of matrices with one or more related matrix operations on the second pair of matrices. For example, the matrix processor 313 can perform one or more related matrix operations on the first pair of matrices and one or more related matrix operations on the second pair of matrices. For example, the matrix processor 313 can perform one or more related matrix operations on the first pair of matrices as a larger matrix W, respectively. A and D A , and then respectively perform matrix multiplication on matrices W1 and D1 as a portion (e.g., a slice thereof) of the larger matrix W B and G BThe result of the matrix multiplication of matrices W1 and D1 is used to calculate the larger matrix W A and D A The matrix multiplication result of matrices W2 and G2 is a partial result of the matrix multiplication used to calculate the larger matrix W2 and G2. The input matrices W1 and D1 and the input matrices W2 and G2 are stored in a pair of weight and data input arrays, such as arrays 321 and 323. In some embodiments, separate output accumulators 329 and 331 are used to accumulate intermediate and / or final results W1*D1 and intermediate and / or final results W2*G2, respectively. For example, output accumulator 329 is used to accumulate intermediate and / or final results of matrix multiplication associated with matrices W1 and D1, and output accumulator 331 is used to accumulate intermediate and / or final results of matrix multiplication associated with matrices W2 and G2.

[0046] In some embodiments, the size of the data input array and the weight input array are designed to fit the entire matrix in a linearized form. For example, a matrix processor capable of performing matrix multiplication on two matrices of size M×N and N×O has an input array of size M×N elements for receiving the corresponding M×N and N×O input matrices and another input array of size N×O elements. In some embodiments, the matrix processor performs calculations on two 8×8 matrices, and the size of the weight input array and the data input array are each designed to receive 64 elements. Similarly, the size of the output accumulator is designed to store the entire result matrix. The size of the output accumulator for storing the matrix multiplication results between two matrices of size M×N and N×O is designed to receive M×O elements. In some embodiments, the matrix processor performs calculations on two 8×8 matrices and stores the intermediate and final matrix results in an accumulator, the size of which is designed to fit 64 elements corresponding to the 8×8 result matrix.

[0047] In the example shown, the input array feeds the matrix computation unit 325. The matrix computation unit 325 is composed of a matrix of computation units (such as computation unit 327). Each computation unit is a processing element that can receive two operands, one element from each input matrix, and perform a computation such as a multiplication on the two input operands. In some embodiments, the computation is a multiplication and addition. For example, two input elements are multiplied, and the result is added to the current result in the accumulator and stored back to the accumulator. In some embodiments, each computation unit such as computation unit 327 includes an arithmetic logic unit for performing arithmetic logic operations such as multiplication, division, addition, or subtraction operations. In some embodiments, multiple operations can be performed in the same clock cycle, such as the multiplication and addition operations required to perform a partial dot product. Each computation unit may include an adder, a multiplier, and / or one or more accumulators corresponding to one or more pairs of data and weight input arrays. In some embodiments, each computation unit such as computation unit 327 includes a floating point multiplier and one or more accumulators. Although in Figure 3 Output accumulators 329 and 331 are depicted as being separate from computational unit 327, but in some embodiments, corresponding portions of output accumulators 329 and 331 are integrated into their respective computational units. For example, the accumulators of each computational unit together constitute output accumulators 329 and 331.

[0048] In various embodiments, the computational units of the matrix computational unit 325 support floating-point operations, such as floating-point multiplication and addition. In various embodiments, each computational unit includes a multiplier and one or more accumulators to perform multiplication and addition operations in a single cycle. Before each matrix calculation begins, the specified accumulator can be cleared. During the process of performing matrix calculations, the specified accumulator is used to accumulate and store intermediate results. In some embodiments, the matrix processor 313 is an 8×8 matrix processor, and the matrix computational unit 325 includes 64 computational units. Each cycle, 128 elements can be loaded into the matrix computational unit 325, with two input elements as operands for each of the 64 computational units. Each computational unit can also access the accumulator value stored in the specified accumulator.

[0049] In some embodiments, matrix multiplication requires multiple clock cycles to complete. For each clock cycle, a single row and a single column are retrieved from the input operand. For example, rows are retrieved from a matrix stored in a data input array, and columns are retrieved from a matrix stored in a weight input array. In some embodiments, data is retrieved by moving the data in the input array by an entire row or column. Each row and each column is a vector, and each vector is copied across the entire computation unit. Each row is copied "down" the row of the matrix computation unit 325, and each column is copied "across" the column of the matrix computation unit 325. For an 8×8 matrix processor, each column of the weight input matrix is ​​8 elements and each row of the data input matrix is ​​8 elements. For each pass, a single weight column is copied for each of the eight columns of the matrix computation unit 325, and a single data row is copied for each of the eight rows of the matrix computation unit 325. By crossing over and copying data to the next row and column once, an 8×8 matrix processor can complete matrix multiplication in 8 cycles. During each cycle, the intermediate results of multiplication and accumulation are stored in a designated accumulator. By the eighth and final cycle, the final matrix result is stored in the designated accumulator. Matrix processors using different dimensions (eg, 4×4 or 16×16 matrices) can be used with correspondingly sized input arrays, accumulators, and computational units.

[0050] In some embodiments, the input data elements are 8-bit floating point values. By utilizing 8-bit values, the bandwidth performance of the matrix processor is significantly improved. By utilizing configurable floating point values ​​and configurable biases, the precision and accuracy required for machine learning training are retained and the bandwidth is increased. Using the 8-bit format, a 64-byte × 64-byte matrix processor can calculate the matrix multiplication of two 8×8 matrices (a total of 128 elements). In contrast, using a 32-bit format, a 64-byte × 64-byte matrix processor can calculate the matrix multiplication of two 4×4 matrices (a total of only 32 elements). By optimizing the matrix elements using a configurable 8-bit floating point format, the bandwidth for loading matrix elements into the matrix processor is significantly improved. The power consumption per unit area is also significantly improved. To prevent overflow errors and underflow errors, the intermediate and final results stored in the designated accumulator utilize a larger bit format, such as 21-bit, 27-bit, or other appropriate floating point formats. Using 8-bit elements as input elements and using a 21-bit format to store intermediate results retains the precision and accuracy required for training while also maintaining the high input bandwidth of the matrix processor. In various embodiments, each output accumulator stores each element of the result matrix using a 21-bit floating point number, such as Figure 5Format 500. In some embodiments, the matrix processor 313 is an 8×8 matrix processor that uses 8-bit floating point input values ​​to perform matrix operations and uses 21-bit floating point values ​​to calculate intermediate and final matrix results. The input array is 64 bytes (64 8-bit elements) and the output accumulator is 168 bytes (64 21-bit elements). In various embodiments, the output accumulator is specified by a matrix calculation instruction. Similarly, the 8-bit floating point format and exponent bias can be configured by a matrix calculation instruction and / or one or more register parameters.

[0051] In some embodiments, the matrix processor 313 supports multiple different 8-bit floating point formats. For example, different formats 400 and 410 are supported and can be selected based on the computing task. Each format allocates a different number of bits to represent the exponent and mantissa of the floating point number. Depending on the use case, one format or another is selected. In the case where high precision numbers are required, more bits can be allocated to the mantissa and a format such as format 400 with more mantissa bits than format 410 is selected. A format with more mantissa bits can be selected to perform gradient descent, where very small deltas are required to retain accuracy. As another example, a format with more mantissa bits can be selected to perform forward propagation to calculate a cost function. As another optimization, each floating point format utilizes a configurable bias. The configurable bias is used to move the exponent range. For example, without an exponent bias, an exponent represented by 3 bits can specify 2 0 with 2 7 (inclusive). A bias of 5 shifts the range of exponents to have values ​​between 2 -5 with 2 +2 As another example, using 4 bits to represent the exponent, and a bias of 15 increases the range of the exponent from 2 0 and 2 31 (inclusive) move to between 2 -15 with 2 +16 In various embodiments, by optimizing the number of bits of the exponent field and the number of bits of the bias, the range of expression using the exponent and the digital coverage of the floating point number can be optimized to preserve the accuracy and precision of the expected input and results.

[0052] In some embodiments, the floating point format supports irregular numbers. For example, an exponent field with a zero value does not need a normalized mantissa without leading zeros. By supporting irregular numbers, the exponent range and the number of values ​​that can be represented can be increased. In various embodiments, each computing unit such as computing unit 327 includes support for performing floating point operations using one or more irregular operands.

[0053] In some embodiments, the value of the configurable bias is limited by the number of bits used to represent the configurable bias. For example, a 3-bit configurable bias can have eight different values ​​(0 to 7 (inclusive)). In some embodiments, as an optimization, the value represented by the configurable bias is not continuous. For example, the eight values ​​represented by the 3-bit configurable bias are not limited to values ​​0 to 7. On the contrary, the bias can be selected from 8 different values. For example, the configurable bias can be selected from eight predetermined values: 1, 3, 5, 7, 9, 11, 15 and 17. In some embodiments, the predetermined value is determined based on the most useful bias. The predetermined value can be selected at least in part to maximize the range and minimize the overlap between the ranges of different biases. In some embodiments, the configurable bias is specified by a matrix processor instruction and / or stored in a register (not shown). In some embodiments, the configurable bias is reconfigurable. For example, after performing an arithmetic operation, the configurable bias can be reconfigured to adjust to a new range of the result. In some embodiments, reconfiguration is specified as part of a calculation instruction. For example, an instruction can specify a new bias for reconfiguring the configurable bias.

[0054] In some embodiments, the computational units of the matrix computational unit can be grouped to also support matrix operations for larger input digital formats. For example, the computational units of each 8×8 matrix computational unit that operates with 8-bit floating-point matrix elements as input can be grouped to perform 4×4 matrix operations using 16-bit floating-point matrix elements as input. In some embodiments, the size of the output accumulator is designed to prevent the loss of accuracy of the quantized result. For example, a 16-bit floating-point format using a unit for a sign bit, 8 bits for an exponent, 7 bits for a mantissa, and a non-configurable exponent bias utilizes a 27-bit intermediate floating-point format for a floating-point result. The 27-bit floating-point format can allocate a unit for a sign bit, 9 bits for an exponent, and 17 bits for a mantissa. The support for the grouped operation mode makes the matrix computation unit more versatile by supporting more operand formats.

[0055] In various embodiments, the matrix operation is performed by splitting the input operand into multiple components and each split component is provided to the different computing units of the group through the grouped operation mode. Each split component is expressed as a floating point number, and when added together, the different split components add up to the original operand. For example, the input operand is split into the most important bit (that is, the high component) and the least important bit (that is, the low component). In various embodiments, the exponent of the high component uses the same exponent value of the input operand, and the exponent of the low component is adjusted to consider deducting the most important bit from the input operand. In certain embodiments, the component of the least important bit is normalized. In certain embodiments, the computing unit supports irregular numbers and the component can be expressed as irregular numbers.

[0056] In various embodiments, when two input operands are multiplied using an operand number format (e.g., 16-bit floating-point operands instead of 8-bit floating-point operands) twice the size of the computing unit format, four computing units are grouped together, and each input operand has a corresponding high component and a low component. By pairing high-high components, high-low components, low-high components, and low-low components and providing different pairs to different computing units of the group, the high and low components of each input operand are provided to the processing element. At each computing unit of the group, matrix multiplication is performed and the result is stored in the output accumulator associated with the computing unit. In some embodiments, the output accumulator utilizes a floating-point format with a higher number of bits than the original input operand. For example, for a 16-bit input operand without a configurable exponent bias, the output accumulator can utilize 27 bits. When the output results of the grouped units are added together, the result is a matrix multiplication of the original input operands. In some embodiments, the result is moved out of the matrix computing unit and added together using a post-processing unit such as a vector computing unit. For example, a floating-point addition instruction is used to add component results to determine the multiplication result. The floating point vector addition instruction may be used to add the components of the resulting vector. In various embodiments, the matrix computation unit is Figure 3 The matrix calculation unit 325 and the post-processing unit is Figure 3 The post-processing unit 317.

[0057] In some embodiments, the node engine 300 includes a plurality of matrix processors 313 and 351 to 357. The functions and components of the matrix processors 351 to 357 are described with respect to the matrix processor 313. In some embodiments, each matrix processor requires at least a minimum number of cycles to complete matrix multiplication, for example, an 8×8 matrix processor requires eight cycles. By incorporating multiple matrix processors in a single node engine, matrix multiplication can be distributed to different matrix processors. The result output can be interleaved to read matrix results from different matrix processors in each cycle. For a set of 8 8×8 matrix processors, each matrix processor can output matrix results every 8 cycles. Interleaved processors allow matrix results for each clock cycle from different processors. In some embodiments, matrix processors of different sizes can be used, for example, a 4×4 processor or a 16×16 processor. Similarly, based on the depth of the matrix processor calculation pipeline, different numbers of matrix processors can be included in the node engine.

[0058] In some embodiments, the matrix instruction specifies a specific matrix operation, a specific matrix processor, specifies an accumulator for storing matrix results, and specifies the location of the matrix operand. The location of the matrix operand can be specified using a register value or a memory address. For example, the matrix instruction can specify a matrix multiplication, a matrix multiplication processor 313, an output accumulator 329, a register of a post-processing unit register file 307, and a memory address of a memory 303. In some embodiments, the control unit 301 issues a matrix instruction. In some embodiments, the operation includes matrix multiplication, matrix addition, dot product, matrix inversion, etc. In some configurations, the output accumulator of each matrix processor uniquely identifies the matrix processor. By specifying a specific output accumulator as a part of the matrix instruction, the matrix processor is selected in an intrinsic manner. For example, using the A0-A11 naming scheme for the accumulators, the first and second output accumulators (e.g., A0 and A1) are mapped to the matrix processor 313, the third and fourth output accumulators (e.g., A2 and A3) are mapped to the matrix processor 351, the fifth and sixth output accumulators (e.g., A4 and A5) are mapped to the matrix processor 352, etc. In this example, accumulators 329 and 331 are referred to as A0 and A1, respectively. Since only the matrix processor 313 can store the result to the accumulator A1, the matrix multiplication instruction specifying the accumulator A1 is issued to the matrix processor 313.

[0059] In some embodiments, output array 315 is used to retrieve the result of one or more matrix processors. In some embodiments, output array 315 includes a multiplexer to determine which matrix processor to load the result into the output array. In some embodiments, the output array is a 64-byte array and two move instructions are required to move the matrix result from the matrix processor to the output array. For example, the matrix result using 21-bit floating point values ​​requires 168 bytes. Each 21-bit floating point value is converted to a 16-bit floating point value during the move command. Only using two move instructions, the result matrix of 64 elements is converted from 64 21-bit floating point values ​​to 64 16-bit floating point values. For example, the up-shift instruction moves the highest 32 elements into the output array, and the down-shift instruction moves the remaining lowest 32 elements into the output array. In various embodiments, the output array is 64 bytes, so that before performing the second move, the result of the first move is first stored in a register (such as a register of the post-processing unit register file 307). In various embodiments, the output array is a temporary output array until the value is moved to a memory or a register. In some embodiments, the move instructions are non-destructive and do not clear the matrix results from the matrix processor, for example, by clearing the source accumulator.

[0060] In some embodiments, the post-processing unit 317 is used to perform post-processing such as normalization, expansion, activation function, pooling, etc. In some embodiments, the post-processing unit 317 is a vector calculation engine that operates on each element of the vector. The post-processing unit can utilize different digital formats including floating point formats, such as 1-byte digital format, 2-byte digital format, and 4-byte digital format. In some embodiments, the number of channels of the post-processing unit 317 can be configured. For example, the post-processing unit 317 using a 64-byte vector can operate on 64 1-byte elements, 32 2-byte elements, or 16 4-byte elements corresponding to 64 channel configurations, 32 channel configurations, and 16 channel configurations. In the example shown, the post-processing unit 317 utilizes the post-processing unit register file 307 to retrieve data for input and to store post-processing results. In some embodiments, additional post-processing units (not shown) can be included in the node engine as needed to perform additional machine learning functions.

[0061] Figure 4 4 is a block diagram illustrating an embodiment of an 8-bit floating point format. In the example shown, 8-bit floating point formats 400 and 410 are different 8-bit floating point formats for representing floating point numbers using a sign, a mantissa, and an exponent. In some embodiments, a node engine such as node engine 300 and a node engine such as Figure 3 Matrix processors such as the matrix processor 313 of the present invention utilize 8-bit floating point formats 400 and 410 for matrix operations. By performing matrix operations using 8-bit floating point formats such as formats 400 and 410 rather than 16-bit floating point formats, 32-bit floating point formats, or other floating point formats, the bandwidth of the matrix processor is significantly increased. In some embodiments, formats 400 and 410 support configurable bias. Configurable bias enables greater range when representing exponents to improve accuracy while still maintaining 8-bit data size. In some embodiments, floating point formats 400 and 410 support irregular numbers to increase the number of values ​​that can be represented.

[0062] In the example shown, the 8-bit floating point format 400 includes a unit for a sign bit 401, 4 bits for an exponent 403, and 3 bits for a mantissa 405. The sign bit 401, the exponent 403, and the mantissa 405 occupy a total of 8 bits and can be used to represent a floating point number. Similarly, the 8-bit floating point format 410 includes a unit for a sign bit 411, 5 bits for an exponent 413, and 2 bits for a mantissa 415. The sign bit 411, the exponent 413, and the mantissa 415 occupy a total of 8 bits and can be used to represent a floating point number. In some embodiments, a configurable bias is used to bias the exponent. For example, the 4-bit exponent 403 of format 400 allows the exponent 403 to have 16 different values ​​(i.e., values ​​0 to 15 (inclusive)). Using 4 bits without a bias (or equivalently, the configurable bias is set to zero), the exponent 403 can represent a value 2 corresponding to an exponent field having values ​​0 and 15, respectively. 0 to 2 15 By using a configurable bias, the range of the index can be shifted. For example, with a configurable bias set to a value of 5, an index of 403 can represent a value of 2. -5 to 2 10 exponent. In various embodiments, the value of the configurable bias is limited by the number of bits used to represent the configurable bias. For example, a 3-bit configurable bias can have eight different values. In some embodiments, the value represented by the configurable bias is not continuous. For example, the eight values ​​represented by the 3-bit configurable bias are not limited to values ​​0 to 7. On the contrary, the bias can be selected from 8 different values. For example, the configurable bias can be selected from eight predetermined values: 1, 3, 5, 7, 9, 11, 15 and 17. In some embodiments, the predetermined value is determined based on the most useful bias. In some embodiments, the predetermined value is at least partially selected to maximize the range of the exponent and minimize the overlap between the ranges of different biases. In some embodiments, the configurable bias is specified by a matrix processor instruction and / or stored in a register (not shown).

[0063] In various embodiments, the matrix processor supports a variety of different 8-bit floating point formats, such as formats 400 and 410. By supporting a variety of formats, precision can be utilized in the exponent or mantissa. For example, certain operations such as gradient descent may require additional precision, and thus the mantissa requires a larger number of bits. As another example, more bits can be used for the mantissa for clustering values ​​together and operations without additional exponent ranges. In contrast, for some operations, the range of values ​​may be larger, and a larger exponent range is required. Using format 410, there are fewer bits dedicated to the mantissa, and more bits dedicated to the exponent. In some embodiments, the format is specified by the matrix processor instruction and can be stored in a register (not shown). In various embodiments, additional floating point formats that are not depicted can be supported. For example, a 4-bit mantissa format and a 3-bit exponent format (not shown) can be supported.

[0064] Figure 5 5 is a block diagram illustrating an embodiment of a 21-bit floating point format. In the example shown, floating point format 500 is a 21-bit floating point format for representing floating point numbers using a sign, a mantissa, and an exponent. In some embodiments, a node engine such as node engine 300 and a node engine such as Figure 3 Matrix processors such as matrix processor 313 of Utility utilize 21-bit floating point formats such as format 500 for certain matrix operations, such as for storing results (and intermediate results) of matrix multiplication and / or matrix addition. In some embodiments, format 500 is composed of a 21-bit floating point format such as format 500. Figure 3 The output accumulator 329 and 331 of the matrix processor are used. For example, if the result is limited to the same 8-bit format, the multiplication result of two 8-bit multiplication operands may cause overflow error or underflow error. The result is prevented from overflow error and underflow error by using a format greater than 8 bits. Similarly, when using 8-bit matrix elements to calculate matrix multiplication, 21-bit floating point formats are used to store intermediate and final results to prevent overflow error or underflow error. The result of using a bit depth less than 32 bits improves memory usage efficiency. In various embodiments, a format 500 with a bit depth of 21 bits is used to optimize both memory usage and accuracy. In some embodiments, format 500 supports configurable bias. Configurable bias allows a larger range to improve accuracy while still maintaining 21-bit data size. In some embodiments, configurable bias is specified by a matrix processor instruction and / or stored in a register (not shown).

[0065] In the example shown, the 21-bit floating point format 500 includes a single bit for a sign bit 501, 7 bits for an exponent 503, and 13 bits for a mantissa 505. The sign bit 501, the exponent 503, and the mantissa 505 take up a total of 21 bits and can be used to represent a floating point number. In some embodiments, a configurable bias is used to bias the exponent. For example, the 7-bit exponent 503 of the format 500 allows the exponent 503 to have 128 different values ​​(i.e., values ​​0 to 127 (inclusive)). Using 7 bits without a bias (or equivalently, the configurable bias is set to zero), the exponent 503 can represent a floating point number with a value of 2 0 to 2 127 , which corresponds to the exponent field having the values ​​0 and 127, respectively.

[0066] In various embodiments, format 500 is used by a user such as Figure 3 The matrix processors of the node engines such as the node engine 300 and the matrix processor 313 are such as Figure 3The output accumulators 329 and 331 of one or more accumulators are used. In some embodiments, registers (not shown) are used to store settings for configurable biases for storing floating point numbers in specific accumulators. In some embodiments, multiple 21-bit formats (e.g., different bit allocations for exponent fields and mantissa fields) can be used and the specific format is specified by a matrix processor instruction. The value of the configurable bias can be specified and / or stored in a register using a matrix processor instruction.

[0067] although Figure 5 Describes the Figure 3 The 21-bit floating point format used by the accumulator of the matrix processor such as the output accumulator 329 and 331 of , but the format with an alternative bit depth can be used. For example, according to the operation requirements, such as the requirements for preventing the loss of accuracy, when supporting the operation of some 16-bit floating point operations, the 27-bit floating point format can be used to prevent the loss of accuracy of the quantization result. As an example, the 27-bit floating point format can include a sign bit for the unit, 9 bits for the exponent, and 17 bits for the mantissa. The 27-bit floating point format can be used for accumulating the multiplication operation of 16-bit floating point operands. In some embodiments, the 16-bit floating point operands are represented by the unit for the sign bit, 8 bits for the exponent, and 7 bits for the exponent.

[0068] Figure 6 is a flow chart illustrating an embodiment of a process for performing matrix calculations. Figure 6 The process is performed by Figure 2 The training platform 223 is used to train the Figure 2 Node Engine 225 or Figure 3 One or more node engines, such as node engine 300 of the training platform, perform matrix calculations. In some embodiments, the training platform receives one or more matrix calculation operations and parallelizes these operations across different node engines. Each node engine can then also parallelize its operations across different matrix processors. As appropriate, the results can be combined at one or more node engines to determine a result, such as a weight matrix for a machine learning model. In some embodiments, Figure 6 The process as Figure 1 The step 105 is performed as part of the step 105.

[0069] At 601, a computation instruction is received. In some embodiments, the computation instruction is provided by, for example, Figure 2The training platform 223 or the like receives the computing instructions. The training platform processes the computing instructions and performs the necessary work division and allocation to different node engines. For example, a computing instruction requesting to convolve an image with a filter is received at a server of a training platform that starts a machine learning training process. In some embodiments, the instruction may include parameters required to execute the computing instruction, including the operations and operands involved. For example, the instruction may include the size of the input operand (e.g., the size of each input matrix), the starting address of each input matrix, the stride parameter, the fill parameter, and / or the matrix, vector and / or post-processing command. For example, the computing instruction may describe the image data size (e.g., 96×96, 1920×1080, etc.) and the bit depth (e.g., 8 bits, 16 bits, etc.) as well as the filter size and bit depth, etc. In many scenarios, the matrix of the matrix calculation may be larger than the matrix that can be adapted inside the matrix processor, so that additional processing can be performed to subdivide the calculation so that it can be performed by different node engines or matrix processors.

[0070] At 603, matrix operations and operands are determined. In the event that one or more matrices of the computation instruction received at 601 are larger than the input matrix of the matrix processor, the computation instruction of 601 is split into small component operations. At 603, matrix operations and operands corresponding to the smaller component operations are determined, and may include slicing, splitting or dividing the original matrix operands into smaller matrices and performing matrix operations on the smaller matrices. The results of the matrix operations on the smaller matrices may be combined to complete the computation instruction received at 601. Different node engines and matrix processors may be assigned to perform different components of the computation instruction. In some embodiments, the elements of the matrix operands may be converted to or targeted for conversion to an 8-bit floating point format. The node engine uses a processor such as Figure 4 8-bit floating point formats such as format 400 or format 410 are used to increase processing and performance bandwidth and power efficiency of matrix processors. In some embodiments, a configurable bias for a corresponding floating point format is selected or will be selected. For example, a format with a high precision mantissa is selected to perform gradient descent operations.

[0071] In various embodiments, a larger matrix is ​​sliced ​​into smaller two-dimensional matrices, where the size is limited to the appropriate dimensions of the matrix processor. In some embodiments, a sliced ​​matrix is ​​a smaller matrix with an address pointing to an element of a referenced original matrix. The sliced ​​matrix can be serialized as a vector for processing. In some embodiments, different slices of a matrix can overlap with previous slices. In various embodiments, a matrix can be sliced ​​only at a boundary corresponding to a multiple of the read buffer size. For example, in the case where the size of each read buffer is 8 bytes, each row of the sliced ​​matrix must start with an address that is a multiple of 8. In the case where a matrix is ​​adapted to a calculation array, no slicing is required (that is, the matrix slices used are just the original matrix).

[0072] At 605, matrix operations are allocated and performed. For example, the matrix operations corresponding to the matrix operations and operands determined at 603 are allocated to one or more node engines and one or more matrix processors of the node engines. In various embodiments, the matrix operations are performed by one or more matrix processors using 8-bit element matrices. The values ​​of the elements of the matrix results are accumulated as 21-bit floating point formats, 27-bit floating point formats or other suitable floating point formats. In various embodiments, the matrix results can be shifted out of the matrix processor in a format including several formats of 8-bit floating point formats, 16-bit floating point formats and 32-bit floating point formats. In various embodiments, each node engine can perform multiple matrix operations in parallel by utilizing multiple matrix processors.

[0073] In some embodiments, a reference to a matrix operand is assigned along with the operation to the node engine. In this way, the node engine can perform a data read to load the corresponding elements of the sliced ​​matrix. In some embodiments, the node engine linearizes the sliced ​​matrix for loading into a memory and / or register, where the input matrix can then be sent to the matrix processor. In some embodiments, the control unit of the node engine coordinates the scheduling, issuance, and synchronization of operations, including: loading the sliced ​​matrix operands (including the specified stride, padding, and other parameters for addressing the matrix operands); and operating the matrix processor. Once a matrix operation is issued to the matrix processor, the matrix processor spends a certain number of clock cycles to complete the matrix operation. In some embodiments, the matrix processor uses Figure 7 and / or Figure 8 The process of performing matrix operations.

[0074] At 607, post-processing is performed. In some embodiments, post-processing can be performed by a node engine and can include additional vector operations performed after completing matrix operations. Post-processing operations can be performed by a post-processing unit such as a vector processor or a vector calculation unit of a node engine. In some embodiments, vector post-processing includes: performing complex operations, such as arithmetic operations, expansion, normalization, and / or applying activation functions such as linear rectifier unit (ReLU) functions to each element of the vector. In some embodiments, depending on the required accuracy, the elements of the vector can be converted / formatted into 8-bit elements, 16-bit elements, or 32-bit elements. In various embodiments, the results of the distributed matrix operations of each node engine can be sent back to the training platform server or redirected by the training platform server and used for other processing. For example, the results of the matrix operations assigned and performed at 605 can be combined and used as operands for additional vector or matrix operations. After starting post-processing at 607, the processing loop returns to 601 to receive additional calculation instructions. In some embodiments, post-processing does not need to be completed before the processing loop returns to 601 to obtain additional calculation instructions.

[0075] Figure 7 is a flow chart illustrating an embodiment of a process for performing matrix calculations. Figure 7 The process is such as Figure 3 The matrix processors 313 and 351 to 357 of the node engine 300 are used to perform matrix calculations. In some embodiments, each matrix processor of the node engine can execute in parallel Figure 7 For example, matrix processors 313 and 351 to 357 each perform parallel operations on different matrix parameters. Figure 7 The process is used to perform convolution using a data matrix and a weight matrix, although each matrix processor may be at a different processing step to stagger the completion of their respective operations. In some embodiments, the process is used to perform convolution using a data matrix and a weight matrix. In some scenarios, the input matrix is ​​a slice of a larger matrix. In various embodiments, Figure 7 The process may be initiated by a matrix calculation instruction via the control unit. The instruction may specify two matrix operands (e.g., memory or register locations for data and weight matrices), a configurable bias, a floating point format, and a specified accumulator to store the matrix calculation result. In some embodiments, the specified accumulator is reset to zero before the matrix calculation begins. In some embodiments, the specified accumulator is Figure 3 The output of accumulator 329 or 331. In some embodiments, Figure 6 605 executions Figure 7 process.

[0076] At 701, a data input matrix is ​​received. For example, the elements of the data input matrix corresponding to the training sensor data are linearized and stored in a data input array of a matrix processor. In some embodiments, the data input matrix is ​​stored in a matrix such as Figure 3 313 of the matrix processor 313. Each data input array is capable of storing an entire linearized matrix for processing by the matrix computation unit of the corresponding matrix processor. Thus, a matrix processor capable of multiplying two 8×8 matrices uses a data input array capable of storing all 64 elements of the input 8×8 data matrix. For example, in some embodiments, each data input array is 64 bytes and stores each element as an 8-bit floating point number. The format of the floating point number can be used Figure 4The format 400 or 410 of , and includes configurable bias. The configurable bias can be specified by a matrix instruction and / or a register. The received data input matrix can be received from a register or from a memory such as an SRAM. In some embodiments, one or more reads are issued to load the entire data input matrix to the matrix processor, but the entire matrix is ​​not immediately available. For example, for a sliced ​​matrix, the data of some rows (or columns) may require additional delays before the data is available. Therefore, the data of the data input array may arrive one by one. In some embodiments, a single read is sufficient to load the entire data input matrix. In some embodiments, the data input matrix is ​​a gradient input matrix.

[0077] At 703, a weight input matrix is ​​received. For example, the elements of the weight input matrix corresponding to the machine learning weights of the filter are linearized and stored in a weight input array of a matrix processor. In some embodiments, the weight input matrix is ​​stored in a matrix such as Figure 3 The weight input arrays 323 of the matrix processor 313 of the embodiment of the present invention are stored in weight input arrays such as the weight input array 323 of the matrix processor 313 of the embodiment of the present invention. Each weight input array is capable of storing an entire linearization matrix for processing by the matrix computation unit of the corresponding matrix processor. Thus, a matrix processor capable of multiplying two 8×8 matrices uses a weight input array capable of storing all 64 elements of the input 8×8 weight matrix. For example, in some embodiments, each weight input array is 64 bytes and each element is stored as an 8-bit floating point number. The format of the floating point number can be used Figure 4 400 or 410, and includes configurable biases. The configurable biases can be specified by matrix instructions and / or registers. The received weight input matrix can be received from a register or from a memory such as an SRAM. In some embodiments, one or more reads are issued to load the entire weight input matrix to the matrix processor, but the entire matrix is ​​not immediately available. For example, for a sliced ​​matrix, some rows (or columns) of weight data may require additional delays before the weight data is available. Therefore, the weight data of the weight input array may arrive one by one. In some embodiments, a single read is sufficient to load the entire weight input matrix. In some embodiments, the weight input matrix is ​​a gradient input matrix.

[0078] At 705, a pair of vector parameters are loaded into the matrix calculation unit. From each input matrix, a vector corresponding to a row and a vector corresponding to a column are loaded as input parameters into a matrix calculation unit such as Figure 3A matrix computation unit such as the matrix computation unit 325 of . As part of the loading process, the column vector is copied across the entire matrix computation unit, and the row vector is copied across the entire matrix computation unit. For example, the entire vector corresponding to a column of the weight input matrix is ​​loaded into the computation unit. Each element of the column vector is copied across the entire row. Therefore, each column of the 8×8 matrix computation unit receives the same 8-element column vector, and the value of each cell loaded into a row of the matrix computation unit is the same. Similarly, the entire vector corresponding to a row of the data input matrix is ​​loaded into the computation unit, and each element of the row vector is copied across the entire column. Therefore, each row of the 8×8 matrix computation unit receives the same 8-element column vector, and the value of each cell loaded into a column of the matrix computation unit is the same. For the 8×8 matrix computation unit, one eighth of the input matrix elements are loaded. At 705, the unloaded vector pairs from each input matrix are loaded into the matrix computation unit. The next available column and row are loaded from the input weight and data matrices through each subsequent cycle of step 705. Therefore, an 8×8 matrix requires at least 8 cycles to complete the load, while a 4×4 matrix requires at least 4 cycles to complete the load.

[0079] At 707, the values ​​of the loaded vectors are multiplied. Figure 3 Each computation unit of a matrix computation unit, such as computation unit 327 of , performs matrix multiplication using the elements loaded at the corresponding computation unit. In various embodiments, the multiplication is performed on two 8-bit floating point values ​​and stored as a high floating point value to prevent overflow and maintain accuracy. In some embodiments, the high floating point format is Figure 5 In some embodiments, the high-order floating point format is a 27-bit floating point format to further reduce the loss of accuracy of the quantization result. For an 8×8 matrix calculation unit, each of the 64 calculation units performs matrix multiplication.

[0080] At 709, the multiplication results are accumulated into the designated accumulators. For example, the multiplication results of each computing unit at 707 are each accumulated into one of the accumulators of the matrix processor. In some embodiments, the matrix processor includes a matrix processor such as Figure 3There is more than one accumulator such as the two output accumulators 329 and 331 of . This is beneficial so that the matrix processor can interleave operations of different matrix operations. In some embodiments, each computation unit includes an accumulator that adds the current value of the element in the accumulator corresponding to the computation unit to the result of the matrix multiplication of the unit. In various embodiments, the size of the accumulator is designed to store the accumulated result of each element of the matrix. Therefore, each accumulator of the 8×8 matrix computation unit has at least 64 elements. In some embodiments, similar to the multiplication result at 707, the elements of the accumulator use higher-order floating point values ​​than the input to the matrix processor to prevent overflow and maintain precision. In some embodiments, the high-order floating point format is Figure 5 In some embodiments, the accumulator for the 8×8 matrix computation unit is 168 bytes to allow 64 elements, each element storing a 21-bit floating point number.

[0081] At 711, it is determined whether additional vectors remain for the matrix operation. For example, to multiply two matrices, at most one column from the weight input matrix and one row from the data input matrix are loaded per clock cycle. In order to complete the entire matrix multiplication, each column and each row must be loaded. An 8×8 matrix requires at least 8 cycles to fully load the two input matrices into the matrix calculation unit. Similarly, a 4×4 matrix requires at least 4 cycles to fully load the two input matrices into the matrix calculation unit. In the event that there are additional vectors remaining to be loaded, processing continues back to 705. In the event that there are no additional vectors remaining to be loaded (both entire input matrices have been loaded), the matrix multiplication is completed and processing continues to 713.

[0082] At 713, the matrix result is loaded from the specified accumulator into the output array. Since the matrix calculation is complete, the matrix result is stored in the specified accumulator. In some embodiments, the elements of the matrix are stored in the specified accumulator as 21-bit floating point values. Therefore, for an 8×8 matrix, the accumulator stores 64 values ​​and is 168 bytes in size. In some embodiments, multiple move operations are required to move the result from the accumulator to the output array, such as Figure 3 The output array 315 of FIG. 1 is a 64-byte bus. In some embodiments, the width of the output array and the bus to the output array is 64 bytes. The accumulator result is converted from a 21-bit floating point value to a 16-bit floating point value that can be stored in two 64-byte components. Using an 8×8 result matrix as an example, two move operations are required to move the result from the accumulator of the matrix processor. For example, an up-shift operation is used to move the high bits of the accumulator (corresponding to the 32 elements of the matrix) into the 64-bit output array as a 16-bit floating point value. Once moved in the output array, the 32 elements can be stored in a 64-bit floating point array such as Figure 3The 21-bit floating point values ​​are converted to 16-bit floating point values ​​by converting the 21-bit floating point values ​​to 16-bit floating point values, and only two move operations are required. In some embodiments, these values ​​can be shifted out as 8-bit floating point values, 16-bit floating point values, or 32-bit floating point values. In the described example, the values ​​are shifted out as 16-bit values ​​for use by processors such as 16-bit floating point values. Figure 3 The output array is connected to the accumulator of each matrix processor of the node engine and acts as a multiplexer to receive the results of the move (e.g., up move instructions and down move instructions) from different matrix processors.

[0083] Figure 8 is a flow chart illustrating an embodiment of a process for performing multiple interleaved matrix calculations. Figure 8 The process is such as Figure 3 The matrix processors 313 and 351 to 357 of the node engine 300 are used to interleave multiple matrix calculations, such as two matrix multiplication operations. Each interleaved matrix calculation in the interleaved matrix calculation can be implemented using multiple intermediate matrix multiplications, wherein the result of the intermediate multiplication is used to calculate a larger matrix calculation. In order to improve processing bandwidth and efficiency, the result of each intermediate matrix multiplication is stored in the matrix processor and is not cleared when interleaving the alternative matrix operations. Different matrix operations can be different, and each matrix operation has a non-overlapping matrix operation number.

[0084] In some embodiments, each matrix processor of a node engine can process more than one matrix operation at a time, one matrix operation corresponding to each output accumulator of the matrix processor. In some embodiments, the ability to interleave multiple matrix operations allows matrix multiplication operations to be performed on very large matrices. Larger matrices are sliced ​​into smaller matrices that fit into the input array of the matrix processor, and the results of the matrix multiplications of the smaller matrices are combined. In various embodiments, for example, the ability to interleave multiple matrix operations increases the bandwidth and performance of the matrix processor by utilizing the matrix computation unit while waiting for a memory read to complete. Thus, when the input operands of a pending matrix operation of a first set of related matrix operations are not available (e.g., due to a delay in a memory read) but the input operands of a pending matrix operation of a second set of related matrix operations are available, the second set of related matrix operations can utilize the matrix computation unit. By utilizing multiple accumulators, the matrix computation unit can switch between multiple matrix computations by storing intermediate results in an accumulator dedicated to a particular set of related matrix operations. In some embodiments, the data input array is Figure 3 The data input array is 321, and the weight input array is Figure 3 The weight input array is 323, and the number of accumulators is Figure 3 The output accumulators 329 and 331. Although relative to Figure 3 The matrix processor 313 shows two accumulators, but additional accumulators may be included to allow interleaved additional matrix operations.

[0085] Figure 8 The process is Figure 7 A special variation of the process of , which utilizes multiple weight input array operands, multiple data input array operands, and multiple output accumulators to support interleaving two matrix multiplication operations. Figure 7 As described, Figure 8 The process is implemented in a similar way Figure 7 The steps include: loading column vectors across matrix computation units, loading row vectors down matrix computation units, multiplying operands by computation units, and accumulating multiplication results in designated accumulators; but taking care not to mix or erase intermediate results of two interleaved matrix operations. In some embodiments, Figure 6 605 executions Figure 8 process.

[0086] At 801, it is determined whether the matrix processor can receive additional matrix operation instructions. Figure 8In the example of , the matrix processor is able to interleave two matrix operations. Determine whether there are two matrix operations currently in the process being executed. In the case where the matrix processor can receive additional matrix operation instructions, the processing continues to 803. For example, the matrix processor can receive additional matrix operation instructions because it is only in the middle of processing a single matrix operation or is idle rather than processing any matrix operation. In the case where the matrix processor cannot receive additional matrix operation instructions, the processing loop returns to 801 until the matrix processor can be used to receive new matrix operation instructions. For example, the matrix processor is currently in the middle of processing two matrix operations and cannot receive another operation until at least one current operation in the current operation is completed. In some embodiments, a ready signal is issued to the control unit to signal the matrix processor to be ready to receive additional instructions.

[0087] At 803, a matrix processor receives a matrix instruction and issues a read request to a related matrix operation. For example, a matrix processor receives a matrix multiplication instruction with two operands corresponding to two input matrices. A read of the values ​​of the matrix operands is issued. These values ​​can be read from registers and / or memory. For example, a matrix parameter can specify an address in a register and / or memory. In some embodiments, a memory read may stop matrix calculations because a memory read may take multiple clock cycles to make data available. In some embodiments, multiple memory reads may be issued because matrices are not stored sequentially in memory. This may be the result of slicing a larger matrix into smaller matrix operands.

[0088] In some embodiments, the received instruction specifies a specific accumulator to store the matrix result. To interleave multiple matrix operations, each operation utilizes its own accumulator. The specified accumulator is used to store intermediate and final matrix results. In some embodiments, the specified accumulator uses a higher-order floating point format than the format used for input operands to store intermediate results. When the result is quantized, the high-order format minimizes the loss of accuracy.

[0089] In various embodiments, when data corresponding to the matrix operand is available, the values ​​are received and prepared for the matrix processor. In some embodiments, the matrix operand is too large for the matrix processor and multiple intermediate matrix operations are performed to complete the matrix instruction. In the event that data is not available, the matrix calculation unit may stop and idle. Instead of remaining idle, the second matrix operation can be performed as long as the data for the second operation is available.

[0090] At 803, processing continues to both 801 and 805. Processing loops back to 801 to fetch new instructions while also continuing to 805 to execute instructions received at 803. In various embodiments, fetching new instructions occurs in parallel with processing the current matrix operation. In some embodiments, the two processing branches to 801 and 805 are implemented using a pipeline-based approach.

[0091] At 805, it is determined whether the data is ready for the current matrix operation. For example, the elements to be loaded from the matrix operand of the current matrix operation must be available for loading into the computational unit of the matrix computation unit. In some embodiments, the data loaded into the matrix computation unit is a slice of the matrix operand sized for the input array of the matrix computation unit. For the weight input array, the pending columns of elements must be ready. For the data input array, the pending element rows must be ready. In the case where the elements of the weight columns and data rows of the current matrix operation are available, processing continues to 807. In the case where the pending elements of the current matrix operation are not available, processing continues to 813. For example, due to delays in memory reads and / or cache misses, the pending elements may not be available. Instead of stopping while waiting for data to become available, the matrix computation unit can potentially be used for alternative matrix operations.

[0092] At 807, the values ​​of the weight columns and data rows from the current matrix operation are loaded into the corresponding computational units, the computational operation is performed on the values, and the computational results are accumulated into the specified accumulators. In some embodiments, the computational operation is a multiplication operation similar to multiplying elements from two different matrices. In some embodiments, Figure 7 The process at 807 is described with reference to steps 701, 703, 705, 707, 709 and / or 711. For example, the value is loaded as an 8-bit floating point value with a configurable bias. The results of calculations such as multiplication and accumulation are stored in a first accumulator as a 21-bit floating point format. In some scenarios, additional configuration related to matrix operations is performed at 807, such as clearing the accumulator, determining the floating point format, and / or determining a configurable bias for the floating point format.

[0093] At 809, it is determined whether the matrix instruction for the current matrix operation is complete. In the event that the matrix instruction is complete, processing continues to 811. In the event that the matrix instruction is not complete, processing continues to 805, where it is determined whether additional data for the current matrix operation is ready to be loaded and processed by the matrix computation unit. Figure 7 Step 711 of 809 is described.

[0094] In some alternative embodiments (not shown), in the case where the matrix instruction is not completed, processing continues to 813, where it is determined whether an alternative matrix operation is pending and whether the data for the pending alternative matrix operation is ready to be loaded and processed by the matrix calculation unit. Under this alternative embodiment, instead of completing the current matrix operation, the matrix calculation unit continues to alternate back and forth between two different matrix operations as long as the data is available, as long as there are two concurrent matrix operations.

[0095] At 811, the matrix results stored in the designated accumulator are loaded into the output array. Because some embodiments store the resulting matrix using a higher bit depth floating point format such as a 21-bit floating point format or a 27-bit floating point format, multiple move instructions may be required to move the results out of the matrix processor. In some embodiments, the matrix results are moved into two 64-byte registers via the output array by first converting the matrix elements to 16-bit floating point values. In some embodiments, the matrix results are moved into two 64-byte registers via the output array. Figure 7 Step 713 of 811 is described. Processing loops back to step 805 where the matrix processor is ready to start a matrix operation or make progress on an alternative matrix operation if pending.

[0096] In some alternative embodiments (shown as dashed lines), processing continues to 813, where it is determined whether an alternative matrix operation is pending and whether data for the pending alternative matrix operation is ready to be loaded and processed by the matrix calculation unit. Under this alternative embodiment, once the current matrix instruction is completed, if there is an alternative matrix operation pending to be completed, the matrix calculation unit switches to the alternative matrix operation.

[0097] At 813, it is determined whether the alternative matrix operation is pending and whether the data for the pending alternative matrix operation is ready to be loaded and processed by the matrix calculation unit. For example, when the second matrix operation is received at 803 while processing the first matrix operation, the pending completed second matrix operation issues a read of its corresponding matrix parameter. Determine whether there is a pending second alternative matrix operation and whether its data is ready to be loaded into the matrix calculation unit. In the case where the operand data for the alternative matrix operation is available, the processing continues to 815. In some embodiments, the operand data is a slice of a larger operand matrix whose size is designed to be used for the input array of the matrix calculation unit. For the weight input array, the pending element column must be ready. For the data input array, the pending element row must be ready. In the case where there is no pending alternative matrix operation or the pending element of the alternative matrix operation is not available, the processing continues to 805. For example, due to delays and / or cache misses from memory reads, the pending element may be unavailable. Instead of stopping while waiting for data to become available, the availability of the data corresponding to the current matrix operation is checked again. The first matrix operation with available data loads its data into the matrix computation unit for processing.

[0098] At 815, a matrix processor including a matrix calculation unit is switched to perform processing on the pending completed alternative matrix operation. The alternative matrix operation is now designated as the current matrix operation, and the previous current matrix operation is designated as the alternative matrix operation. Since the first matrix operation may have stopped (or in some embodiments, completed), the matrix calculation unit is now committed to the pending completed second matrix operation. In various embodiments, as appropriate, the corresponding output accumulator is designated as the source of the previous intermediate result and the destination for accumulating the intermediate and final results. Processing continues to 807, where the newly designated current matrix operation is calculated to progress.

[0099] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways to implement the invention. The disclosed embodiments are illustrative rather than restrictive.

Claims

1. A microprocessor system comprising: a matrix computing unit, comprising one or more processing elements; as well as a control unit configured to provide a matrix processor instruction to the matrix computation unit, wherein the matrix processor instruction specifies a floating point operand formatted using an exponent, the exponent being biased using a specified bias, wherein the floating point operand is a matrix, The specified bias is configurable by the control unit using the matrix processor instruction, and the matrix processor instruction specifies a floating point format for the floating point operand from a floating point format group.

2. The system of claim 1, wherein each element of the matrix uses an 8-bit floating point format.

3. The system of claim 2, wherein the 8-bit floating point format allocates 1 bit for a sign bit, 4 bits for an exponent field, and 3 bits for a mantissa field.

4. The system of claim 2, wherein the 8-bit floating point format allocates 1 bit for a sign bit, 5 bits for an exponent field, and 2 bits for a mantissa field.

5. The system of claim 1, wherein each floating point format in the set of floating point formats utilizes the same total number of bits to represent a floating point number and utilizes a different number of bits for a mantissa field of the floating point number.

6. The system of any one of claims 1 to 5, wherein the specified bias is configured using a register parameter.

7. The system of any one of claims 1 to 5, wherein the specified bias is selected from a set of non-contiguous predetermined floating point exponent biases.

8. The system of any one of claims 1 to 5, wherein the specified bias is reconfigurable via the matrix processor instructions.

9. The system of any one of claims 1 to 5, wherein each of the one or more processing elements comprises a floating point multiplier and an accumulator.

10. The system of any one of claims 1 to 5, wherein each of the one or more processing elements is configured to perform floating point multiplication operations in parallel with other processing elements.

11. The system of any one of claims 1 to 5, wherein the matrix processor instruction specifies a designated accumulator for storing intermediate results of the matrix calculation unit.

12. The system according to any one of claims 1 to 5, further comprising: a post-processing unit, the control unit being configured to provide post-processing instructions to the post-processing unit; as well as a post-processing register file, wherein the post-processing instruction specifies a post-processing unit operand stored in the post-processing register file, The matrix calculation unit is configured to receive the matrix processor instruction.

13. The system of claim 12, wherein the post-processing unit is a vector calculation unit.

14. The system of claim 12, wherein the post-processing unit operand is a vector operand and the post-processing instruction specifies a data size for each vector element of the post-processing unit operand.

15. A method for matrix calculation, comprising: receiving a matrix processor instruction from a control unit, wherein the matrix processor instruction specifies a floating point operand formatted using an exponent, the exponent being biased using a specified bias, wherein the specified bias is configurable by the control unit using the matrix processor instruction; issuing one or more read instructions for a data value of the floating point operand; receiving the data value of the floating point operand; as well as One or more received data values ​​are loaded into a matrix computation unit, wherein the floating point operand is a matrix, and wherein the matrix processor instruction specifies a floating point format for the floating point operand from a floating point format group.

Images