Efficient selection of single instruction multiple data operations for neural processing units.

Abstract

A system and method for efficient selection of single instruction multiple data operations for a neural processing unit.An exemplary processor system includes a matrix processor configured to perform convolutions associated with a neural network, and a single instruction multiple data (SIMD) processor in communication with the matrix processor, the SIMD processor configured to execute an operation group based on a current location associated with processing of the neural network, the operation group being selected from a plurality of SIMD programs, and the operation group being selected from the SIMD programs responsive to the current location.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Application No. 63 / 364,451, entitled "EFFICIENT SELECTION OF SINGLE INSTRUCTION MULTIPLE DATA OPERATIONS FOR NEURAL PROCESSING UNITS," filed May 10, 2022, the disclosure of which is incorporated herein by reference.

[0002] The present disclosure relates to processing units, and more particularly to single instruction multiple data processing units. [Background technology]

[0003] Neural networks are essential for a variety of different applications and form the basis of modern technology. For example, neural networks can be used to perform object classification on images captured via a user device (e.g., a smartphone). In this example, the neural network may be a convolutional neural network that applies convolutional layers, pooling layers, and one or more fully connected layers to classify objects depicted in the images. As another example, neural networks may be used for translation of text between languages. In this example, the neural network may be a recurrent neural network.

[0004] With respect to a convolutional neural network, the network may be separated into convolutional layers, pooling layers, and the like. An exemplary convolutional layer may apply a large number of filters or kernels to the input data. For example, a first convolutional layer may perform a convolution between the image data and a filter or kernel. As another example, a subsequent convolutional layer may perform a convolution between a feature map (e.g., an output from a previous layer) and a filter or kernel. These layers may require a significant number of operations. For example, a convolutional layer may require millions or billions of multiplications and additions to process. [Brief description of the drawings]

[0005] [Figure 1A] 1 is a block diagram illustrating an exemplary processor system including a matrix processor and one or more single instruction multiple data (SIMD) processors.

[0006] [Figure 1B] 1 is a block diagram showing details of an exemplary SIMD program that may be selected for implementation by a SIMD processor.

[0007] [Diagram 2] FIG. 1 is a diagram of an example SIMD program selected during processing of a layer of a neural network.

[0008] [Diagram 3] 1 is a flowchart of an example process for processing a neural network using SIMD stanzas.

[0009] [Figure 4] 1 is a block diagram illustrating an example vehicle including a vehicle processor system.

[0010] DETAILED DESCRIPTION OF THE DRAWINGS The embodiments of the present disclosure and their advantages are best understood by referring to the following detailed description, in which like reference numerals are used to identify like elements shown in one or more of the figures, and the subject matter shown is intended to illustrate embodiments of the present disclosure and is not intended to limit the disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] This application describes techniques for efficiently executing operations or programs (e.g., groups of operations) according to a current location, processing step, or processing operation (collectively referred to herein as a location) associated with a neural network. As described below, these operations may be performed by one or more single instruction, multiple data (SIMD) processors that are connected to or can otherwise receive and provide information to a convolution engine or processor (e.g., matrix processor 102). For example, there may be 1, 2, 5, 17, etc. SIMD processors. In some embodiments, there may be 2, 5, 17, etc. processors. Operations may be associated with quantizing received data, normalizing (e.g., evenly redistributing data values ​​within the range of the bit size used for the operation), clearing state, loading constants, storing statistics associated with processing of the neural network or a portion thereof (e.g., a layer), etc.

[0012] As known to those skilled in the art, prediction is an exemplary technique for selectively performing operations depending on the state of a mask or predicate (e.g., a predicate register). For example, prediction may use a condition instruction associated with each predicate that is evaluated to determine whether it is true. In this example, the condition instruction is evaluated on every pass through a portion of code. Thus, typically, no operations may be performed for predicates that evaluate negatively. Thus, prediction may turn off instructions depending on the predicate, but typically may not skip these instructions to avoid the significant complexity associated with dynamically skipping instructions. Instead, computation cycles are consumed even for negatively evaluated predicates. Thus, prediction may degrade fetch performance due to skipped instructions.

[0013] Certain computational tasks, such as neural network processing, may have predictable processing paths that repeat during the operation of the computational task. For example, a convolutional neural network may process data using ordered layers in which operations are performed on input data, and then the data is routed to subsequent layers. In this example, a convolution engine or processor may be used to perform the computationally intensive convolutions. A SIMD processor may be used for other tasks, such as quantizing data as described above. Using predictions for operations performed by a SIMD processor may result in significant wasted computation cycles and increased energy usage. For example, different operations may be performed depending on a particular location in the neural network. In this example, the predictions may skip a significant number of operations until they reach an operation specific to a particular location.

[0014] Conversely, the present application describes efficient techniques for selecting an operation depending on the location in the neural network (e.g., process a layer, an output channel, a portion of an output channel, etc.). The location, also referred to herein as a stanza, may indicate an operation or group of operations to be performed by the SIMD processor. Advantageously, a program counter may be set based on the stanza (e.g., using a state machine). The program counter may thus be limited to operations between the start and end of the operation associated with the stanza. These operations may then be performed by the SIMD processor.

[0015] In this way, the techniques described herein can avoid the additional computation cycles used by prediction to evaluate predicates. Instead, due to the hierarchical structure of processing a neural network, a location (e.g., a stanza) may indicate a particular operation to be performed. For example, a program may be divided or separated into groups of operations that correspond to the natural nesting order of neural network processing (e.g., start of layer, start of output channel, end of output channel, end of layer, etc.). Thus, instruction fetching may begin and end at different points in the program, but by way of example, a contiguous set of instructions.

[0016] It should be noted that although neural networks and SIMD processors (e.g., neural processing units) associated with neural network processing are described herein, the techniques described herein are applicable to other processors, for example, any application-specific processor that employs software routines at predictable points in a data stream may employ the techniques of this disclosure. Exemplary Block Diagram

[0017] 1A is a block diagram illustrating an exemplary processor system 100 including a matrix processor 102 and one or more single instruction multiple data (SIMD) processors 110A-110N. In some embodiments, there are multiple SIMD processors 110A-110N. In some embodiments, the processor system 100 may be used to perform autonomous or semi-autonomous operation of a vehicle. For example, the processor system 100 may be provided in a vehicle (e.g., an electric vehicle) and perform autonomous or semi-autonomous operation using sensor data. The sensors may include, for example, image sensors, radar, ultrasonic sensors, Lidar, etc.

[0018] The matrix processor 102 may be used in some embodiments to perform convolutions associated with a convolutional neural network or transfer network. For example, the input data 104 may be convolved with the weight data 106. The matrix processor 102 may have multiple multiply-accumulate units to perform the convolution. As an example, the matrix processor may use input and weight data organized or formatted to facilitate larger convolution operations. For example, the input data 104 may be in the form of a three-dimensional matrix (e.g., two-dimensional data across multiple input channels). In this example, the output data may span multiple output channels. Thus, the matrix processor 102 may process larger input data by combining or flattening each two-dimensional output channel into a vector, such that an entire channel or a large portion of it can be processed by the matrix processor 102. As another example, data may be efficiently reused such that the weight data is shared across convolutions. For an output channel, the weight data 106 may be the weight data (e.g., kernel) used to compute that output channel.

[0019] In some embodiments, the techniques described herein employ the exemplary matrix processors described in U.S. Pat. No. 11,157,287, U.S. Patent Publication No. 2019 / 0026250, and U.S. Pat. No. 11,157,441, which are incorporated by reference in their entireties and form a part of this disclosure as if set forth herein.

[0020] Thus, in some embodiments, the matrix processor 102 may continue to compute the convolution for a particular output channel. For example, the matrix processor 102 may compute the convolution based on the weight data 106 and the input data 104, which may represent at least a portion of the convolution for a particular output channel. It may take millions or billions of operations to compute a particular output channel. In some embodiments, the portions of the operations associated with a particular output channel may be referred to as passes.

[0021] As described below, during the operation of the above-described convolutional neural network, the SIMD processors 110A-110N may perform different operations. For example, the SIMD processors 110A-110N may receive an output from the matrix processor 102. In this example, the output may indicate a processing result associated with the convolution of the input data 104 and the weight data 106. For example, the operation may include quantizing the processing result. In addition, the SIMD processor may determine statistics associated with the processing of a particular output channel or a layer for which a particular output channel has been determined. Thus, the SIMD processors 110A-110N may monitor statistics such as averages, minimums, maximums, etc. Similarly, the SIMD processors 110A-110N may determine statistical information for each output channel that has been determined for a layer.

[0022] As another example, the SIMD processors 110A-110N may provide information to the matrix processor. For example, the SIMD processors 110A-110N may read constants required for processing a particular layer. In this example, the constants may be read and used by the matrix processor or the SIMD processor. As another example, the SIMD processors 110A-110N may clear state associated with processing after a layer, output channel, or pass.

[0023] The SIMD processors 110A-110N may perform operations according to a current position (also referred to herein as a stanza) associated with the processing of the convolutional neural network described above. For example, the operations performed by the SIMD processors 110A-110N may be grouped. Thus, a compiler may divide a program including operations into chunks (e.g., individual programs) that may be executed separately by the SIMD processors 110A-110N during operation. As an example, during operation of the processor system 100, an operation group, or two or more operation groups, may be selected for execution by the SIMD processors 110A-110N. In some embodiments, a program counter may be used to restrict the execution of operations to an operation group according to a current stanza. As an example, the program counter may be limited to a start pointer and an end pointer such that an operation group (e.g., a particular program) is executed by the SIMD processors 110A-110N. Thus, the selection and execution may be controlled by hardware (e.g., a program counter, a hardware-based state machine) provided in the processor system 100.

[0024] In this manner, SIMD processors 110A-110N may perform particular operations depending on whether processing of a particular layer has begun, whether processing of a particular output channel has begun, whether a particular pass has begun, etc. This flexibility is possible without requiring computationally intensive prediction as described above. Thus, the techniques described herein limit the extent to which operations may be skipped and not performed.

[0025] The SIMD processors 110A-110N may further be in communication with a memory, not shown in FIG. 1A. For example, the memory may be an SRAM, and the SIMD processors 110A-110N may provide processing results for storage in the SRAM. In this example, the processing results may represent a quantized version of the convolution provided by the matrix processor 102.

[0026] FIG. 1B is a block diagram illustrating details of an example SIMD program 152 that may be selected for implementation by SIMD processors 110A-110N. As shown in FIG. 1A, SIMD processors 110A-110N may perform operations in response to a current stanza. For example, the current stanza may indicate a location associated with the processing of neural network 150. As one example, the location may indicate a layer is being processed. As another example, the location may indicate an output channel is being processed. As another example, the location may indicate a pass associated with the processing of an output channel is being processed. In some embodiments, the location may indicate, for example, a particular layer, a particular output channel, a particular pass, etc.

[0027] In the illustrated embodiment, SIMD program 152 is separated (e.g., grouped) into operations associated with layers, output channels, paths, etc. During operation of processor system 100, these different groups may be selected (e.g., via a program counter) and implemented by SIMD processors 110A-110N based on a current stanza. As shown, selected operations 154 from SIMD program 152 are provided to SIMD processors 110A-110N.

[0028] As shown, SIMD programs 152 may be organized according to normal (e.g., periodic) processing through neural network 150. For example, different SIMD programs 152 may be selected for execution (154) based on progress through neural network 150. In this example, a state machine may be used to select a SIMD program based, for example, on a program counter. Thus, a particular operation may be selected based on the layer (e.g., "layer A") being initiated. Similarly, once this layer is initiated, a particular output channel (e.g., "output channel A") may be initiated. Similarly, a first pass may be initiated. Once this first pass is completed, one or more remaining passes to determine the output channel may be performed. Optionally, particular operations may be performed by SIMD processors 110A-110N for the first pass and the remaining passes. Subsequently, a next output channel may be initiated, which may have the same or different operation as output channel A. As described herein, the operations may include, for example, quantizing data, determining statistics, etc.

[0029] Below are further examples of operations including exemplary pseudocode. Each example includes one or more operations that take place in different stanzas and form a larger operation.

[0030] For example, the operation may include an argument max operation, in which, for example, the maximum value (or the exponent of the maximum value) of a path of elements may be output. The element (e.g., new element) may be an input element (e.g., a single value) that is sent from the matrix processor to a SIMD lane (e.g., a SIMD processor) for post-processing. For example, pseudocode referencing a particular stanza is as follows: [Table 1]

[0031] As another example, the operation may include a per-channel reduction, which allows the SIMD processors 110A-110N to find the average value contained in the channel, for example, the value between the SIMD processors 110A-110N. For this operation, data may be moved between the SIMD processors to reduce it. For example, pseudocode is as follows: [Table 2]

[0032] As another example, the operations may include storing the input elements while optionally collecting and storing layer-wise average values. For example, pseudocode may include the following: [Table 3]

[0033] Thus, SIMD program 152 may provide any flexibility to the programmer to cause operations to be performed by SIMD processors 110A-110N. Note that in some embodiments, these operations may be single instructions that cause SIMD processors 110A-110N to each operate in lockstep. In some embodiments, SIMD processors 110A-110N may be grouped separately. For example, a subgroup of SIMD processors 110A-110N may execute a first SIMD program and a different subgroup may execute a second SIMD program. In this manner, there may be groups of SIMD processors that execute different operations.

[0034] 2 is a diagram of an example SIMD program selected during processing of a layer of a neural network. In FIG. 2, an example process flow 200 associated with processing of a neural network is shown. For example, in the process flow 200, different positions or stanzas associated with processing are identified. These positions may form a hierarchy of nested operations (e.g., start operation group A, start operation group B, stop operation group B, start operation group C, stop operation group C, stop operation group A).

[0035] In block 202, the location indicates the start of a layer. For example, the layer may be a convolutional layer in a neural network. As another example, the layer may be a different layer (e.g., a pooling layer). In block 204, the location indicates the start of an output channel. In some embodiments, as described above in FIG. 1A, the layer 202 may process based on the processed output channels (e.g., process the output channels separately). However, this processing is an example and the techniques described herein are not limited thereto.

[0036] In block 206, a location indicates the start of a pass for output channel 204. In the illustrated example, a SIMD program is grouped between the start of output channel 204 and the end of output channel 208. For example, in the first pass, the SIMD program may clear current state, read constants, etc. The "all" block may indicate an operation that is always performed when the location is reached. For example, the operation may include quantizing the data. For example, the first pass may be the first pass, and subsequent passes may not perform the group of operations of the first pass. However, they may perform the operations of the "all" block. Thus, the "all" block may be processed arbitrarily when any of blocks 202-208 is reached.

[0037] The last pass may indicate an operation to be performed upon completion of a pass. The last pass may also indicate a final pass associated with an output channel. For example, the operation may include determining statistics for an individual pass or pass associated with an output channel. Similarly, the completion of an output channel may be an operation associated with the completion of an output channel or an operation associated with the completion of all output channels of a layer. For example, the operation may include writing per channel statistics, such as average, maximum, minimum values ​​for each output channel or all channels (e.g., for a layer). Other operations are described in more detail above, e.g., in conjunction with pseudocode. Exemplary Flowchart

[0038] 3 is a flow chart of an example process 300 for processing a neural network using SIMD stanzas. For convenience, the process 300 is described as being performed by a system (e.g., processor system 100). In some embodiments, a program counter may identify a current stanza and cause an operation to be performed by a SIMD processor (e.g., SIMD processors 110A-110N).

[0039] In block 302, the system executes a neural network. As described herein, the neural network may be a convolutional neural network. For example, the neural network may be used for partially autonomous or semi-autonomous driving. In this example, the neural network may be used to identify objects and object characteristics (e.g., classification, speed, acceleration, location, etc.) around the vehicle.

[0040] At block 304, the system identifies a current location associated with the processing of the neural network. As discussed above, the current location may be a stanza associated with the neural network. For example, the location may include a layer, an output channel, a path, an end of one or more paths, an end of an output channel, an end of a layer, etc.

[0041] In some embodiments, a program counter may be used to identify the current location, where the program counter may indicate a current instruction (e.g., a pointer to the current instruction). Based on the current location, the program counter can identify an operation for the current location. For example, the program counter may be limited (e.g., by the processor) to a range of instructions that includes the identified operation (e.g., an operation group). As discussed above, program operations may be grouped. These groups may be associated (e.g., by a compiler) with respective start and end pointers. Thus, the program counter can be used to limit the execution of an operation group for the current location based on the start and end pointers.

[0042] At block 306, the system obtains output from the matrix processor. At certain locations, the SIMD processor may use the output from the matrix processor. For example, the SIMD processor may perform an operation to quantize the output. At other locations, the SIMD processor may not use or have not yet accessed the output from the matrix processor. As an example, the SIMD processor may load a constant (e.g., into the matrix processor or into the SIMD processor for use in quantization, determining statistics, etc.).

[0043] At block 308, the system executes the SIMD program associated with the processing location. A particular program (e.g., a group of operations) may be selected for execution by the SIMD processor, as described at block 304. In some operations, the SIMD processor may use output from the matrix processor (e.g., for quantization). Vehicle block diagram

[0044] 4 illustrates a block diagram of a vehicle 400 (e.g., vehicle 102). The vehicle 400 may include one or more electric motors 402 that move the vehicle 400. The electric motors 402 may include, for example, induction motors, permanent magnet motors, etc. Batteries 404 (e.g., one or more battery packs each including multiple batteries) may be used to power the electric motors 402, as known to those skilled in the art.

[0045] The vehicle 400 further includes a propulsion system 406 that can be used to set the gear (e.g., propulsion direction) for the vehicle. For an electric vehicle, the propulsion system 406 may adjust the operation of the electric motor 402 to change the propulsion direction.

[0046] In addition, the vehicle includes a processor system 100 having one or more single instruction multiple data processors (e.g., SIMD processors 110A-110N) as described herein. The processor system 100 may process data, such as images received from image sensors located around the vehicle 400. The processor system 100 may also output information to and receive information (e.g., user input) from a display 408 provided on the vehicle 400. Other embodiments

[0047] All of the processes described herein may be implemented and fully automated via software code modules executed by a computing system comprising one or more computers or processors. The code modules may be stored in any type of non-transitory computer readable medium or other computer storage device. Parts or all of the methods may be implemented in dedicated computer hardware.

[0048] Many other variations beyond those described herein will be apparent from this disclosure. For example, depending on the embodiment, certain operations, events, or functions of any of the algorithms described herein may be performed in a different order, or may be added, combined, or removed entirely (e.g., not all operations or events described may be required to implement an algorithm). Furthermore, in certain embodiments, operations or events may be performed simultaneously rather than sequentially, for example, via multi-threading, interrupt processing, or multiple processors or processor cores, or on other parallel structures. Additionally, different tasks or processes may be performed by different machines and / or computing systems that can function together.

[0049] The various example logic blocks, modules, and engines described in connection with the embodiments disclosed herein may be implemented or executed by a machine, such as a processing unit or processor, a DSP (digital signal processor), an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), or other programmable logic circuit, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The processor may be a microprocessor, but alternatively the processor may be a controller, a microcontroller, a state machine, or any combination thereof. The processor may include electrical circuitry configured to process computer-executable instructions. In another embodiment, the processor includes an FPGA or other programmable device that performs logical operations without processing computer-executable instructions. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. Although described herein primarily with respect to digital technology, the processor may include primarily analog elements. For example, some or all of the signal processing algorithms described herein may be implemented in analog circuitry or mixed analog-digital circuitry. The computing environment may include any type of computer system, including, but not limited to, computer systems based on microprocessors, mainframe computers, digital signal processors, handheld computing devices, device controllers, or computing engines within appliances, to name a few.

[0050] Conditional language such as "can," "could," "might," or "may," among others, is understood as typically used in context to convey that certain embodiments include certain features, elements, and / or steps, but other embodiments do not, unless otherwise noted. Thus, such conditional language is not typically intended to imply that the features, elements, and / or steps are somehow required by one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included in or performed in any particular embodiment, with or without user input or prompting.

[0051] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is understood in its normal context to indicate that an item, term, etc. can be either X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z), unless otherwise indicated. Thus, such disjunctive language is typically not, and should not be intended to imply that a particular embodiment requires that at least one of X, at least one of Y, or at least one of Z, each be present.

[0052] Any process descriptions, elements, or blocks in the flow diagrams described herein and / or shown in the accompanying figures should be understood as potentially representing modules, segments, or portions of code that include one or more executable instructions for implementing a particular logical function or element in the process. As will be appreciated by those skilled in the art, alternative implementations in which elements or functions may be omitted, performed, or described in a different order than that shown, including substantially simultaneously or in reverse order, depending on the functionality involved, are included within the scope of the embodiments described herein.

[0053] Unless otherwise noted, articles such as "a" or "an" should generally be construed to include one or more recited items. Thus, phrases such as "an apparatus configured to" are intended to include one or more of the recited apparatuses. Such one or more recited apparatuses may also be collectively configured to perform the recited contents. For example, "a processor configured to perform A, B, and C" may include a first processor configured to perform recited contents A working in conjunction with a second processor configured to perform recited contents B and C.

[0054] It should be understood that many different changes and modifications can be made to the above-described embodiments, and that the elements are understood to be in other preferred embodiments, and all such modifications and modifications are intended to be included herein within the scope of the present disclosure.

Claims

1. A matrix processor configured to perform convolutions related to neural networks, The system comprises one or more single-instruction multiple data (SIMD) processors that communicate with the matrix processor, The SIMD processor is configured to execute action groups based on the current position related to the processing of the neural network. A processor system in which the operation group is selected from a plurality of SIMD programs, the operation group is selected from the SIMD programs according to the current position, and for each SIMD processor, a first operation is performed based on a first position and a second operation is performed based on a second position.

2. The processor system according to claim 1, wherein the matrix processor is configured to perform convolutions related to the output layer of the convolutional layer included in the neural network.

3. The processor system according to claim 1, wherein the SIMD processor includes a plurality of SIMD processors.

4. The processor system according to claim 1, wherein the current position indicates one or more processes among layers, output channels, or paths, and the path indicates a processing chunk or part of a process related to an output channel.

5. The processor system according to claim 4, wherein the current position indicates the start or end of a layer, output channel, or path.

6. The processor system according to claim 1, wherein the operation group is selected using a program counter, the program counter is used to restrict execution between a start pointer and an end pointer, and the start pointer and the end pointer identify the operation group from the SIMD program.

7. The processor system according to claim 1, wherein the operation group is associated with quantization of the output from the matrix processor, and / or the operation group is associated with determining statistical information.

8. The processor system according to claim 1, wherein each of the plurality of SIMD programs is accessed in accordance with the processing flow of the neural network, and prediction is not used so as to not evaluate the individual operations included in the SIMD program that preceded the current position.

9. A method implemented by a processor system, Steps to run a neural network, The steps include identifying the current position related to the neural network, A method comprising the step of executing a SIMD program associated with the current location via one or more SIMD processors, wherein the SIMD program is selected from a plurality of SIMD programs, the SIMD program is selected from the SIMD programs according to the current location, and for each SIMD processor, a first operation is performed based on a first location and a second operation is performed based on a second location.

10. The method according to claim 9, wherein the SIMD program is associated with quantization of the output from a matrix processor, and / or the SIMD program is associated with determining statistical information.

11. The method according to claim 9, wherein the processor system is configured to perform convolutions related to the output layer of the convolutional layer included in the neural network.

12. The method according to claim 9, wherein the current position indicates one or more processes among layers, output channels, or paths, and the path indicates a processing chunk or part of a process related to an output channel.

13. The method according to claim 9, wherein the current position indicates the start or end of a layer, output channel, or path.

14. The method according to claim 9, wherein the SIMD program is selected using a program counter, the program counter is used to restrict execution between a start pointer and an end pointer, and the start pointer and the end pointer identify the SIMD program from the plurality of SIMD programs.

15. The method according to claim 9, wherein each of the plurality of SIMD programs is accessed in accordance with the processing flow of the neural network, and prediction is not used so as to not evaluate the individual operations included in the SIMD program that preceded the current position.

Images