Processing modules, circuits, methods, and integrated circuits for neural network models

Abstract

The application discloses a processing module, a circuit, a method, and an integrated circuit of a neural network model. The processing module comprises a first buffer and a systolic array. The first buffer is configured to obtain input data of a plurality of neural network models from a memory. The first buffer is further configured to transmit the input data of the plurality of neural network models to the systolic array. The systolic array is configured to start a plurality of pipelines, and the plurality of pipelines are run in parallel to facilitate processing of the input data of the plurality of neural network models. In this way, the plurality of neural network models are processed in parallel to improve processing efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a processing module, circuit, method, and integrated circuit for a neural network model. Background Technology

[0002] With the development of artificial intelligence technology, neural network models are widely used in scenarios such as intelligent driving (e.g., autonomous driving, assisted driving, and automatic driving), smart healthcare, and cloud computing. However, the processing of neural network models often involves handling multiple models simultaneously, which can lead to insufficient utilization of computing resources and thus affect processing efficiency. Summary of the Invention

[0003] This application provides a processing module, circuit, method, and integrated circuit for a neural network model to improve processing efficiency.

[0004] In a first aspect, a processing module for a neural network model is provided, comprising: a first buffer and a systolic array; the first buffer is configured to acquire input data of multiple neural network models from memory; the first buffer is also configured to transmit the input data of the multiple neural network models to the systolic array; the systolic array is configured to initiate multiple pipelines, enabling the multiple pipelines to run in parallel and process the input data of the multiple neural network models. The systolic array is used to achieve parallel operation of multiple pipelines, thereby processing the input data of multiple neural network models in parallel and improving processing efficiency.

[0005] In one implementation, multiple neural network models include a first neural network model comprising multiple layers. A first buffer is configured to retrieve input data from memory for the multiple layers. The first buffer is also configured to transfer the input data of the multiple layers to a systolic array. The systolic array is configured to initiate multiple pipelines, enabling the pipelines to run in parallel to process the input data of the multiple layers. The parallel processing of input data from multiple layers of the neural network model using a systolic array improves processing efficiency.

[0006] In one implementation, the above systolic array is configured to start multiple pipelines at different times, allowing the multiple pipelines to run in parallel. Alternatively, the above systolic array is configured to start multiple pipelines at the same time, allowing the multiple pipelines to run in parallel.

[0007] In one implementation, the processing module further includes a second buffer; multiple pipelines include a first pipeline and a second pipeline; the second buffer is configured to store the processing results of the multiple pipelines; a pulsation array is configured to start the first pipeline at a first moment; the pulsation array is also configured to start the second pipeline when the amount of data of the first processing result of the first pipeline in the second buffer reaches a first quantity, so that the first pipeline and the second pipeline run in parallel. Using the amount of data of the first processing result stored in the second buffer to trigger the start of the next pipeline achieves flexible control of multiple pipelines, and storing the first processing result in the second buffer reduces the storage capacity occupied by the first buffer.

[0008] In one implementation, the second buffer includes multiple sub-buffers, each including a first sub-buffer configured to store a first processing result. A pulsating array is configured to activate a second pipeline when the amount of data representing the first processing result in the first sub-buffer reaches a first quantity; the first quantity is less than or equal to the storage capacity of the first sub-buffer. Further dividing the second buffer into multiple sub-buffers and using the amount of data stored in the sub-buffers to trigger the execution of different pipelines allows for the parallel execution of multiple models, thereby improving processing efficiency.

[0009] In one implementation, multiple neural network models include a first neural network model, which includes multiple layers; a systolic array configured to process the first layer among the multiple layers and store the first processing result in a first sub-buffer; the first processing result includes the processing result of the first layer; the systolic array configured to store the first processing result in a first buffer when the amount of data of the first processing result in the first sub-buffer reaches the storage capacity of the first sub-buffer; and the systolic array configured to retrieve the first processing result from the first buffer and process the second layer among the multiple layers according to the first processing result.

[0010] The first processing result is stored in the first buffer to prevent the pulsating array from malfunctioning due to data overflow in the first sub-buffer, thereby improving reliability and further enhancing processing efficiency.

[0011] In one implementation, multiple neural network models include a first neural network model, which includes multiple layers, including a first layer and a second layer; a first pipeline includes multiple sub-pipelines, including a first sub-pipeline and a second sub-pipeline; a pulsating array is configured to start the second sub-pipeline when the processing results of the first sub-pipeline in the first sub-buffer reach a second quantity, so that the first sub-pipeline and the second sub-pipeline run in parallel; the first sub-pipeline processes the input data of the first layer when it runs, and the second sub-pipeline processes the input data of the second layer when it runs.

[0012] Based on the parallel operation of multiple models, inter-layer parallelism is implemented within the neural network model, thereby further reducing latency and saving the storage capacity occupied by the buffer, thus improving processing efficiency.

[0013] In one implementation, the systolic array includes multiple processing units, which are divided into multiple systolic subarrays based on the input data of multiple neural network models. Each systolic subarray includes at least two processing units, and the number of multiple systolic subarrays is the same as the number of multiple sub-buffers. The multiple systolic subarrays are configured to start multiple pipelines at different times.

[0014] Multiple processing units are allocated to multiple pipelines based on the input data of multiple neural network models, forming multiple systolic subarrays. Within each systolic array, the processing units are dynamically allocated. This dynamic allocation method enables the systolic array to run multiple pipelines in parallel, processing the input data of multiple models concurrently and improving processing efficiency.

[0015] In one implementation, the pulsating array further includes multiple third buffers connected to the outputs of multiple pulsating subarrays, and the multiple third buffers are configured to transmit the processing results of the multiple pulsating subarrays to the multiple subbuffers.

[0016] The processing results of multiple pulsating subarrays are transmitted to multiple sub-buffers to prevent the processing results from flowing between pulsating subarrays, thereby improving reliability.

[0017] In one implementation, multiple pipelines include a third pipeline and a fourth pipeline, and multiple pulsating subarrays include a first subarray configured to run the third pipeline; the first subarray is configured to run the fourth pipeline when the third pipeline finishes running.

[0018] When the pipeline running by a subarray in a pulsating array finishes running (i.e., when the subarray is idle), the idle pulsating subarray can be used to achieve cyclic parallelism, further improving the efficiency of parallel processing.

[0019] In a second aspect, a processing circuit for a neural network model is provided, comprising: a control module and a transmission module, and further comprising the processing module described in the first aspect above. The control module is configured to generate a first control instruction, a second control instruction, and a third control instruction; the transmission module is configured to retrieve data of multiple neural network models from a memory according to the first control instruction and store it in a first buffer; the processing module is configured to process the data of the multiple neural network models according to the second control instruction and generate processing results of the multiple neural network models; and transmit the processing results of the multiple neural network models to the first buffer; the transmission module is further configured to transmit the processing results of the multiple neural network models in the first buffer to a memory according to the third control instruction.

[0020] Thirdly, a method for controlling neural network model processing is provided, including:

[0021] Generate a first control instruction, which is configured to control the transmission module to retrieve data of multiple neural network models from the memory and store them in the first buffer of the first aspect and any one of the processing modules in the first aspect;

[0022] A second control instruction is generated, which is configured to control the processing module to process the data of multiple neural network models and generate the processing results of multiple neural network models; and to transmit the processing results of multiple neural network models to the first buffer.

[0023] A third control instruction is generated, which is configured to control the transmission module to transfer the processing results of multiple neural network models in the first buffer to the memory.

[0024] Fourthly, an integrated circuit is provided, including the processing circuit provided in the second aspect above.

[0025] Fifthly, a controller is provided, comprising the integrated circuit provided in the fourth aspect above.

[0026] Sixthly, a vehicle is provided, including the controller provided in the fifth aspect above.

[0027] Sixthly, a computer program product is provided, including instructions, wherein when the instructions are executed by a processor, the processing method provided in the third aspect above is executed. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A schematic diagram of the processing circuit of a neural network model provided in an embodiment of this application;

[0030] Figure 2 A schematic diagram of the arrangement of a pulsating array provided in an embodiment of this application;

[0031] Figure 3 A schematic diagram of another pulsating array arrangement provided in an embodiment of this application;

[0032] Figure 4 This is a schematic diagram of the structure of a pulsating array provided in an embodiment of this application;

[0033] Figure 5 This application provides a schematic diagram of a process for parallel operation of multiple pipelines.

[0034] Figure 6 A schematic diagram of the processing module of another neural network model provided in an embodiment of this application;

[0035] Figure 7 A schematic diagram illustrating another partitioning of the pulsating subarray provided in an embodiment of this application;

[0036] Figure 8 This is a schematic diagram of another pulsating array provided in an embodiment of this application;

[0037] Figure 9 This is a flowchart illustrating the method for controlling neural network model processing provided in an embodiment of this application. Detailed Implementation

[0038] To more clearly illustrate the technical solutions in the embodiments of this application, the specific implementation methods of this application will be described below with reference to the accompanying drawings. The accompanying drawings described below are merely some embodiments of this application. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without creative effort. Adjustments and improvements made without departing from the concept of this application are all within the protection scope of this application.

[0039] To keep the drawings simple, each figure only schematically shows the parts related to the corresponding embodiment, and they do not represent the actual structure of the product. In addition, for the sake of simplicity and ease of understanding, some figures only schematically show parts of components with the same structure or function, and there may actually be more or fewer components with the same structure or function.

[0040] In the embodiments of this application, unless otherwise expressly specified and limited, ordinal numbers, such as "first," "second," etc., are used only to distinguish and describe related objects, and should not be construed as indicating or implying the relative importance or order between related objects; furthermore, they do not represent the number of related objects. "Multiple" includes two or more, and other quantifiers are similar. " / " is used to describe the relationship between related objects, indicating an "or" relationship between related objects. "And / or" is used to describe the relationship between related objects, including any combination relationship between related objects, such as "a and / or b" including: "a alone," "b alone," or "a and b." "One or more" or "at least one" of multiple objects refers to any object or any combination of multiple objects, such as "one or more of a1, a2, a3" or "at least one of a1, a2, a3" including: "a1 alone," "a2 alone," "a3 alone," "a1 and a2," "a1 and a3," "a2 and a3," or "a1, a2 and a3."

[0041] In this embodiment of the application, "connection" includes direct or indirect connection between objects. It can be directly connected through a medium (e.g., wires, wiring, etc.), or indirectly connected through other components, or it can be an internal connection.

[0042] A neural network model is a machine learning algorithm that simulates the connection and interaction patterns of neurons in the human brain to accomplish various machine learning tasks. A neural network model consists of multiple artificial neurons (hereinafter referred to as neurons), which are interconnected by weights. Each neuron can receive input signals from other neurons. These input signals are amplified or amplified through weighted processing. The output of a neuron is an activation function (or nonlinear function), used to process the weighted input signals and produce the neuron's output. Neural network models can be, for example, feedforward neural networks (FNN), convolutional neural networks (CNN), recurrent neural networks (RNN), long short-term memory networks (LSTM), etc., or improved models based on these neural network models. This application does not limit the type of neural network model.

[0043] The neural network model can be further divided into multiple layers, each containing multiple neurons. These layers are connected sequentially by weights. Each layer processes its input signal to generate an output signal, which is then passed to the next layer. The output signal of the last layer is the output signal (or processing result) of the neural network model. This application does not limit the way layers are divided; for example, they can be divided according to their function or position in the neural network model. For instance, convolutional layers can be used for image processing tasks; pooling layers can be used to reduce data computation; fully connected layers can be used to output the results of convolution calculations in a convolutional neural network, and so on. Regarding the structure of the neural network model, such as the type of layers, the number of layers of different types, the number of neurons in each layer, the connection methods between different neurons, the values ​​of the weights, and the type of activation function, this application does not impose any restrictions. Furthermore, in this application, the term "model" refers to a "neural network model."

[0044] In the processing of neural network models, there is a need to process multiple models. Traditional methods include sequential processing or multi-core scheduling. Sequential processing, where one model is processed before moving on to the next, continues until all models are processed. This sequential approach leads to excessive processing time (or latency) and may consume too many computing resources, resulting in insufficient resource utilization. Excessive processing time or insufficient resource utilization leads to decreased processing efficiency, especially when processing complex models with large datasets. Multi-core scheduling uses a multi-core architecture, where different processor cores work concurrently or sequentially through scheduling. For example, different cores can be scheduled to process different models. This approach results in a loss of energy efficiency, thus affecting processing performance and causing a decrease in processing efficiency.

[0045] Considering the above factors, embodiments of this application provide a processing module, circuit, method, and integrated circuit for neural network models. By utilizing pulsating sequences to process multiple models in parallel, latency is reduced, resource utilization is improved, and thus processing efficiency is enhanced. In embodiments of this application, the multiple neural network models can be identical, partially identical, or completely different models.

[0046] The following example uses a convolutional neural network (CNN). A CNN consists of multiple convolutional layers, each containing at least one two-dimensional image, also called a feature map. The number of feature maps in each convolutional layer can be the same or different. Each convolutional layer includes multiple weight matrices, also called kernels. The feature map output from the previous layer can be convolved with each kernel within the convolutional layer to produce a new feature map, which serves as the input to the next layer. The same convolutional layer can contain the same or different kernels, and the number and weights of kernels in different convolutional layers can also be the same or different.

[0047] The following explanation is provided in conjunction with the accompanying drawings. Please refer to them. Figure 1 This is a schematic diagram of the processing circuit (or processor) of a neural network model provided in an embodiment of this application. Figure 1 As shown, the processing circuit 100 includes a control module 110, a transmission module 120, and a neural network model processing module 130. The control module 110 is configured to generate a first control instruction, a second control instruction, and a third control instruction. The transmission module 120 reads input data of multiple neural network models from the memory 10 according to the first control instruction and stores it in the processing module 130. This input data includes, for example, input features and convolutional kernels of each layer in each convolutional neural network model. The processing module 130 processes the input data of the multiple neural network models according to the second control instruction and generates processing results for the multiple neural network models. The transmission module 120 is also configured to transmit the processing results of the multiple neural network models in the processing module 130 to the memory 10 according to the third control instruction. In this embodiment, the processing result of the neural network model refers to the computational result output by the last layer of the neural network model.

[0048] This application does not limit the type of the processing circuit 100 described above. For example, it can be a central processing unit (CPU), a microcontroller (MCU), a microprocessor unit (MPU), a graphics processing unit (GPU), or a digital signal processor (DSP). In another example, the processing circuit 100 can realize its processing capability through the logical relationship of the hardware circuit, which can be fixed or reconfigurable. For example, the processor can be a dedicated processor, such as a processor implemented by an application-specific integrated circuit (ASIC), which realizes its processing capability by designing the logical relationship between the components within the circuit; another example is a processor implemented by a programmable logic device (PLD), which realizes its processing capability by configuring the logical relationship between the logic devices through a configuration file; for example, a processor implemented by a field-programmable gate array (FPGA). In another example, the processor can be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), deep learning processing unit (DPU), etc. In one implementation, the processing circuit 100 can be integrated into an integrated circuit, such as a system-on-chip (SOC). Further, this integrated circuit can be located in a controller, such as the controller of an electronic device like a mobile phone or computer, or the controller of a vehicle such as a vehicle, ship, aircraft (e.g., flying vehicle, or drone), robot (e.g., industrial robot or home robot), or surveying equipment. Taking an intelligent driving scenario as an example, the processing circuit 100 can be integrated into an onboard SOC; further, the onboard SOC can be integrated into an onboard controller.The above-mentioned vehicle controllers include, for example, a domain control unit (DCU), an electronic control unit (ECU), a vehicle central computer (VCC), a zone controller (zonal / zone ECU, or zone control unit, ZCU), or a vehicle control unit (VCU). Domain controllers include, for example, a vehicle domain controller (VDC), a cockpit domain controller (CDC), or an advanced driving assistance system / autonomous driving (ADAS / AD) domain controller (ADC).

[0049] Furthermore, this application does not limit the type of memory 10. For example, memory 10 can be read-only memory (ROM) or random access memory (RAM). For example, it can be non-volatile memory or volatile memory. For example, it can be on-chip memory resources or off-chip memory resources. ROM includes, for example, mask ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash ROM, etc.; RAM includes, for example, static random-access memory (SRAM) or dynamic random-access memory (DRAM), etc. On-chip storage resources refer to the storage circuitry integrated within the chip. These circuits are integrated with other circuits on the chip (such as processing circuits) to provide storage space. Examples include on-chip static random access memory (SRAM) resources or on-chip configuration register space resources. Off-chip storage resources refer to independent storage circuitry or chips, such as off-chip double data rate synchronous dynamic random access memory (DDR SDRAM, or DDR for short).

[0050] Please continue to refer to this. Figure 1 Any of the processing procedures mentioned in any of the above embodiments, such as convolution operations on feature maps, data transfer between layers, etc., can be implemented by the processing module 130. The processing module 130 processes the data of the above multiple neural network models according to the above second control instructions and generates the processing results of the multiple neural network models. For example... Figure 1As shown, the processing module 130 includes a first buffer 131 and a systolic array 132. The first buffer 131 is configured to retrieve input data of multiple neural network models from the memory 10 and store the input data. The first buffer 131 is also configured to transfer the input data to the systolic array 132. In one implementation, the first buffer 131 is connected to the transmission module 120, and the first buffer 131 is configured to retrieve input data of multiple neural network models from the memory 10 through the transmission module 120. Further, for example, if the memory 10 is an off-chip storage resource (e.g., the memory 10 is DDR), the transmission module 120 can be a direct memory access (DMA) module, which can directly retrieve input data of multiple neural network models from the memory 10, improving transmission efficiency.

[0051] After acquiring the input data, the systolic array 132 initiates multiple pipelines to process the input data of multiple neural network models. The input data of a neural network model enters the model and undergoes layer-by-layer computation within the model. The final result of the model is output by the last layer; this process constitutes a pipeline operation. Multiple neural network models will generate multiple pipelines. The processing results of multiple pipelines are stored in the second buffer 132. For example, taking convolutional neural network model A as an example, assuming A includes four convolutional layers, layer1, layer2, layer3, and layer4, each layer requires convolution operations. The output result of each layer is the pipeline operation result, and the output result of the last layer (e.g., layer4) is the processing result of model A. The convolution operations are performed by the systolic array 132, and the systolic array 132 stores the operation results of these pipelines into the first buffer 131. In one implementation, such as... Figure 1 As shown, the processing module may also include a second buffer 133. In this case, the processing results of multiple pipelines are stored in the second buffer 133, which can reduce the storage capacity occupied by the first buffer. Both the first buffer and the second buffer are on-chip storage resources.

[0052] This application does not limit the computational operations performed by the systolic array. For example, it can perform convolution operations, matrix multiplication, multiply accumulate (MAC) and other operations.

[0053] A systolic array is a parallel computing architecture composed of multiple processing elements (PEs) arranged in a specific topology. Data and control commands are transmitted between different PEs. The structure of a systolic array is described below with reference to the accompanying diagram. A systolic array can be a one-dimensional or two-dimensional array and can have different arrangements. Figure 2 This is a schematic diagram of the arrangement of a pulsating array provided in an embodiment of this application. Figure 3 This is a schematic diagram illustrating another arrangement of pulsating arrays provided in an embodiment of this application. Figure 2 , Figure 3 As shown, each square is used to indicate a PE, without limiting the actual structure, shape, size or dimensions of the PE. Figure 2 This illustrates a one-dimensional arrangement of a pulsating array. Figure 2 Multiple physical exciters (PEs) are arranged horizontally to form a pulse array 200. In other implementations, multiple PEs can also be arranged vertically. This application embodiment does not limit the arrangement direction. Adjacent PEs can transmit data. For example, assuming... Figure 2 If the data transmission direction is from left to right, the input data first enters the first PE from the left, is processed, and is then passed to the next PE, and so on, until it is passed to the rightmost PE. Figure 3 This paper illustrates a two-dimensional arrangement of a pulsating array, which can be extended from a one-dimensional arrangement. Multiple physical exciters (PEs) arranged in multiple rows (e.g., horizontally) and multiple columns (e.g., vertically) constitute the pulsating array 300. Similar to the one-dimensional arrangement, data can also be transferred between PEs that are adjacent horizontally or vertically in the two-dimensional array. This application does not limit the size and arrangement of the pulsating array; for example, it can be an 8*8 or 16*16 two-dimensional array, which can be determined according to the hardware design.

[0054] Input data propagates forward in a pulsating array according to a certain pattern, driven by a clock signal. Each PE (Processing Entity) in the pulsating array has a certain processing capability, completing a portion of the computation within each clock cycle, and then passing the result to the next PE in the next clock cycle. Please refer to [reference needed]. Figure 4 This is a schematic diagram of a pulsating array provided in an embodiment of this application. Figure 4As shown, the input data can be divided into multiple blocks of data, and the input data is passed from right to left. The SUM module at the end of each column is used to perform an accumulation operation on that column, and the SUM output is stored in the second buffer 132. Assume that in clock cycle T1, block data 1 enters PE1; in clock cycle T2, block data 2 enters PE2, and simultaneously block data 1 is passed to PE4; in clock cycle T3, block data 3 enters PE3, and simultaneously block data 1 is passed to PE7, block data 2 is passed to PE5, and so on, until all block data is passed to the last column. Thus, starting from T2, the systolic array begins to process block data 1 and block data 2 in parallel, which is partially parallel; starting from T3, the systolic array begins to process block data 1-3 in parallel, which is fully parallel. The amount of data in each block can be determined according to the processing capacity of the PE, and this embodiment does not impose any limitations.

[0055] The process of passing block data in the PE line is the pipeline operation process. In one implementation, the above block data can be the input data of a neural network model. The systolic array can start multiple pipelines at different times, enabling multiple pipelines to run in parallel, thereby processing the input data of multiple neural network models in parallel to improve processing efficiency. For example, block data 1-3 are input data 1 of model 1, input data 2 of model 2, and input data 3 of model 3, respectively. In this case, T1, T2, and T3 are the start times of different pipelines. If models 1-3 correspond to pipelines 1-3 respectively, then starting from T2, the systolic array starts running pipelines 1 and 3 in parallel, which is partial parallelism; starting from T3, the systolic array starts running pipelines 1-3 in parallel, which is full parallelism. It should be noted that... Figure 4 If each row of the systolic array is used to run different pipelines, PEs located in the same column but belonging to different pipelines do not transfer data, such as PE1 and PE2, PE5 and PE6, etc. Furthermore, T1-T3 can be adjacent clock cycles or non-adjacent clock cycles; this application does not impose any restrictions on this. In this way, the systolic array starts different pipelines at different times, allowing multiple pipelines to run in parallel to handle scenarios with multiple concurrent models, thus improving processing efficiency compared to the traditional serial approach. In addition, the systolic array can start multiple pipelines simultaneously, enabling multiple pipelines to run in parallel, thereby processing the input data of multiple neural network models in parallel and further improving processing efficiency. (Continue to reference...) Figure 4Assuming T1 = T2 = T3 = T, at this point, block data 1-3 simultaneously enter the systolic array. This systolic array runs pipelines 1-3 simultaneously at time T. In one implementation, due to differences in input data between different models, the execution times of different pipelines also differ. For example, the execution times of pipelines 1-3 are a, b, and c respectively, assuming a < b < c. When pipeline 1 finishes running (e.g., at time T4), pipelines 2 and 3 are still running. At this time, another model's pipeline can be started. For example, block data 4 of model 4 is input to PE1 at time T4, starting pipeline 4 of model 4, allowing pipelines 2-4 to run in parallel. This increases the utilization rate of the systolic sequence, further improving processing efficiency. Figure 4 The example shown is of a single line of PEs running a pipeline. In one implementation, multiple lines of PEs can be used to run a pipeline, which will be explained in detail in subsequent embodiments. In one implementation, the above block data can also be input data from different layers of a neural network model. Furthermore, the block data can also be input data from some layers of a neural network model.

[0056] The timing of starting different pipelines in the systolic array can be directly determined by the control module. For example, the control module can directly control the first buffer to input different blocks of data into the systolic array at different times. In one implementation, the start time of different pipelines can also be determined based on the amount of data stored in the second buffer.

[0057] Please refer to Figure 5 This is a schematic diagram of a multi-pipeline parallel operation process provided in an embodiment of this application. Figure 5 Taking four pipelines running in parallel (i.e., four models running in parallel) as an example, such as... Figure 5 As shown, the entire parallel operation process includes three stages: pipeline startup, full parallelism, and pipeline termination. At the start, the block data input to each pipeline is retrieved from the first buffer, and the final output of each pipeline is stored in the first buffer. t1-tn represents the block data divided according to the input data of each model. This parallel operation method saves time in multi-model processing and improves processing efficiency. Furthermore, after the block data enters the pipeline, it flows continuously in the pulsating array, meaning the pulsating array reuses the input data multiple times. Therefore, the pipeline can reduce the repeated retrieval of the same data during operation, preventing excessive occupation of the first buffer's storage capacity, further improving processing efficiency.

[0058] At the pipeline startup time, block data t1 of model m1 is processed at the beginning. After t1 of m1 is processed, m2 can use the data output by m1 for processing, and at the same time, t2 of m1 is processed, and so on. When all pipelines start processing their respective block data, the fully parallel stage begins. When a pipeline finishes processing, the pipeline ends, until all pipelines have finished processing, completing the processing of models m1-m4.

[0059] The processing result of each pipeline can be stored in a first buffer or a second buffer. Multiple pipelines include a first pipeline and a second pipeline, and the first and second pipelines can be any two pipelines from a set of multiple pipelines. Figure 5 For example, multiple pipelines include M1, M2, M3, and M4, corresponding to models m1, m2, m3, and m4 respectively. For instance, the first pipeline is M2, and the second pipeline is M3. The following example uses the second buffer to store the pipeline processing results. Assume the systolic array starts M2 at the first moment. After M2 runs, its processing result (referred to as the first processing result for ease of distinction) is stored in the second buffer. When the amount of data from the first processing result in the second buffer reaches a certain threshold, M3 is started. This is the second moment, at which point M2 and M3 run in parallel. When the first and second pipelines are other pipelines among multiple pipelines, for example, the first pipeline is M1 and the second pipeline is M2, the activation methods for M1 and M2 are similar to those for M2 and M3, and will not be elaborated here. Thus, the amount of data from the previous pipeline's processing result stored in the second buffer triggers the activation of the next pipeline, achieving flexible control over multiple pipelines. The first quantity mentioned above can be determined, for example, based on the storage capacity of the second buffer and the size of each model; however, this application embodiment does not impose any limitations on this. In one implementation, the first data is less than or equal to the storage capacity of the second buffer.

[0060] Furthermore, the second buffer can be divided into multiple sub-buffers, and the processing result of a pipeline can be stored in at least one of these sub-buffers. Please refer to [reference needed]. Figure 6 This is a schematic diagram of the processing module of another neural network model provided in this application embodiment. The processing module 600 includes a first buffer 610, a second buffer 620, and a systolic array 630. The second buffer 620 includes multiple sub-buffers (tile buffers), including the first sub-buffer, which can be any one of the multiple sub-buffers. Figure 6For example, taking tilebuffer621 in the diagram as the first sub-buffer, this tile buffer is used to store the first processing result of the first pipeline. When the amount of data of the first processing result in this tile buffer reaches the first quantity mentioned above, the pulsating array 630 starts the second pipeline. The first quantity is less than or equal to the storage capacity of the first sub-buffer. By triggering different pipelines through the tile buffer, multiple models can be run in parallel to improve processing efficiency. It should be noted that... Figure 6 The tile buffers can also be arranged in other ways. In this embodiment, the number of tile buffers and their positions in the second buffer 620 are not limited.

[0061] Furthermore, the neural network model includes multiple layers, which can be divided into multiple groups. In one implementation, the multiple layers include a first layer and a second layer. The first layer can include a single layer, or it can include a group of multiple layers (hereinafter referred to as the first group); similarly, the second layer can also include a single layer, or it can include the second group. Multiple tile buffers can be used to store the processing results of different groups, where the first processing result includes the processing result of the first group. When the amount of data of the first processing result in the tile buffer reaches the storage capacity of the tile buffer, the systolic array stores the first processing result in the first buffer. The tile buffer retrieves the first processing result from the first buffer, and the systolic array processes the remaining groups (e.g., the second group) sequentially based on the first processing result. For example, assuming the model includes 10 layers a1 to a10, the storage capacity of the tile buffer can, for example, allow for systolic processing of up to 4 layers of data. During the first pipeline operation, the systolic array processes a1 to a4 sequentially, with the processing result of a4 placed in the first buffer. When processing a5 to a8, the result of a4 is retrieved from the first buffer, and a5 to a8 are processed sequentially based on the result of a4, with the processing result of a8 placed in the first buffer. When processing a9 to a10, the processing result of a8 is retrieved from the first buffer, and a9 to a10 are processed sequentially. In this way, the processing results of the groups can be stored in the first buffer, preventing the systolic array from malfunctioning due to tile buffer data overflow, reducing the probability of processing efficiency degradation caused by systolic array anomalies, and further improving reliability. In addition, the above method of processing the model into multiple groups can significantly improve processing efficiency when the model size is large.

[0062] Furthermore, the systolic array can be further divided into multiple systolic subarrays to run multiple pipelines. Each systolic subarray includes at least two PEs (Executable Processors). Multiple systolic subarrays are configured to launch multiple pipelines, and the number of systolic subarrays is the same as the number of sub-buffers. In one implementation, there is a correspondence between sub-buffers and systolic subarrays; for example, a sub-buffer is used to store the output data of its corresponding systolic subarray. Please refer to [reference needed]. Figure 6 631 and 632 are pulsating subarrays, each used to run different pipelines. Please refer to [reference needed]. Figure 7 This is a schematic diagram of another pulsating subarray partitioning provided in an embodiment of this application. Figure 7 The dashed box outlines several possible forms of the systolic subarray, such as a one-dimensional or two-dimensional array. This application does not limit the method of dividing the systolic subarray. The number of physical objects (PEs) in the systolic array is fixed. The number of PEs can be allocated to each pipeline based on the amount of input data for different models (e.g., 30 frames of images), thus forming multiple systolic subarrays. If the amount of input data is large, more PEs can be allocated to the systolic subarray; if the amount of input data is small, fewer PEs can be allocated, thereby achieving dynamic allocation of PEs within the systolic array. This dynamic allocation of PEs enables the systolic array to run multiple pipelines in parallel to process the input data of multiple models, improving processing efficiency.

[0063] In one implementation, please refer to Figure 8 This is a schematic diagram of another pulsating array structure provided in an embodiment of this application. (Refer to the reference...) Figure 8 , Figure 7 and Figure 4 ,like Figure 4 As shown, adjacent PEs in the pulsating array are interconnected (not shown in the figure). It is assumed that... Figure 7 The output terminals of the pulsating subarrays 633, 634, and 635 can be used... Figure 8 The system connects to the bus in a specific manner. The outputs of multiple systolic subarrays are connected to multiple third buffers. These third buffers transmit the processing results of the multiple systolic subarrays to the bus, and then, via the bus, further transmit the processing results to a second buffer. When the second buffer includes multiple sub-buffers, the processing results of the systolic subarrays can be transmitted to their respective sub-buffers, preventing the results from flowing between the systolic subarrays. The third buffers can be, for example, first-in-first-out (FIFO) buffers or other on-chip storage resources.

[0064] Furthermore, cyclic parallelism can be achieved using pulsating subarrays, further reducing latency and improving processing efficiency. Please refer to [link / reference needed]. Figure 6 For example, the systolic subarray 631 (referred to as the first subarray for ease of distinction) runs pipeline M5 (referred to as the third pipeline for ease of distinction). When M5 finishes running, the first subarray can continue to run other pipelines waiting to be run, such as M6 (referred to as the fourth pipeline for ease of distinction). That is, when any subarray in the systolic array is idle, the idle systolic subarray can be used to achieve cyclic parallelism, further improving the parallel processing efficiency.

[0065] In one implementation, based on the above multi-model parallelism embodiments, inter-layer parallelism is implemented within the model, thereby further shortening latency, saving cached data, and improving processing efficiency. Inter-layer parallelism is similar to the multi-model parallelism process. Taking the first neural network model among multiple neural network models as an example, the first neural network model includes multiple layers, including a first layer and a second layer. The first and second layers can be any two layers from the multiple layers. The first pipeline can be further divided into multiple sub-pipelines, including a first sub-pipeline and a second sub-pipeline. The first sub-pipeline processes the input data of the first layer, and the second sub-pipeline processes the input data of the second layer. The processing results of the first pipeline are stored in a first sub-buffer. For example, when the systolic array starts the first sub-pipeline, its processing results are stored in the first sub-buffer. When the processing results of the first sub-pipeline in the first sub-buffer reach a second quantity, the systolic array starts the second sub-pipeline, enabling the first and second sub-pipelines to run in parallel, that is, to process the first and second layers of the first neural network model in parallel. The above inter-layer parallelism method further divides the pipeline into multiple sub-pipelines and triggers the operation of the sub-pipelines based on the amount of data stored in the first sub-buffer, thereby achieving parallel operation within the model and further improving processing efficiency. The second quantity can be determined, for example, based on the storage capacity of the first sub-buffer and the scale of each layer; this embodiment does not impose such limitations. Furthermore, the above inter-layer parallelism method can be applied to at least one model of multiple neural network models. Further, when the pipeline is parallelized across layers, the pulsating subarray running the pipeline can be divided into finer-grained subarrays, for example, dividing the pulsating subarray into parallel regions for running different sub-pipelines. In one implementation, the first and second sub-pipelines can also be started simultaneously. The above inter-layer parallelism method is similar to the multi-model parallelism method and can be referred to the description above.

[0066] Based on the same technical concept, embodiments of this application also provide a method for controlling neural network model processing, which can be executed by the above-mentioned control module. For example... Figure 9 As shown, the method includes:

[0067] S910: Generate a first control instruction. This first control instruction controls the transmission module to retrieve input data from multiple neural network models from memory and store it in the first buffer of the neural network model processing module. This implements the parallel processing of any of the above multiple models.

[0068] S920: Generate a second control instruction, which is used to control the processing module to process the input data of multiple neural network models and generate the processing results of multiple neural network models; and transmit the processing results of multiple neural network models to the first buffer.

[0069] S930: Generate a third control instruction, which is used to control the transmission module to transfer the processing results of multiple neural network models in the first buffer to the memory.

[0070] Furthermore, embodiments of this application also provide a computer program product, including instructions, wherein the above processing method is executed when the instructions are executed by a processor.

[0071] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail or in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Furthermore, the above embodiments can be freely combined as needed.

Claims

1. A processing module for a neural network model, characterized in that, include: First buffer, second buffer, and pulsation array, The first buffer is configured to retrieve input data from a plurality of neural network models from memory; The first buffer is also configured to transmit the input data of the plurality of neural network models to the systolic array; The pulsating array includes multiple processing units arranged in multiple rows and columns, and multiple accumulation modules. The multiple processing units are connected to each other to transmit data and control commands. The pulsating array is configured to start pipelines, so that multiple pipelines run in parallel to process the input data of the multiple neural network models. The multiple pipelines correspond to the multiple neural network models respectively. The plurality of processing units are divided into a plurality of pulsating subarrays according to the amount of input data of the plurality of neural network models. The plurality of pulsating subarrays are used to run the plurality of pipelines respectively. Each pulsating subarray includes at least two of the processing units, and at least some of the processing units belong to different pulsating subarrays. The second buffer is used to store the processing results of the multiple pipelines; the second buffer includes multiple sub-buffers, each sub-buffer having a corresponding relationship with the pulsating subarray, and each sub-buffer being used to store the output data of its corresponding pulsating subarray; The plurality of production lines includes a first production line and a second production line; The pulsating array is configured to start a first pipeline and store the first processing result of the first pipeline in a first sub-buffer corresponding to the pulsating sub-array running the first pipeline; The pulsating array is further configured to start the second pipeline when the amount of data of the first processing result stored in the first sub-buffer reaches a first quantity, so that the first pipeline and the second pipeline run in parallel to realize the parallel operation of the multiple neural network models.

2. The processing module according to claim 1, characterized in that, The pulse array is configured to start the multiple pipelines at different times, so that the multiple pipelines run in parallel.

3. The processing module according to claim 1, characterized in that, The plurality of neural network models include a first neural network model, which includes multiple layers. The pulsating array is configured to process a first layer of the plurality of layers; and store the first processing result in a first sub-buffer; the first processing result includes the processing result of the first layer; The pulse array is configured to store the first processing result in the first buffer when the amount of data of the first processing result in the first sub-buffer reaches the storage capacity of the first sub-buffer. The pulsating array is configured to obtain the first processing result from the first buffer; The second layer among the plurality of layers is then processed based on the first processing result.

4. The processing module according to claim 1, characterized in that, The plurality of neural network models include a first neural network model, which includes multiple layers, including a first layer and a second layer; the first pipeline includes multiple sub-pipelines, including a first sub-pipeline and a second sub-pipeline. The pulse array is configured to start the second sub-pipeline when the processing result of the first sub-pipeline in the first sub-buffer reaches a second quantity, so that the first sub-pipeline and the second sub-pipeline run in parallel; the first sub-pipeline processes the input data of the first layer when it runs, and the second sub-pipeline processes the input data of the second layer when it runs.

5. The processing module according to claim 1, characterized in that, The pulsating subarray includes at least two processing units, and the number of the plurality of pulsating subarrays is the same as the number of the plurality of sub-buffers.

6. The processing module according to claim 5, characterized in that, It also includes multiple third buffers, which are connected to the outputs of the multiple pulsating subarrays. The plurality of third buffers are configured to transmit the processing results of the plurality of pulsating subarrays to the plurality of subbuffers.

7. The processing module according to claim 5 or 6, characterized in that, The plurality of pipelines includes a third pipeline and a fourth pipeline, and the plurality of pulsating subarrays includes a first subarray. The first subarray is configured to run the third pipeline; The first subarray is configured to run the fourth pipeline when the third pipeline finishes running.

8. A processing circuit for a neural network model, characterized in that, include: The control module and the transmission module further include the processing module as described in any one of claims 1-7. The control module is configured to generate a first control command, a second control command, and a third control command; The transmission module is configured to retrieve data of multiple neural network models from the memory according to the first control instruction and store them in the first buffer; The processing module is configured to process the data of the plurality of neural network models according to the second control instruction and generate the processing results of the plurality of neural network models; and to transmit the processing results of the plurality of neural network models to the first buffer. The transmission module is further configured to transmit the processing results of the plurality of neural network models in the first buffer to the memory according to the third control instruction.

9. A method for processing a neural network model, characterized in that, include: Generate a first control instruction, which is used to control the transmission module to obtain input data of multiple neural network models from the memory and store them in the first buffer of the processing module according to any one of claims 1-7; A second control instruction is generated, which is used to control the processing module to process the input data of the plurality of neural network models and generate the processing results of the plurality of neural network models; and to transmit the processing results of the plurality of neural network models to the first buffer. A third control instruction is generated, which is used to control the transmission module to transmit the processing results of the plurality of neural network models in the first buffer to the memory.

10. An integrated circuit, characterized in that, Includes the processing circuit as described in claim 8.

11. A controller, characterized in that, Including the integrated circuit as described in claim 10.

12. A vehicle, characterized in that, Includes the controller as described in claim 11.

Images