Model reasoning acceleration method and device based on DPU, equipment and medium

Abstract

The invention provides a DPU-based model reasoning acceleration method, device and equipment and a medium, and the method comprises the steps: carrying out the color separation of a to-be-reasoned data image, and obtaining at least two single-channel images; synchronously performing feature extraction on each single-channel image based on the dual-shrinkage array and the at least two convolution kernels to obtain a feature map; the feature map is used for model reasoning; the dual-shrinkage array is used for matrix multiplication required by feature extraction and realizing parallel computing; the dual-shrinkage array is realized by the DPU through a hardware description language. According to the invention, through the DPU dual-shrinkage array and at least two convolution kernels, feature extraction is carried out on a plurality of single-channel images and parallel calculation is carried out on a large-scale matrix, so that the system throughput is improved, and large-model reasoning acceleration is realized.

Description

Technical Field

[0001] The present invention relates to the field of model reasoning acceleration technology, and in particular to a DPU-based model reasoning acceleration method, device, equipment and medium. Background Art

[0002] With the development of large language model (large model) applications, traditional computing platforms based on central processing units (CPU) and graphics processing units (GPU) often cannot meet the requirements of high real-time performance and low power consumption of large model reasoning. At present, artificial intelligence (AI) accelerators can be built using data processing units (DPU) hardware based on field programmable gate arrays (FPGA) to run large models to accelerate the reasoning process of large models. Among them, DPU adopts FPGA + CPU architecture. DPU is designed for data-intensive operations and can efficiently process computing tasks that require a large number of input / output (I / O) operations, reduce the burden of host CPU computing scheduling, and improve overall efficiency.

[0003] Although the acceleration methods in related technologies improve the model reasoning process according to the hardware deployment environment or development performance, and often use hardware acceleration based on the GPU structure, they do not consider the computing characteristics of the large model itself to design the architecture, resulting in the system throughput to be improved and unable to meet the high concurrency requirements of large model reasoning tasks. Summary of the invention

[0004] The present invention provides a DPU-based model reasoning acceleration method, device, equipment and medium, which are used to solve the problem that the prior art does not take into account the computing characteristics of the large model itself to design the reasoning acceleration architecture, resulting in the system throughput to be improved and unable to meet the high concurrency requirements of large model reasoning tasks. By using a dual contraction array and at least two convolution kernels, feature extraction of multiple single-channel images is performed simultaneously and large-scale matrices are calculated in parallel, thereby improving the system throughput and realizing large model reasoning acceleration.

[0005] The present invention provides a DPU-based model reasoning acceleration method, comprising: Separate the colors of the image to be inferred to obtain at least two single-channel images; Based on a double contraction array and at least two convolution kernels, feature extraction is performed on each of the single-channel images simultaneously to obtain a feature map; the feature map is used by the model to infer the image to be inferred; the double contraction array is used to parallelly calculate the matrix multiplication required for feature extraction, and the double contraction array is implemented by the DPU through a hardware description language.

[0006] According to a DPU-based model inference acceleration method provided by the present invention, based on a dual contraction array and at least two convolution kernels, feature extraction is synchronously performed on each of the single-channel images to obtain a feature map, including: based on the feature data in the single-channel image, determining the target feature data corresponding to the convolution kernel; the target feature data is data that is convolved with each of the convolution kernels respectively; based on the processing units in the dual contraction array, convolving the weights of each of the convolution kernels with the corresponding target feature data respectively to obtain a convolution sub-result; passing the convolution sub-result to the next processing unit for accumulation to obtain at least two output results; wherein one of the output results corresponds to one of the convolution kernels; and determining the feature map based on the at least two output results and the bias corresponding to the output results.

[0007] According to a DPU-based model inference acceleration method provided by the present invention, the target feature data corresponding to the convolution kernel is determined based on the feature data in the single-channel image, including: inputting the feature data in the single-channel image into a data buffer row by row; mapping the feature data in a memory based on a parallel output interface of the data buffer; determining a delay time for obtaining the feature data based on a computing node for feature extraction in the dual contraction array; and continuously offsetting a read address in the memory based on the delay time to obtain the target data.

[0008] According to a DPU-based model inference acceleration method provided by the present invention, the feature map is determined based on the at least two output results and the bias corresponding to the output results, including: determining the first feature map in the single-channel image based on the at least two output results and the bias corresponding to the output result; performing nonlinear transformation and quantization processing on the first feature map to obtain a second feature map; and performing pooling processing on the second feature map to obtain the target feature map.

[0009] According to a DPU-based model inference acceleration method provided by the present invention, the second feature map is pooled to obtain the target feature map, including: based on a row buffer and a register, performing row-by-row maximum pooling on the elements of the second feature map to obtain first pooled data; the row buffer is implemented by the DPU through a hardware description language; based on the row buffer and the register, performing column-by-column maximum pooling on the first pooled data to obtain second pooled data; based on the second pooled data, obtaining the target feature map.

[0010] According to a DPU-based model inference acceleration method provided by the present invention, based on a row buffer and a register, the elements of the second feature map are subjected to row-by-row maximum pooling to obtain first pooled data, including: inputting the elements in the second feature map one by one into the row buffer in a horizontal direction; based on the parallel output interface of the row buffer, storing the elements in the second feature map one by one in a horizontal direction in the register; when the register stores the next element, comparing the element value of the next element with the element value of the previous element stored in the register, and retaining the row elements with larger element values; based on the row elements, obtaining the first pooled data.

[0011] According to a DPU-based model inference acceleration method provided by the present invention, the first pooled data is subjected to column-based maximum pooling based on a row buffer to obtain second pooled data, including: inputting two adjacent rows of elements in the first pooled data into the row buffer; based on the parallel output interface of the row buffer, storing the two adjacent rows of elements in a register; based on the memory, comparing the element values ​​of the first row elements of the two rows of elements with the element values ​​of the corresponding second row elements of the two rows of elements, and retaining the column elements with larger element values; based on the column elements, obtaining the second pooled data.

[0012] The present invention also provides a DPU-based model reasoning acceleration device, comprising the following modules: A separation module, used for separating the colors of the image to be inferred to obtain at least two single-channel images; An extraction module is used to synchronously extract features from each of the single-channel images based on a dual contraction array and at least two convolution kernels to obtain a feature map; the feature map is used by the model to infer the image to be inferred; the dual contraction array is used to parallelly calculate the matrix multiplication required for feature extraction, and the dual contraction array is implemented by the DPU through a hardware description language.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the DPU-based model reasoning acceleration method as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the DPU-based model reasoning acceleration methods described above.

[0015] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the DPU-based model reasoning acceleration methods described above.

[0016] The DPU-based model reasoning acceleration method, device, equipment and medium provided by the present invention separates the colors of the image to be inferred to obtain at least two single-channel images; based on a double contraction array and at least two convolution kernels, the features of each single-channel image are extracted synchronously to obtain a feature map, and the double contraction array is a matrix multiplication required for parallel computing feature extraction by the DPU through a hardware description language. In this way, by dividing the image to be inferred into multiple single-channel images, parallel processing of different color channels of image data is achieved, and the double contraction array is used to synchronously perform parallel computing on large-scale matrices, increasing the parallel computing of deep learning tasks, improving the data throughput of the platform, and further accelerating the reasoning process of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the DPU-based model reasoning method provided by the present invention.

[0019] Figure 2 It is a schematic diagram of the contraction array structure provided by the present invention.

[0020] Figure 3 It is a schematic diagram of the data input structure provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the accelerator provided by the present invention.

[0022] Figure 5 It is a schematic diagram of resource utilization of different acceleration methods provided by the present invention.

[0023] Figure 6 It is a schematic diagram of time efficiency under different acceleration methods provided by the present invention.

[0024] Figure 7 This is a schematic diagram of the structure of the DPU-based model reasoning acceleration device provided by the present invention. Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention.

[0025] Reference numerals: 400: accelerator; 410: first convolution unit; 420: second convolution unit; 430: third convolution unit; 440: bias buffer; 450: adder; 460: LeakyReLU activation and quantization inference unit; 470: maximum pooling unit. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The rapid development of large language models (large models) has made them a key element of machine learning tasks in various edge computing systems, such as autonomous driving, smart cities, and smart factories. These applications often need to meet high real-time and low power requirements, while traditional CPU and GPU-based computing platforms often find it difficult to meet such requirements. Therefore, a new, efficient, low-power platform is needed to support and accelerate edge intelligent applications.

[0028] For the new, efficient and low-power platform, the relevant technologies mainly include the following methods: Method 1, integrate the AI ​​intelligent computing system including DPU, generate and export the Bit stream file and transfer it to PetaLinux, use PetaLinux to build a Linux system with DPU AI intelligent computing module and burn it to the SD card; reason, optimize and compile the model into a file recognizable by DPU on the general computing platform; start the FPGA development board through the SD card, run the file on the FPGA-based Linux system, and realize DPU The IP call and model reasoning have the advantages of low difficulty in developing a general computing platform and high performance, low power consumption and customizability of the FPGA platform; Method 2: Pre-analyze the FPGA system-on-chip to obtain DPU deployment information; According to the principles of different hardware components, extract the hardware factors that will affect the CNN reasoning of the FPGA system-on-chip, and establish the system hardware model and DPU execution model through experimental verification; According to the model, a general formula is established to construct a general shared bus arbitration and memory access contention model of the FPGA system-on-chip; Method 3: Detect the candidate redundant data of the convolution kernel of the neural network to be calculated, and the candidate redundant data includes the network topology and convolution kernel parameters; According to the candidate redundant data, the neural network is optimized and reduced and re-reasoned to obtain the optimized network; The data to be processed is calculated based on the optimized network; The re-reasoning process and the calculation process of the data to be processed are performed using a DPU multi-core heterogeneous processor system. The present invention combines the network model itself and the network model calculation architecture for neural network calculation acceleration, and realizes the acceleration of the neural network reasoning process through the synergy of software optimization design and hardware acceleration calculation.

[0029] However, the defect of the above method 1 is that it develops an intelligent computing platform based on DPU, but does not design the architecture for the computing characteristics of the large model itself, and there is still room for improvement in throughput. The defect of method 2 is that it mainly analyzes the real-time performance of the DPU deployment environment, focusing on the configuration parameters of the general shared bus arbitration and memory access contention model, but does not consider the computing characteristics of the machine learning model itself, and mainly considers the reasoning process. The data flow structure of the reasoning process of the large model is more complex, and the amount of calculation increases. This method cannot meet the high concurrency requirements of the visual large model reasoning task. The defect of method 3 is that it mainly considers the heterogeneous computing characteristics of different layers of the neural network model, and uses the integrated DPU structure to accelerate the reasoning of the neural network model. However, method 3 ignores the potential for parallel computing of the same convolution layer and does not consider the optimization of the large amount of cached data in the data buffer. Especially for machine learning models with huge parameters such as large models, cache optimization is the key to accelerating model reasoning.

[0030] In summary, the acceleration methods in related technologies improve the model reasoning process based on the hardware deployment environment or development performance, and do not consider the reasoning characteristics of the large model itself to design the architecture, resulting in the system throughput to be improved and unable to meet the high concurrency requirements of large model reasoning tasks.

[0031] Based on the above problems, the present invention proposes a DPU-based model reasoning acceleration method, which realizes parallel processing of different color channels of image data by dividing the image to be inferred into multiple single-channel images. In addition, a dual contraction array is used to synchronously perform parallel calculations on large-scale matrices, thereby increasing the parallel calculation of deep learning tasks, improving the data throughput of the platform, and further accelerating the reasoning process of the model.

[0032] Combine the following Figure 1-Figure 6 The DPU-based model reasoning acceleration method of the present invention is described. The method can be applied to the reasoning acceleration of any model, including but not limited to the Convolutional Vision Transformer model (CvT), the target detection model, etc. This method can be applied to an AI accelerator system. The execution subject of this method can be an electronic device, or a DPU-based model reasoning acceleration method set in the electronic device. The DPU-based model reasoning acceleration device can be implemented by software, hardware, or a combination of the two.

[0033] Figure 1 : is a flow chart of the DPU-based model reasoning acceleration method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: Separate the colors of the image to be inferred to obtain at least two single-channel images.

[0034] Here, the data image to be inferred may be any image to be recognized, classified or perform other inference tasks.

[0035] Here, the color of the image to be inferred may include red, green, blue and other color images, that is, a color image is a single-channel image. For example, the color image to be inferred is divided into three single-channel images of red, green and blue.

[0036] Step 102: Based on the double contraction array and at least two convolution kernels, synchronously extract features from each of the single-channel images to obtain a feature map.

[0037] The feature map is used by the model to infer the image to be inferred; the dual contraction array is used to parallelize the matrix multiplication required for feature extraction, and the dual contraction array is implemented by the DPU through a hardware description language. Here, the model can be a large model containing at least two convolution kernels, such as a CvT model.

[0038] Here, the systolic array is also called a systolic array, which is an array composed of a group of processing elements (PE). The dual systolic array can also be called a dual systolic array, where the systolic array has only one group of processing elements to calculate the convolution operation, while the dual systolic array has two groups of processing elements to realize two convolution kernels to independently and synchronously perform convolution operations on a single-channel image.

[0039] It should be noted that, although increasing the number of parallel processing units in the systolic array can expand the parallelism in the systolic array and further improve the throughput, it will also increase the complexity of routing and require more DPU resources. That is, the reasoning of large models will not grow linearly with the systolic array. Therefore, through multiple tests, the potential relationship between additional parallelism and resource utilization is obtained, and it is found that the dual systolic array can balance parallelism and resource utilization.

[0040] Here, the convolution kernel can also be called a filter, which is generally 3 3 or 5 5 sizes. Among them, 3 The convolution kernel of 3 has 9 weights, 5 The convolution kernel of 5 has 25 convolution kernels.

[0041] Here, the hardware description language may be Very-High-Speed ​​Integrated Circuit Hardware Description Language (VHDL) or Verilog, which is limited in the present invention.

[0042] It should be noted that by writing the hardware description language in the DPU, the DPU can realize the double contraction array.

[0043] It should be noted that each single-channel image has an independent convolution unit, for example, convolution unit Convolution_R, convolution unit Convolution_G and convolution unit Convolution_B, and each convolution unit has its weight buffer and feature map buffer.

[0044] In an embodiment of the present invention, at least two single-channel images are obtained by separating the colors of the image to be inferred; based on a double contraction array and at least two convolution kernels, feature extraction is performed on each of the single-channel images synchronously to obtain a feature map, and the double contraction array is a matrix multiplication operation required for parallel computing feature extraction by the DPU through a hardware description language. In this way, by dividing the image to be inferred into multiple single-channel images, parallel processing of different color channels of image data is achieved, and the double contraction array is used to synchronously and independently implement the parallel calculation of large-scale matrices of multiple convolution kernels on single-channel images, increasing the parallel calculation of deep learning tasks, improving the data throughput of the platform, and further accelerating the reasoning process of the model. , increasing the parallel calculation of deep learning tasks, improving the data throughput of the platform, and further accelerating the reasoning process of the model.

[0045] Furthermore, based on the double contraction array and at least two convolution kernels, the feature extraction is performed on each of the single-channel images synchronously to obtain a feature map, including: Based on the feature data in the single-channel image, the target feature data corresponding to the convolution kernel is determined; the target feature data is data that is convolved with each convolution kernel respectively; based on the processing units in the double contraction array, the weights of each convolution kernel are convolved with the corresponding target feature data to obtain convolution sub-results; the convolution sub-results are passed to the next processing unit for accumulation to obtain at least two output results; wherein one output result corresponds to one convolution kernel; based on the at least two output results and the offsets corresponding to the output results, the feature map is determined.

[0046] Here, feature data refers to each pixel value in a single-channel image. It can be understood that each pixel value needs to be multiplied by the corresponding weight in the convolution kernel for convolution operation. Therefore, feature data of the same size as the convolution kernel needs to be selected for calculation each time.

[0047] Figure 2 Schematic diagram of the contraction array structure provided by the present invention, such as Figure 2 As shown, W0-W8 are the weights of the convolution kernel, g0-g8 are the gradient values, and each processing unit (PE) in the contraction array sequentially performs the product of the input data (1-9) and the corresponding weight (W0-W8) to obtain the convolution sub-result, and then passes the convolution sub-result to the next PE for the next data to continue the operation. For example, the first group of data (such as F1 and W0) will complete the multiplication calculation in the first PE, and then the result will be passed to the next PE for it to continue the operation with the next group of data (such as F2 and W1), and accumulate in sequence until the product of the last group of data (such as F9 and W8) is added to the result calculated by the previous PE to obtain the output result. The formula executed by each PE can be as follows (1): P = FW + C (1) Among them, P represents the processing unit, F is the feature map data, W is the weight of the convolution kernel, and C is the accumulation term.

[0048] Here, the offset may be obtained from an offset buffer.

[0049] It should be noted that the dual contraction array inputs the feature data into two sets of parallel PE arrays so that the arrays simultaneously calculate the convolution of two independent convolution kernels to obtain two output results.

[0050] In an embodiment of the present invention, the dual contraction array can simultaneously calculate the convolution of two independent convolution kernels, thereby realizing parallel processing and dual-path output. The advantages of the dual contraction array lie in its parallel processing capability and dual-path output characteristics. By processing two sets of independent convolution kernels at the same time, the dual contraction array can complete more computing tasks in the same time, thereby greatly improving the overall computing efficiency. In addition, the dual-path output characteristic enables the dual contraction array to output two sets of convolution results at the same time, further improving the throughput of the system.

[0051] In one embodiment, determining the target feature data corresponding to the convolution kernel based on the feature data in the single-channel image includes: The feature data in the single-channel image are input into a data buffer row by row; based on the parallel output interface of the data buffer, the feature data are mapped in a memory; based on the computational nodes of feature extraction in the double contraction array, the delay time for obtaining the feature data is determined; based on the delay time, the read address is continuously offset in the memory to obtain the target data.

[0052] Here, the line buffer can be understood as a temporary memory including one input interface and multiple output interfaces, and the feature data is mapped in the memory through conversion of the line buffer.

[0053] Here, the delay duration may be the same as the duration between computing nodes in the double contraction array, or the difference between the delay duration and the duration between adjacent computing nodes is less than a preset duration threshold, which is not limited in the present invention.

[0054] Specifically, the feature data in the single-channel image is input into the data buffer row by row, and the feature data is mapped in the memory through the data buffer. According to the computing node of the feature extraction in the double contraction array, the delay time for obtaining the feature data is determined, and based on the delay time, the read address is continuously offset in the memory to obtain the target data. According to the target data, convolution operations are performed with each convolution kernel respectively to obtain the output results corresponding to each convolution kernel.

[0055] Figure 3 It is a schematic diagram of the data input structure provided by the present invention, such as Figure 3As shown in the figure, the feature map is the image that is operated with the convolution kernel. The data input structure designs a row buffer (i.e., data buffer), where the row buffer consists of three first-in-first-out (FIFO) queues. The row buffer is responsible for inputting image data row by row and transmitting the three data points of the same row to the convolution operation through three parallel output interfaces.

[0056] For example, for the convolution unit calculation of a large model, the size of the convolution kernel is 3x3, and 9 data points need to be obtained from the feature map at the same time for calculation. To achieve this, the row buffer uses 9 pseudo-dual-port RAMs for data buffering, one port is dedicated to reading data and the other port is dedicated to writing data. When the data is written, the three numbers output from the row buffer are written to each group of three random access memories (Random Access Memory, RAM), which store the same row data. After three write operations, a complete feature map will be formed in the RAM. Then the reading process begins, and the 9 data points required for the convolution can be retrieved by offsetting the read address.

[0057] In an embodiment of the present invention, feature data is mapped in RAM through a data buffer to obtain corresponding target data, and data is obtained according to the delay duration. A delay mechanism is introduced to ensure that the data is on the computing node when the convolution computing unit needs it, thereby ensuring that the accelerator can perform convolution computing under efficient data flow without the computing unit being idle due to data waiting, thereby optimizing performance and optimizing the system for high throughput and low latency.

[0058] In one embodiment, determining the feature map based on the at least two output results and the bias corresponding to the output results includes: determining a first feature map in the single-channel image based on the at least two output results and the bias corresponding to the output result; performing nonlinear transformation and quantization processing on the first feature map to obtain a second feature map; and performing pooling processing on the second feature map to obtain the target feature map.

[0059] It should be noted that after the convolution operation, a bias needs to be added to adjust the offset of the output result, where the bias can be any suitable value, such as 0 or a very small constant. In this way, the problem of gradient explosion or gradient disappearance can be avoided.

[0060] Here, the nonlinear transformation can be achieved through an activation function, where the activation function includes but is not limited to a semi-linear activation (Leaky ReLU) function, a Sigmoid function, a Tanh function, and the like.

[0061] Here, quantization processing means that when implementing the inference of the convolutional neural network in hardware, all operations are implemented using integer or custom data types to reduce the size and computational complexity of the model so that it can run more efficiently on the hardware.

[0062] Here, the pooling process is to reduce the dimension of the feature map, highlight the most significant features, and provide a basis for subsequent layers or final predictions.

[0063] In the embodiment of the present invention, by adding bias to the output results, performing nonlinear changes, quantization processing, pooling processing and other optimization operations, the hardware can efficiently execute the model and accelerate the reasoning of the model.

[0064] In one embodiment, performing pooling processing on the second feature map to obtain the target feature map includes: Based on the row buffer and the register, the elements of the second feature map are subjected to row-by-row maximum pooling to obtain first pooled data; the row buffer is implemented by the DPU through a hardware description language; based on the row buffer and the register, the first pooled data are subjected to column-by-column maximum pooling to obtain second pooled data; based on the second pooled data, the target feature map is obtained.

[0065] Here, the row buffer is a temporary storage device including one input interface and multiple output interfaces. The elements in the feature map are stored in registers according to certain rules for pooling through the row buffer.

[0066] Specifically, the element values ​​of each row in the feature map are first pooled to obtain the first pooled data; then the element values ​​of each column in the first pooled data are pooled to obtain the second pooled data, and the second pooled data are visualized to obtain the target feature map.

[0067] In an embodiment of the present invention, the maximum pooling is decomposed into two steps of row-by-row maximum pooling and column-by-column maximum pooling through a row buffer system, which effectively simplifies the pooling process. This not only simplifies the hardware implementation, but also effectively utilizes DPU resources to perform the pooling operations required by the neural network.

[0068] In one embodiment, based on the row buffer and the register, the elements of the second feature map are subjected to row-by-row maximum pooling to obtain the first pooled data, including: inputting the elements in the second feature map one by one into the row buffer in a horizontal direction; based on the parallel output interface of the row buffer, storing the elements in the second feature map one by one in the register in a horizontal direction; when the register stores the next element, comparing the element value of the next element with the element value of the previous element stored in the register, and retaining the row elements with larger element values; based on the row elements, obtaining the first pooled data.

[0069] It should be noted that pooling is an operation to reduce the dimension of feature maps. For example, the input data before pooling is 480 256, after pooling it becomes 240 256.

[0070] In the embodiment of the present invention, the row pooling process is effectively simplified through the row buffer and the register, which not only simplifies the hardware implementation but also effectively utilizes the DPU resources to perform the pooling operations required by the neural network.

[0071] In one embodiment, the first pooled data is subjected to column-wise maximum pooling based on a row buffer to obtain second pooled data, including: inputting two adjacent rows of elements in the first pooled data into the row buffer; based on a parallel output interface of the row buffer, storing the two adjacent rows of elements in a register; based on the memory, comparing the element values ​​of the first row elements of the two rows of elements with the element values ​​of the corresponding second row elements of the two rows of elements, and retaining the column elements with larger element values; and obtaining the second pooled data based on the column elements.

[0072] For example, the data before column pooling is 240 256, the data after column pooling is 240 128,.

[0073] In the embodiment of the present invention, the column pooling process is effectively simplified through row buffers and registers, which not only simplifies the hardware implementation but also effectively utilizes DPU resources to perform the pooling operations required by the neural network.

[0074] In an embodiment of the present invention, the DPU-based model reasoning acceleration method can be applied to an accelerator, which can be constructed using a hardware description language and implemented using DPU hardware.

[0075] For example, Figure 4 Schematic diagram of the structure of the accelerator provided by the present invention, such as Figure 4As shown, the accelerator 400 includes: a first convolution unit 410: for convolving a red single-channel image; a second convolution unit 420: for convolving a green single-channel image; a third convolution unit 430: for convolving a blue single-channel image; a bias buffer 440, for applying a bias to the result from the contraction array; an adder 450: for combining the contribution of each convolution unit and adding a bias before passing the result to the activation function; a LeakyReLU activation and quantization inference unit 460: for implementing the Leaky ReLU function, introducing nonlinearity in the processing pipeline, and performing quantization to improve inference efficiency; a maximum pooling unit 470, for reducing the dimension of the feature map, highlighting the most significant features, and preparing for subsequent layers or final predictions. Each convolution unit includes a weight buffer: used to store the weights of the convolution kernel and provide them to the contraction array according to the needs of the convolution operation; a feature map buffer: used to temporarily save the input data, which can be the original input image or the output from the previous layer, ready to be processed by the contraction array; and a contraction array: used to efficiently perform matrix multiplication operations to achieve the parallel computing required in deep learning tasks. Figure 4 In addition to the units shown in , it can also include: a multi-head attention unit, which is used to use a self-attention mechanism so that the output of each position is the weighted sum of the inputs of all positions; a multi-layer perceptron: which is used to increase the nonlinear expression ability of the model by using a multi-layer perceptron.

[0076] It should be noted that the weight buffer is a key component that is designed to store the weights of the convolutional layers that need to be processed. It is implemented using a read-only memory (ROM) where the weight data is preloaded into a data file in the ROM. This setup ensures that the weight data is available at startup, eliminating the need for runtime loading, thereby improving the efficiency of the accelerator. In the context of the computational unit's ability to process two convolution kernels simultaneously, each kernel has 9 values, and a total of 18 weights need to be output. These 18 values ​​are concatenated into a binary string. Once read from the ROM, this binary string is split and assigned to 18 different variables, in keeping with the architectural requirement to output the kernel weight data in a structured and ordered manner. After the convolution operation of the three rows of feature maps is completed, the system must switch to the next set of convolution kernel weights to continue processing. This is done seamlessly through a carefully coordinated control mechanism, which signals the completion of feature map processing. Upon receiving this signal, the weight buffer module updates to the next pair of kernel weights in preparation for the next computation cycle. This process repeats, allowing the one-dimensional accumulation calculation for all convolution kernels to be executed efficiently.

[0077] The adder uses a data input structure (i.e. the data buffer mentioned above), a weight buffer structure, a dual contraction array structure, and a pooling calculation structure to accelerate the convolution operation in parallel. The core of this is the dual contraction array architecture, which has efficient data transmission and parallel processing capabilities, and is particularly suitable for the sequential data flow required by large visual models. The dual contraction array usually leads to lower latency due to its pipeline nature, allowing continuous data input and output. Compared with the traditional contraction array structure, the dual contraction array structure improves throughput by expanding the parallelism within the contraction array. The addition tree, as another architecture, focuses on hierarchical aggregation of the results of the multipliers. This has advantages in reducing logic depth and potentially increasing clock speeds. However, compared with the contraction array architecture, the addition tree architecture may introduce synchronization issues and may require more complex control logic to effectively manage data flow.

[0078] In order to verify and test the effectiveness of the DPU-based model reasoning acceleration method proposed in the present invention, the following simulation experiment was carried out.

[0079] The experiments were developed using Xilinx Vivado 2023.2 and do not require the deployment of physical hardware. The simulation environment provided by Vivado allows for comprehensive testing and verification of DPU-based AI accelerator designs. The Vivado simulation environment includes the Vivado 2023.2 tool, which leverages Vivado’s suite features, such as the logic simulator, to perform reasoning behavior and timing simulations. A VHDL / Verilog test bench was developed to simulate the operation of the AI ​​model and test the performance of the DPU design architecture. Multiple iterative runs were performed to evaluate functionality, timing, and potential boundary cases.

[0080] Figure 5 is a schematic diagram of resource utilization of different acceleration methods provided by the present invention, such as Figure 5 As shown in the figure, the horizontal axis represents the expected resource utilization (%), the vertical axis includes flip-flop (FF), global clock buffer (BUFG), look-up table random access memory (LUTRAM), look-up table (LUT), gray represents the DPU-based model reasoning acceleration method provided by the present invention, orange represents the adder tree (Adder), and blue represents the contraction array. Figure 5It can be seen that the resource usage of the adder tree and double systolic array designs increases slightly compared to the original systolic array. In particular, the utilization of flip-flops increases significantly in the double systolic array, indicating that a more complex control logic is used to manage the additional parallel computation. While the usage of lookup tables (LUTs) and global buffers (BUFGs) remains relatively consistent across designs, the significant increase in flip usage in the double systolic array design may affect scalability and maximum frequency. From the resource utilization graphs provided, it is clear that the double systolic array architecture consumes more flip-flops compared to the original systolic array and adder tree architectures. The increase in resource usage is consistent with the design's intention to enhance parallel computation capabilities.

[0081] Figure 6 It is a schematic diagram of time efficiency under different acceleration methods provided by the present invention, such as Figure 6 As shown, the blue line represents the CPU, the dark orange line represents the systolic array, the gray line represents the addition tree, and the light orange line represents the double systolic array. The horizontal axis represents efficiency and the vertical axis represents time (ms). Figure 6 The following conclusions can be drawn: Systolic Arrays: The original systolic array is specialized for convolution computations, providing computation times that are 5 times faster than the CPU. Its architecture is designed for efficient data flow and parallel processing, which fits well with the operation of convolutional neural networks. However, while systolic arrays are faster than CPUs, they may not provide the same level of performance improvement in tasks outside of their specialization such as non-matrix multiplications or algorithms that do not benefit from their parallel data flow.

[0082] Adder Tree: Adder Tree is a common DPU parallel computing structure that provides approximately double the performance of CPU performance. It can be a more familiar architecture for DPU developers and may provide wider applicability for a variety of computing tasks. Although the adder tree has its advantages, it cannot outperform the systolic array in this specific application. This may be due to the fact that the data processing of convolution is less optimized for the task, indicating that the adder tree may be better suited for other types of calculations.

[0083] Double-Systolic Array: This architecture computes 10 times faster than a CPU, highlighting the significant advantages of parallel processing in a DPU. By doubling the parallel computing capacity, it greatly improves data throughput and operation speed. But the increased parallelism leads to higher resource consumption, which may present scalability issues on resource-limited DPUs. There is a point of diminishing returns where additional parallelism does not equate to proportional performance gains, especially when the communication bandwidth of the DPU becomes a bottleneck.

[0084] In the specific context of accelerating CvT convolutional layers, dual systolic arrays provide the best performance, justifying the trade-off of increased resource utilization. The results suggest that for applications where speed is critical and resources are available, a dual systolic array architecture may be the best choice. Conversely, when resource conservation is more critical, or for applications that require more general computational tasks, systolic arrays or adder trees may be more appropriate.

[0085] In an embodiment of the present invention, the modules in the visual large model that can be accelerated based on DPU hardware are studied for the large model reasoning task, and a DPU-based parallel acceleration and scheduling structure is designed for the convolution operation therein to achieve flexible allocation of resources; an accelerator architecture is designed, including a data input structure, a weight buffer structure, a dual contraction array structure and a pooling calculation structure, and different accelerator structures are integrated to expand the contraction array by increasing parallel computing operations to improve throughput; a VHDL / Verilog test platform is developed to simulate the computing operations of large model reasoning, and a large model reasoning acceleration instance is generated for the DPU design.

[0086] The DPU-based model reasoning acceleration device provided by the present invention is described below. The DPU-based model reasoning acceleration device described below and the DPU-based model reasoning acceleration method described above can refer to each other.

[0087] Figure 7 : is a schematic diagram of the structure of the DPU-based model reasoning acceleration device provided by the present invention, such as Figure 7 As shown, the DPU-based model reasoning acceleration device 700 includes: A separation module 701 is used to separate the colors of the image to be inferred to obtain at least two single-channel images; The extraction module 702 is used to synchronously extract features from each of the single-channel images based on a dual contraction array and at least two convolution kernels to obtain a feature map; the feature map is used by the model to infer the image to be inferred; the dual contraction array is used to parallelly calculate the matrix multiplication required for feature extraction, and the dual contraction array is implemented by the DPU through a hardware description language.

[0088] In some embodiments, the extraction module 702 is specifically used to: determine the target feature data corresponding to the convolution kernel based on the feature data in the single-channel image; the target feature data is the data that is convolved with each convolution kernel respectively; based on the processing units in the double contraction array, convolve the weights of each convolution kernel with the corresponding target feature data respectively to obtain a convolution sub-result; pass the convolution sub-result to the next processing unit for accumulation to obtain at least two output results; wherein one output result corresponds to one convolution kernel; determine the feature map based on the at least two output results and the bias corresponding to the output results.

[0089] In some embodiments, the extraction module 702 is further specifically used to: input the feature data in the single-channel image into a data buffer row by row; map the feature data in a memory based on a parallel output interface of the data buffer; determine a delay time for obtaining the feature data based on a computing node for feature extraction in the double contraction array; and based on the delay time, continuously offset the read address in the memory to obtain the target data.

[0090] In some embodiments, the extraction module 702 is further specifically used to: determine a first feature map in the single-channel image based on the at least two output results and the bias corresponding to the output results; perform nonlinear transformation and quantization processing on the first feature map to obtain a second feature map; and perform pooling processing on the second feature map to obtain the target feature map.

[0091] In some embodiments, the extraction module 702 is further specifically used to: based on the row buffer and the register, perform row-by-row maximum pooling on the elements of the second feature map to obtain first pooled data; based on the row buffer and the register, perform column-by-column maximum pooling on the first pooled data to obtain second pooled data; based on the second pooled data, obtain the target feature map.

[0092] In some embodiments, the extraction module 702 is further specifically used to: input the elements in the second feature map one by one in a horizontal direction into the row buffer; based on the parallel output interface of the row buffer, store the elements in the second feature map one by one in a horizontal direction in a register; when the register stores the next element, compare the element value of the next element with the element value of the previous element stored in the register, and retain the row elements with larger element values; based on the row elements, obtain the first pooled data.

[0093] Figure 8 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the DPU-based model reasoning acceleration method, which includes: separating the colors of the image to be reasoned to obtain at least two single-channel images; based on a dual contraction array and at least two convolution kernels, synchronously extracting features from each of the single-channel images to obtain a feature map; the feature map is used by the model to reason about the image to be reasoned; the dual contraction array is used to parallelly calculate the matrix multiplication required for feature extraction, and the dual contraction array is implemented by the DPU through a hardware description language.

[0094] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0095] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the DPU-based model inference acceleration method provided by the above methods, which includes: separating the colors of the image to be inferred to obtain at least two single-channel images; based on a double contraction array and at least two convolution kernels, synchronously extracting features from each of the single-channel images to obtain a feature map; the feature map is used by the model to infer the image to be inferred; the double contraction array is used to parallelly calculate the matrix multiplication required for feature extraction, and the double contraction array is implemented by the DPU through a hardware description language.

[0096] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the DPU-based model inference acceleration method provided by the above-mentioned methods, the method comprising: separating the colors of the image to be inferred to obtain at least two single-channel images; based on a double contraction array and at least two convolution kernels, synchronously performing feature extraction on each of the single-channel images to obtain a feature map; the feature map is used by the model to infer the image to be inferred; the double contraction array is used to parallelly calculate the matrix multiplication required for feature extraction, and the double contraction array is implemented by the DPU through a hardware description language.

[0097] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0098] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A DPU-based model reasoning acceleration method, characterized in that: include: Separate the colors of the image to be inferred to obtain at least two single-channel images; Based on a double contraction array and at least two convolution kernels, synchronously extracting features from each of the single-channel images to obtain a feature map; The feature map is used by the model to infer the data image to be inferred; the dual contraction array is used for parallel calculation of matrix multiplication operations required for feature extraction, and the dual contraction array is implemented by the DPU through a hardware description language.

2. The DPU-based model reasoning acceleration method according to claim 1, characterized in that: The method of synchronously extracting features from each of the single-channel images based on the double contraction array and at least two convolution kernels to obtain a feature map includes: Based on the feature data in the single-channel image, determining the target feature data corresponding to the convolution kernel; the target feature data is the data for performing convolution operations with each of the convolution kernels respectively; Based on the processing units in the double contraction array, a convolution operation is performed on the weight of each convolution kernel and the corresponding target feature data to obtain a convolution sub-result; Passing the convolution sub-result to the next processing unit for accumulation to obtain at least two output results; wherein one output result corresponds to one convolution kernel; The feature map is determined based on the at least two output results and the biases corresponding to the output results.

3. The DPU-based model reasoning acceleration method according to claim 2, characterized in that: The determining, based on the feature data in the single-channel image, target feature data corresponding to the convolution kernel comprises: Inputting the feature data in the single-channel image into a data buffer row by row; Mapping the feature data in a memory based on a parallel output interface of the data buffer; Determining a delay time for acquiring the feature data based on a computing node for feature extraction in the dual contraction array; Based on the delay time, the read address is continuously shifted in the memory to obtain the target data.

4. The DPU-based model reasoning acceleration method according to claim 2, characterized in that: The determining the feature map based on the at least two output results and the biases corresponding to the output results includes: Determining a first feature map in the single-channel image based on the at least two output results and the offsets corresponding to the output results; Performing nonlinear transformation and quantization processing on the first feature map to obtain a second feature map; Perform pooling processing on the second feature map to obtain the target feature map.

5. The DPU-based model reasoning acceleration method according to claim 4, characterized in that: The performing pooling processing on the second feature map to obtain the target feature map includes: Based on a row buffer and a register, performing row-by-row maximum pooling on elements of the second feature map to obtain first pooled data; the row buffer is implemented by the DPU through a hardware description language; Based on the row buffer and the register, performing column-wise maximum pooling on the first pooled data to obtain second pooled data; Based on the second pooled data, the target feature map is obtained.

6. The DPU-based model reasoning acceleration method according to claim 5, characterized in that: The step of performing row-by-row maximum pooling on the elements of the second feature map based on the row buffer and the register to obtain first pooled data includes: Input the elements in the second feature map into the row buffer one by one in a horizontal direction; Based on the parallel output interface of the row buffer, the elements in the second feature map are stored one by one in a register in a horizontal direction; When the register stores a next element, the element value of the next element is compared with the element value of the previous element stored in the register, and the row element with the larger element value is retained; Based on the row elements, the first pooled data is obtained.

7. The DPU-based model reasoning acceleration method according to claim 5, characterized in that: The performing column-wise maximum pooling on the first pooled data based on the row buffer to obtain second pooled data includes: Input two adjacent rows of elements in the first pooled data into the row buffer; Based on the parallel output interface of the row buffer, storing the two adjacent rows of elements in a register; Compare the element values ​​of the first row elements of the two rows of elements with the element values ​​of the corresponding second row elements of the two rows of elements based on the memory, and retain the column element with the larger element value; Based on the column elements, the second pooled data is obtained.

8. A DPU-based model reasoning acceleration device, characterized in that: include: A separation module, used for separating the colors of the image to be inferred to obtain at least two single-channel images; An extraction module, used for synchronously extracting features from each of the single-channel images based on a double contraction array and at least two convolution kernels to obtain a feature map; The feature map is used by the model to infer the data image to be inferred; the dual contraction array is used for parallel calculation of matrix multiplication operations required for feature extraction, and the dual contraction array is implemented by the DPU through a hardware description language.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the DPU-based model reasoning acceleration method as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the DPU-based model reasoning acceleration method as described in any one of claims 1 to 7 is implemented.

Images