Dpu-based model inference acceleration method, apparatus, device, and medium

Abstract

The application provides a DPU-based model inference acceleration method, device, equipment and medium, which comprises the following steps: separating the colors of a to-be-inferred data image to obtain at least two single-channel images; performing feature extraction on each single-channel image based on a double-shrink array and at least two convolution kernels to obtain a feature map; and the feature map is used for model inference; the double-shrink array is used for matrix multiplication operation required by feature extraction and parallel computing; and the double-shrink array is realized by a DPU through a hardware description language. Through the double-shrink array of the DPU and the at least two convolution kernels, feature extraction is performed on multiple single-channel images, and parallel computing is performed on a large-scale matrix, the system throughput is improved, and thus large model inference acceleration is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model inference acceleration, and in particular to a model inference acceleration method and device based on a DPU, equipment and a medium. BACKGROUND

[0002] With the development of large language model (large model) applications, traditional computing platforms based on central processing units (CPUs) and graphics processing units (GPUs) often cannot meet the high real-time and low power consumption requirements of large model inference. Currently, an artificial intelligence (AI) accelerator based on a field programmable gate array (FPGA) data processing unit (DPU) hardware can be used to run a large model to accelerate the inference process of the large model. The DPU adopts an FPGA + CPU architecture, and is designed for data-intensive operations. It can efficiently process computing tasks that require a large number of input / output (I / O) operations, reduce the burden of host CPU computing scheduling, and improve overall efficiency.

[0003] Although the acceleration method in the related art improves the model inference process according to the deployment environment or development performance of the hardware, and often uses hardware acceleration based on a GPU structure, it does not consider the computing characteristics of the large model itself to design the architecture, resulting in a system throughput that needs to be improved and the inability to meet the high concurrency requirements of the large model inference task. SUMMARY

[0004] The present application provides a model inference acceleration method, device, equipment and medium based on a DPU, to solve the defect that the existing technology does not consider the computing characteristics of the large model itself to design the inference acceleration architecture, resulting in a system throughput that needs to be improved and the inability to meet the high concurrency requirements of the large model inference task. It realizes feature extraction on multiple single-channel images and parallel computation on large-scale matrices by using a double contraction array and at least two convolution kernels, improves the system throughput, and thus realizes large model inference acceleration.

[0005] The present application provides a model inference acceleration method based on a DPU, comprising:

[0006] Separating the color of the image to be inferred to obtain at least two single-channel images;

[0007] The feature extraction is performed on each single-channel image synchronously based on a double shrinkage array and at least two convolution kernels, and a feature map is obtained; the feature map is used for the model to perform inference on the to-be-inferred image; the double shrinkage array is used for parallel calculation of matrix multiplication required by feature extraction, and the double shrinkage array is realized by a DPU through a hardware description language.

[0008] According to the DPU-based model inference acceleration method provided in the application, the feature data in the single-channel image is used to determine target feature data corresponding to the convolution kernel; the target feature data is data subjected to convolution operation with each convolution kernel respectively; the weight of each convolution kernel and the corresponding target feature data are subjected to convolution operation by a processing unit in the double shrinkage array respectively, and a convolution sub-result is obtained; the convolution sub-result is transmitted to a next processing unit for accumulation, and at least two output results are obtained; one output result corresponds to one convolution kernel; and the feature map is determined based on the at least two output results and the bias corresponding to the output result.

[0009] According to the DPU-based model inference acceleration method provided in the application, the feature data in the single-channel image is used to determine target feature data corresponding to the convolution kernel, which includes: the feature data in the single-channel image is input into a data buffer row by row; the feature data is mapped in a memory based on a parallel output interface of the data buffer; the delay time length for obtaining the feature data is determined based on a calculation node of feature extraction in the double shrinkage array; and the target data is obtained by constantly offsetting the reading address in the memory based on the delay time length.

[0010] According to the DPU-based model inference acceleration method provided in the application, the feature map is determined based on the at least two output results and the bias corresponding to the output result, which includes: the first feature map in the single-channel image is determined based on the at least two output results and the bias corresponding to the output result; the first feature map is subjected to nonlinear transformation and quantization processing, and the second feature map is obtained; and the target feature map is obtained by performing pooling processing on the second feature map.

[0011] According to the DPU-based model inference acceleration method provided by the application, the second feature map is subjected to row-by-row maximum pooling based on a row buffer and a register to obtain first pooling data; the row buffer is implemented by a hardware description language by the DPU; the first pooling data is subjected to column-by-column maximum pooling based on the row buffer and the register to obtain second pooling data; and the target feature map is obtained based on the second pooling data.

[0012] According to the DPU-based model inference acceleration method provided by the application, the second feature map is subjected to row-by-row maximum pooling based on a row buffer and a register to obtain first pooling data, which comprises: inputting elements in the second feature map into the row buffer one by one in the horizontal direction; storing the elements in the second feature map in the register one by one in the horizontal direction based on a parallel output interface of the row buffer; when the next element is stored in the register, comparing an element value of the next element with an element value of a previous element stored in the register, and retaining a row element with a larger element value; and obtaining the first pooling data based on the row element.

[0013] According to the DPU-based model inference acceleration method provided by the application, the first pooling data is subjected to column-by-column maximum pooling based on a row buffer to obtain second pooling data, which comprises: inputting two adjacent rows of elements in the first pooling data into the row buffer; storing the two adjacent rows of elements in a register based on a parallel output interface of the row buffer; comparing an element value of a first row of elements in the two rows of elements with an element value of a corresponding second row of elements in the two rows of elements based on the memory, and retaining a column element with a larger element value; and obtaining the second pooling data based on the column element.

[0014] The application further provides a DPU-based model inference acceleration device, comprising the following modules.

[0015] A separation module is configured to separate colors of a to-be-inferred image to obtain at least two single-channel images.

[0016] An extraction module is configured to simultaneously extract features from each single-channel image based on a double-shrink array and at least two convolution kernels to obtain a feature map; the feature map is used for inference of the to-be-inferred image by the model; the double-shrink array is used for parallel calculation of matrix multiplication required for feature extraction; and the double-shrink array is implemented by a hardware description language by the DPU.

[0017] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the DPU-based model inference acceleration method according to any one of the above when executing the computer program.

[0018] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the DPU-based model inference acceleration method according to any one of the above.

[0019] The application further provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the DPU-based model inference acceleration method according to any one of the above.

[0020] The application provides a DPU-based model inference acceleration method, device, equipment and medium, which separates the colors of a to-be-inferred image to obtain at least two single-channel images, and synchronously extracts features of each single-channel image based on a double-shrink array and at least two convolution kernels to obtain a feature map, and the double-shrink array is a matrix multiplication required by DPU to realize parallel computation of feature extraction through a hardware description language. In this way, the to-be-inferred image is divided into multiple single-channel images to realize parallel processing of different color channels of image data, and a large-scale matrix is synchronously and parallelly computed by using a double-shrink array to increase parallel computation of a deep learning task and improve data throughput of a platform, thereby further accelerating the inference process of a model. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0022] Figure 1 is a flowchart of the DPU-based model inference method provided by the present application.

[0023] Figure 2 is a schematic diagram of a shrink array structure provided by the present application.

[0024] Figure 3 is a schematic diagram of a data input structure provided by the present application.

[0025] Figure 4 is a schematic diagram of the structure of an accelerator provided by the present application.

[0026] Figure 5 is a schematic diagram of resource utilization of different acceleration methods provided by the present application.

[0027] Figure 6 is a time efficiency diagram of different acceleration methods provided by the present application.

[0028] Figure 7 is a structural diagram of a DPU-based model inference acceleration device provided by the present application

[0029] Figure 8 is a structural diagram of an electronic device provided by the present application.

[0030] Reference signs:

[0031] 400: accelerator; 410: first convolution unit; 420: second convolution unit; 430: third convolution unit; 440: bias buffer; 450: adder; 460: LeakyReLU activation and quantization inference unit; 470: maximum pooling unit. DETAILED DESCRIPTION

[0032] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0033] The rapid development of large language models (large models) makes them a key element of machine learning tasks in various edge computing systems, such as autonomous driving, smart cities, and smart factories. These applications often need to meet the demands of high real-time performance and low power consumption, while traditional CPU and GPU-based computing platforms often struggle to meet these demands. Therefore, a new type of efficient low-power platform is needed to support and accelerate edge intelligence applications.

[0034] For a new, efficient low-power platform, the related art mainly includes the following methods: Method one, an AI intelligent computing system containing a DPU is integrated, a Bit stream file is generated and exported and imported into PetaLinux, a Linux system with a DPU AI intelligent computing module is built by using PetaLinux and burned into an SD card; the model is inferred, optimized and compiled into a DPU recognizable file on a general computing platform; the FPGA development board is started through the SD card, the file is run on the Linux system based on FPGA, DPU IP calling and model inference are realized, and the advantages of small development difficulty of the general computing platform and high performance, low power consumption and customizability of the FPGA platform are combined; Method two, the DPU deployment information is obtained by pre-analyzing the FPGA system on chip; according to the principles of different hardware components, the hardware factors that will affect the CNN inference of the FPGA system on chip are extracted, and the system hardware model and the DPU execution model are established through experimental verification; a general formula is established according to the model, and a general shared bus arbitration and memory access contention model of the FPGA system on chip is constructed; Method three, the convolution kernel candidate redundant data of the neural network to be calculated is detected, the candidate redundant data includes network topology structure and convolution kernel parameters; the neural network is optimized and reduced according to the candidate redundant data and re-inferred to obtain an optimized network; the calculation of the to-be-processed data is based on the optimized network; the re-inference process and the calculation process of the to-be-processed data are based on a DPU multi-core heterogeneous processor system. The present application accelerates the neural network calculation from two aspects of the network model itself and the network model calculation architecture, realizes the acceleration of the inference process of the neural network through the synergistic effect of software optimization design and hardware acceleration operation.

[0035] However, the defect of the above-mentioned method one is that the intelligent computing platform based on DPU is developed, but the system structure is not designed according to the calculation characteristics of the large model itself, and there is still room for improvement in terms of throughput. The defect of method two is that it mainly analyzes the real-time performance of the DPU deployment environment, focuses on the configuration parameters of the general shared bus arbitration and memory access contention model, but does not consider the calculation characteristics of the machine learning model itself, and mainly only considers the inference process. The data flow structure of the inference process of the large model is more complex, and the amount of operation increases, so this method cannot meet the high concurrency demand of the visual large model inference task. The defect of method three is that it mainly considers the heterogeneous computing characteristics of different layers of the neural network model, and uses the integrated DPU structure to accelerate the inference of the neural network model. However, method three ignores the potential of parallel computing of convolution layers in the same layer and does not consider the optimization and utilization of a large amount of cache data in the data buffer. Especially for large models, which are machine learning models with a huge number of parameters, cache optimization is the key to model inference acceleration.

[0036] In conclusion, the acceleration method in the related art improves the model inference process according to the deployment environment or the development performance of hardware, and does not consider the inference characteristics of the large model itself to design the architecture, resulting in that the throughput of the system needs to be improved and cannot meet the high concurrency requirement of the large model inference task.

[0037] Based on the above problems, the present application provides a model inference acceleration method based on DPU, which realizes parallel processing of different color channels of image data by dividing the to-be-inferred image into multiple single-channel images, and uses a double contraction array to synchronously perform parallel calculation on a large-scale matrix, increases the parallel calculation of the deep learning task, improves the data throughput of the platform, and further accelerates the inference process of the model.

[0038] The following will be described in combination with Figures 1-6 The model inference acceleration method based on DPU provided by the present application can be applied to the inference acceleration of any model, including but not limited to Convolutional Vision Transformers (CvT), a target detection model, etc. The present method can be applied to an AI accelerator system, and the execution subject of the present method can be an electronic device or a model inference acceleration method based on DPU arranged in the electronic device. The model inference acceleration method based on DPU can be realized by software, hardware or a combination of both.

[0039] Figure 1 is a flowchart of the model inference acceleration method based on DPU provided by the present application, as Figure 1 shown, the method comprises the following steps:

[0040] Step 101, separating the colors of the to-be-inferred data image to obtain at least two single-channel images.

[0041] Here, the to-be-inferred data image can be any image for identification, classification or other inference tasks.

[0042] Here, the colors of the to-be-inferred image can include red, green, blue and other color images, that is, a color image is a single-channel image. For example, a colored to-be-inferred image is divided into three single-channel images of red, green and blue.

[0043] Step 102, based on a double contraction array and at least two convolution kernels, synchronously performing feature extraction on each of the single-channel images to obtain a feature map.

[0044] The feature map is used for the model to infer the to-be-inferred image; the double contraction array is used for parallel calculation of matrix multiplication required for feature extraction, and the double contraction array is realized by a DPU through a hardware description language.

[0045] Here, the contraction array is also called a systolic array, which is an array composed of a group of processing elements (PEs). The double contraction array can also be called a double systolic array, where the contraction array is only a group of processing elements that calculate convolution operations, and the double contraction array is a group of processing elements that implement two convolution kernels to independently and synchronously perform convolution operations on a single-channel image.

[0046] It should be noted that although increasing the number of parallel processing elements in the contraction array can expand the parallelism in the contraction array and further improve the throughput, it will also increase the complexity of routing and require more DPU resources, that is, the inference of the large model will not increase linearly with the contraction array, therefore, through multiple tests of the potential relationship between additional parallelism and resource utilization, it is found that the double contraction array can balance parallelism and resource utilization.

[0047] Here, the convolution kernel can also be called a filter, which is generally 3 3 or 5 5 size. Among them, the 3 3 convolution kernel has 9 weights, and the 5 5 convolution kernel has 25 convolution kernels.

[0048] Here, the hardware description language can be Very-High-Speed Integrated Circuit Hardware Description Language (VHDL) or Verilog, which is limited by the present application.

[0049] It should be noted that by writing the hardware description language in the DPU, the DPU realizes the double contraction array.

[0050] It should be noted that each single-channel image has an independent convolution unit, such as convolution unit Convolution_R, convolution unit Convolution_G, and convolution unit Convolution_B, and each convolution unit has its weight buffer and feature map buffer.

[0051] In the embodiment of the present application, the color of the to-be-reasoned image is separated to obtain at least two single-channel images; feature extraction is performed on each single-channel image based on a double-shrink array and at least two convolution kernels to obtain a feature map, and the double-shrink array is a matrix multiplication operation required by DPU to realize parallel computation of feature extraction through a hardware description language. In this way, by separating the to-be-reasoned image into multiple single-channel images, parallel processing of different color channels of image data is realized, and by using the double-shrink array to synchronously and independently realize parallel computation of a large-scale matrix of a single-channel image by multiple convolution kernels, parallel computation of a deep learning task is increased, data throughput of a platform is improved, and the inference process of a model is further accelerated.

[0052] Further, the feature extraction based on the double-shrink array and the at least two convolution kernels and on each single-channel image to obtain a feature map comprises:

[0053] Based on the feature data in the single-channel image, target feature data corresponding to the convolution kernel is determined; the target feature data is data subjected to convolution operation with each convolution kernel respectively; the weight of each convolution kernel and the corresponding target feature data are subjected to convolution operation by a processing unit in the double-shrink array to obtain a convolution sub-result; the convolution sub-result is transmitted to a next processing unit for accumulation to obtain at least two output results; wherein one output result corresponds to one convolution kernel; and the feature map is determined based on the at least two output results and the bias corresponding to the output result.

[0054] Here, the feature data refers to each pixel value in the single-channel image, and it can be understood that each pixel value needs to be multiplied by a corresponding weight in the convolution kernel for convolution operation, and therefore, the same size of feature data as the convolution kernel needs to be selected for calculation each time.

[0055] Figure 2 is a shrink array structure diagram provided by the present application, as Figure 2 shown, W0-W8 are weights of the convolution kernel, and g0-g8 are gradient values; each processing element (PE) in the shrink array sequentially performs multiplication of input data (1-9) and corresponding weights (W0-W8) to obtain a convolution sub-result, and then transmits the convolution sub-result to a next PE for continuous operation of the next data; for example, the first group of data (such as F1 and W0) will complete multiplication calculation in the first PE, and then the result is transmitted to the next PE for continuous operation with the next group of data (such as F2 and W1); and the accumulation is sequentially performed until the multiplication of the last group of data (such as F9 and W8) is added to the result calculated by the previous PE to obtain an output result; and the formula executed by each PE can be the following formula (1):

[0056] P = FW + C (1)

[0057] Wherein, P represents a processing unit, F is feature map data, W is a weight of a convolution kernel, and C is a cumulative term.

[0058] Here, the bias can be obtained from a bias buffer.

[0059] It should be noted that the double shrinkage array is obtained by inputting the feature data into two groups of parallel PE arrays, so that two independent convolution kernels are calculated at the same time, and two output results are obtained.

[0060] In the embodiment of the application, the double shrinkage array can simultaneously calculate the convolution of two independent convolution kernels, thereby realizing parallel processing and double output. The advantage of the double shrinkage array lies in its parallel processing capability and double output characteristics. By simultaneously processing two groups of independent convolution kernels, the double shrinkage array can complete more computing tasks in the same time, thereby greatly improving the overall computing efficiency. In addition, the double output characteristics enable the double shrinkage array to simultaneously output two groups of convolution results, further improving the throughput of the system.

[0061] In an embodiment, based on the feature data in the single-channel image, the target feature data corresponding to the convolution kernel is determined, comprising:

[0062] The feature data in the single-channel image is input into the data buffer row by row; the feature data is mapped in the memory based on the parallel output interface of the data buffer; the delay duration for obtaining the feature data is determined based on the computing nodes of feature extraction in the double shrinkage array; and the target data is obtained by continuously offsetting the reading address in the memory based on the delay duration.

[0063] Here, the row buffer can be understood as a temporary memory containing an input interface and multiple output interfaces, and the feature data is mapped in the memory through the conversion of the row buffer.

[0064] Here, the delay duration can be the same as the duration between the computing nodes in the double shrinkage array, or the difference between the delay duration and the duration between adjacent computing nodes is less than a preset duration threshold, which is not limited in the application.

[0065] Specifically, the feature data in the single-channel image is input into the data buffer row by row, the feature data is mapped in the memory through the data buffer, the delay duration for obtaining the feature data is determined according to the computing nodes of feature extraction in the double shrinkage array, the target data is obtained by continuously offsetting the reading address in the memory based on the delay duration, and the output results corresponding to each convolution kernel are obtained by performing convolution operation on the target data and each convolution kernel, respectively.

[0066] Figure 3 is a schematic diagram of a data input structure provided by the present application, as shown in Figure 3 As shown, the feature map is the image operated with the convolution kernel, and the data input structure designs a row buffer (i.e., a data buffer), wherein the row buffer is composed of three First In First Out (FIFO) queues, and the row buffer is responsible for inputting image data row by row and transmitting three data points of the same row to the convolution operation through three parallel output interfaces.

[0067] For example, for the convolution unit calculation of a large model, the size of the convolution kernel is 3x3, and 9 data points need to be obtained from the feature map at the same time for calculation. In order to achieve this, the row buffer uses 9 pseudo-dual-port RAMs for data buffering, one port is used for reading data and the other port is used for writing data. When data is written, three numbers output from the row buffer are written into each group of three random access memories (RAMs), which store the same row data. After three write operations, a complete feature map will be formed in the RAM. Then the reading process begins, and through the offset reading address, the 9 data points required for convolution can be retrieved.

[0068] In the embodiment of the present application, the feature data is mapped in the RAM through the data buffer, the corresponding target data is obtained, and a delay mechanism is introduced according to the delay duration, which ensures that the data is exactly on the calculation node when the convolution calculation unit needs data, ensuring that the accelerator can perform convolution calculation under efficient data flow without causing the calculation unit to be idle due to data waiting, thereby optimizing the performance and optimizing the high throughput and low delay of the system.

[0069] In an embodiment, the determining the feature map based on the at least two output results and the bias corresponding to the output result comprises: determining a first feature map in the single-channel image based on the at least two output results and the bias corresponding to the output result; performing nonlinear transformation and quantization processing on the first feature map to obtain a second feature map; and performing pooling processing on the second feature map to obtain the target feature map.

[0070] It should be noted that a bias needs to be added after the convolution operation for adjusting the offset of the output result, wherein the bias can be any suitable value, such as 0 or a very small constant. In this way, the problem of gradient explosion or gradient disappearance can be avoided.

[0071] Here, the nonlinear transformation can be realized by an activation function, wherein the activation function includes but is not limited to a Leaky ReLU function, a Sigmoid function, a Tanh function, etc.

[0072] Here, the quantization processing refers to implementing all operations by using integer types or custom data types when implementing inference of the convolutional neural network in hardware, so as to reduce the size and calculation complexity of the model, and make the model run more efficiently on hardware.

[0073] Here, the pooling processing is to reduce the dimension of the feature map, highlight the most significant features, and provide a basis for subsequent layers or final prediction.

[0074] In the embodiment of the application, by adding bias to the output result, performing nonlinear change, quantization processing, pooling processing and some optimization operations, the hardware can efficiently execute the model and accelerate the inference of the model.

[0075] In an embodiment, the pooling processing on the second feature map to obtain the target feature map comprises:

[0076] Based on the row buffer and the register, the elements of the second feature map are row-wise maximum pooled to obtain first pooling data; the row buffer is implemented by the DPU through a hardware description language; based on the row buffer and the register, the first pooling data is column-wise maximum pooled to obtain second pooling data; and based on the second pooling data, the target feature map is obtained.

[0077] Here, the row buffer is a temporary memory including an input interface and multiple output interfaces, and elements in the feature map are stored in the register for pooling according to certain rules through the row buffer.

[0078] Specifically, the element values of each row in the feature map are first pooled to obtain first pooling data; and then the element values of each column in the first pooling data are pooled to obtain second pooling data, and the second pooling data is visualized to obtain the target feature map.

[0079] In the embodiment of the application, the maximum pooling is decomposed into row-wise maximum pooling and column-wise maximum pooling through the row buffer system, which effectively simplifies the pooling process, not only simplifies the hardware implementation, but also effectively utilizes the DPU resources to perform the pooling operation required by the neural network.

[0080] In an embodiment, the row-by-row maximum pooling of the elements of the second feature map based on the row buffer and the register to obtain the first pooled data comprises: inputting the elements in the second feature map into the row buffer in a horizontal direction one by one; storing the elements in the second feature map in a register in a horizontal direction one by one based on the parallel output interface of the row buffer; when the register stores a next element, comparing the element value of the next element with the element value of a previous element stored in the register, and retaining a row element with a larger element value; and obtaining the first pooled data based on the row element.

[0081] It should be noted that pooling is an operation for reducing the dimension of a feature map. For example, the input data before pooling is 480 256, and becomes 240 256 after pooling.

[0082] In the embodiment of the application, the row-by-row pooling process is effectively simplified by using the row buffer and the register, which not only simplifies the hardware implementation, but also effectively utilizes the DPU resources to perform the pooling operation required by the neural network.

[0083] In an embodiment, the column-by-column maximum pooling of the first pooled data based on the row buffer to obtain the second pooled data comprises: inputting two adjacent rows of elements in the first pooled data into the row buffer; storing the two adjacent rows of elements in a register based on the parallel output interface of the row buffer; comparing the element value of a first row of elements in the two rows of elements with the element value of a corresponding second row of elements in the two rows of elements based on the memory, and retaining a column element with a larger element value; and obtaining the second pooled data based on the column element.

[0084] For example, the data before column pooling is 240 256, and the data after column pooling is 240 128.

[0085] In the embodiment of the application, the column-by-column pooling process is effectively simplified by using the row buffer and the register, which not only simplifies the hardware implementation, but also effectively utilizes the DPU resources to perform the pooling operation required by the neural network.

[0086] In the embodiment of the application, the DPU-based model inference acceleration method can be applied to an accelerator, which can be constructed by a hardware description language and implemented by DPU hardware.

[0087] For example, Figure 4 is a structural schematic diagram of the accelerator provided by the application, as Figure 4As shown, accelerator 400 comprises: a first convolution unit 410: for convolving the red single channel image; a second convolution unit 420: for convolving the green single channel image; a third convolution unit 430: for convolving the blue single channel image; a bias buffer 440 for applying a bias to the results from the contraction array; an adder 450: for combining the contributions of each convolution unit, adding the bias before passing the result to the activation function; a LeakyReLU activation and quantization inference unit 460: for implementing the Leaky ReLU function, introducing nonlinearity in the processing pipeline, and quantizing to improve inference efficiency; a max pooling unit 470 for reducing the dimensionality of the feature map, highlighting the most salient features, and preparing for subsequent layers or final prediction. Among them, each convolution unit comprises a weight buffer: for storing the weights of the convolution kernel and providing them to the contraction array as needed for convolution operations; a feature map buffer: for temporarily saving input data, which can be the original input image or the output from the previous layer, in preparation for processing by the contraction array; a contraction array: for performing matrix multiplication operations to achieve the parallel computing required in deep learning tasks. In addition to the units shown in Figure 4

[0088] It should be noted that the weight buffer is a key component designed to store the weights of the convolution layer to be processed. It is implemented using read-only memory (ROM), in which the weight data is preloaded into a data file in the ROM. This setting ensures that the weight data can be used at startup, eliminating the need for runtime loading and improving the efficiency of the accelerator. In the context of the ability of the computing unit to process two convolution kernels simultaneously, each kernel has 9 values, and a total of 18 weight values need to be output. These 18 values are connected into a binary string. Once read from the ROM, this binary string is split and assigned to 18 different variables, which is consistent with the requirement of the architecture to output kernel weight data in a structured and orderly manner. After the convolution operation of the three rows of feature maps is completed, the system must switch to the next set of convolution kernel weights to continue processing. This is seamlessly done through a carefully coordinated control mechanism, which marks the completion of the feature map processing. After receiving this signal, the weight buffer module is updated to the next pair of kernel weights, preparing for the next calculation period. This process is repeated, allowing all one-dimensional cumulative calculation kernels for convolution to be effectively executed.

[0089] ​The data input structure (i.e., the data buffer mentioned above), the weight buffer structure, the double systolic array structure, and the pooling calculation structure are used in the adder to perform parallel acceleration of convolution operations. The core of it is the double systolic array architecture, which has efficient data transmission and parallel processing capabilities, and is particularly suitable for the sequential data flow required by visual large models. The double systolic array usually has a low latency due to its pipelined nature, allowing continuous data input and output. Compared with the traditional systolic array structure, the double systolic array structure improves the throughput by expanding the parallelism within the systolic array. The adder tree, as another architecture, focuses on hierarchical aggregation of multiplier results. This has advantages in reducing logic depth and potentially increasing clock speed. However, compared with the systolic array architecture, the adder tree architecture may introduce synchronization problems and may require more complex control logic to effectively manage data flow.

[0090] In order to verify and test the effectiveness of the model inference acceleration method based on DPU proposed by the present application, the following simulation experiment is carried out.

[0091] The experiment is developed using Xilinx Vivado 2023.2, and does not require the deployment of physical hardware. The simulation environment provided by Vivado allows comprehensive testing and verification of the design of the DPU-based artificial intelligence accelerator. The Vivado simulation environment includes the Vivado 2023.2 tool, which utilizes the suite features of Vivado, such as the logic simulator, to perform inference behavior and timing simulation. A VHDL / Verilog test platform is developed to simulate the operation of artificial intelligence models and test the performance of the DPU design architecture. Multiple iterations are run to evaluate functionality, timing, and potential boundary conditions.

[0092] Figure 5 is a schematic diagram of resource utilization of different acceleration methods provided by the present application, as Figure 5 shown, the horizontal axis represents the expected resource utilization (%), and the vertical axis includes Flip-Flop (FF), Global Buffer (BUFG), Look-Up Table Random Access Memory (LUTRAM), Look-Up Table (LUT), gray represents the model inference acceleration method based on DPU provided by the present application, orange represents the adder tree (Adder), and blue represents the systolic array, as Figure 5As can be seen, the resource usage of the addition tree and dual-shrink array designs is slightly higher compared to the original shrink array. In particular, the utilization of flip-flops is significantly increased in the dual-shrink array, indicating the use of more complex control logic to manage the additional parallel computation. While the use of lookup tables (LUTs) and global buffers (BUFGs) remains relatively consistent across different designs, the significant increase in flip-flop usage in the dual-shrink array design may affect scalability and maximum frequency. The resource utilization graph clearly shows that the dual-shrink array architecture consumes more triggers compared to the original shrink array and addition tree architectures. This increase in resource usage is consistent with the design's intention to enhance parallel computing capabilities.

[0093] Figure 6 These are schematic diagrams illustrating the time efficiency under different acceleration methods provided by this invention, such as... Figure 6 As shown, the blue line represents the CPU, the dark orange line represents the shrinking array, the gray line represents the addition tree, and the light orange line represents the dual shrinking array. The horizontal axis represents efficiency, and the vertical axis represents time (ms). Figure 6 The following conclusions can be drawn:

[0094] Shrinking Arrays: The original shrinking arrays were specifically designed for convolution computations, offering up to 5 times faster computation time than CPUs. Their architecture was designed for efficient data flow and parallel processing, which aligns well with the operations of convolutional neural networks. However, while shrinking arrays are faster than CPUs, they may not offer the same level of performance improvement in tasks outside their specialization, such as multiplications or algorithms that do not benefit from their parallel data flow.

[0095] Addition Trees: Addition trees are a common parallel computing architecture in DPUs, offering approximately double the performance of CPUs. For DPU developers, this can be a more familiar architecture and may provide broader applicability for a variety of computational tasks. Despite its advantages, addition trees do not outperform shrinking arrays in this specific application. This is likely because convolutional data processing is less optimized for certain tasks, suggesting that addition trees may be better suited for other types of computation.

[0096] Dual Shrink Array: This architecture is 10 times faster than a CPU, highlighting the significant advantages of parallel processing in a DPU. By doubling the capacity for parallel computing, it greatly improves data throughput and operation speed. However, the increased parallelism leads to higher resource consumption, which can cause scalability issues on resource-constrained DPUs. There is a diminishing returns point; additional parallelism does not equate to a proportional performance improvement, especially when the DPU's communication bandwidth becomes a bottleneck.

[0097] In the specific context of accelerating CvT convolution layers, the double-squeeze array provides the best performance, demonstrating that the tradeoff of increased resource utilization is justified. The results show that for applications where speed is critical and resources are available, the double-squeeze array architecture can be the best choice. Conversely, when resource conservation is more critical, or for applications that require more general-purpose computing tasks, the squeeze array or add tree can be more appropriate.

[0098] In the embodiments of the present application, modules in visual large models that can be accelerated based on DPU hardware are studied for large model inference tasks, and parallel acceleration and scheduling structures based on DPU are designed for convolution operations therein to realize flexible allocation of resources; an accelerator architecture is designed, including data input structure, weight buffer structure, double-squeeze array structure and pooling calculation structure, and different accelerator structures are integrated to expand the squeeze array by increasing parallel computing operations and improve throughput; a VHDL / Verilog test platform is developed to simulate the computing operations of large model inference, and a large model inference acceleration instance is generated for DPU design.

[0099] The DPU-based model inference acceleration device provided by the present application is described below, and the DPU-based model inference acceleration device described below can be referred to in conjunction with the DPU-based model inference acceleration method described above.

[0100] Figure 7 is a structural schematic diagram of the DPU-based model inference acceleration device provided by the present application, as Figure 7 shown, the DPU-based model inference acceleration device 700 comprises:

[0101] A separation module 701 is configured to separate the colors of a to-be-inferred image to obtain at least two single-channel images.

[0102] An extraction module 702 is configured to simultaneously extract features from each single-channel image based on a double-squeeze array and at least two convolution kernels to obtain a feature map; the feature map is used for inference of the to-be-inferred image by the model; the double-squeeze array is used for parallel computation of matrix multiplication required for feature extraction, and the double-squeeze array is realized by a hardware description language.

[0103] In some embodiments, the extraction module 702 is specifically configured to: determine target feature data corresponding to the convolution kernel based on feature data in the single-channel image; the target feature data is data respectively subjected to convolution operation with each of the convolution kernel; perform convolution operation on the weight of each of the convolution kernel and the corresponding target feature data based on the processing unit in the double contraction array to obtain a convolution sub-result; pass the convolution sub-result to the next processing unit for accumulation to obtain at least two output results; wherein one of the output results corresponds to one of the convolution kernel; and determine the feature map based on the at least two output results and the bias corresponding to the output result.

[0104] In some embodiments, the extraction module 702 is further specifically configured to: input feature data in the single-channel image into a data buffer row by row; map the feature data in the memory based on a parallel output interface of the data buffer; determine a delay duration for obtaining the feature data based on a calculation node of feature extraction in the double contraction array; and constantly offset a reading address in the memory based on the delay duration to obtain the target data.

[0105] In some embodiments, the extraction module 702 is further specifically configured to: determine a first feature map in the single-channel image based on the at least two output results and the bias corresponding to the output result; perform nonlinear transformation and quantization processing on the first feature map to obtain a second feature map; and perform pooling processing on the second feature map to obtain the target feature map.

[0106] In some embodiments, the extraction module 702 is further specifically configured to: perform row-by-row maximum pooling on elements of the second feature map based on a row buffer and a register to obtain first pooling data; perform column-by-column maximum pooling on the first pooling data based on the row buffer and the register to obtain second pooling data; and obtain the target feature map based on the second pooling data.

[0107] In some embodiments, the extraction module 702 is further specifically configured to: input elements in the second feature map into the row buffer one by one in a horizontal direction; store the elements in the second feature map in a register one by one in a horizontal direction based on a parallel output interface of the row buffer; when the register stores a next element, compare an element value of the next element with an element value of a previous element stored in the register, and retain a row element with a larger element value; and obtain the first pooling data based on the row element.

[0108] Figure 8 is a structural schematic diagram of an electronic device provided by the present application, as Figure 8As shown, the electronic device can include a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 complete mutual communication through the communications bus 840. The processor 810 can invoke a logic instruction in the memory 830 to execute a DPU-based model inference acceleration method, which includes: separating colors of a to-be-inferred image to obtain at least two single-channel images; synchronously performing feature extraction on each of the single-channel images based on a double-shrink array and at least two convolution kernels to obtain a feature map; the feature map is used for the model to infer the to-be-inferred image; the double-shrink array is used for parallel calculation of matrix multiplication required for feature extraction, and the double-shrink array is realized by a hardware description language by the DPU.

[0109] In addition, the logic instruction in the memory 830 described above can be realized in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0110] On the other hand, the present application also provides a computer program product, which includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, and the computer can execute the DPU-based model inference acceleration method provided by the above-mentioned method, which includes: separating colors of a to-be-inferred image to obtain at least two single-channel images; synchronously performing feature extraction on each of the single-channel images based on a double-shrink array and at least two convolution kernels to obtain a feature map; the feature map is used for the model to infer the to-be-inferred image; the double-shrink array is used for parallel calculation of matrix multiplication required for feature extraction, and the double-shrink array is realized by a hardware description language by the DPU.

[0111] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements a DPU-based model inference acceleration method provided by each of the above methods, the method comprising: separating colors of a to-be-inferred image to obtain at least two single-channel images; performing feature extraction on each of the single-channel images based on a double-shrink array and at least two convolution kernels to obtain a feature map; the feature map being used by the model to infer the to-be-inferred image; the double-shrink array being used for parallel calculation of matrix multiplication required for feature extraction, and the double-shrink array being implemented by the DPU through a hardware description language.

[0112] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0113] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software and the necessary general hardware platform, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0114] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A DPU-based model inference acceleration method, characterized in that, The method comprises: separating colors of a to-be-reasoned data image to obtain at least two single-channel images; inputting feature data in the single-channel images into a data buffer row by row; mapping the feature data in the memory based on a parallel output interface of the data buffer; determining a delay duration for obtaining the feature data based on a computing node of feature extraction in a double-shrink array; continuously offsetting a reading address in the memory based on the delay duration to obtain target feature data; the target feature data is data subjected to convolution operation with respective convolution kernels; performing convolution operation on weights of respective convolution kernels and corresponding target feature data based on processing units in the double-shrink array to obtain convolution sub-results; delivering the convolution sub-results to a next processing unit for accumulation to obtain at least two output results; one of the output results corresponds to one of the convolution kernels; determining a first feature map in the single-channel image based on the at least two output results and a bias corresponding to the output results; performing nonlinear transformation and quantization processing on the first feature map to obtain a second feature map; performing pooling processing on the second feature map to obtain the target feature map; the feature map is used for reasoning of the to-be-reasoned data image by a model; the double-shrink array is used for parallel computation of matrix multiplication operation required by feature extraction; and the double-shrink array is realized by a DPU through a hardware description language.

2. The DPU-based model inference acceleration method of claim 1, wherein, The method of performing pooling processing on the second feature map to obtain the target feature map comprises: performing row-by-row maximum pooling on elements of the second feature map based on a row buffer and a register to obtain first pooling data; the row buffer is realized by the DPU through the hardware description language; performing column-by-column maximum pooling on the first pooling data based on the row buffer and the register to obtain second pooling data; obtaining the target feature map based on the second pooling data.

3. The DPU-based model inference acceleration method of claim 2, wherein, The method of performing row-by-row maximum pooling on elements of the second feature map based on a row buffer and a register to obtain first pooling data comprises: inputting elements in the second feature map into the row buffer one by one in a horizontal direction; storing the elements in the second feature map in a register one by one in a horizontal direction based on a parallel output interface of the row buffer; when a next element is stored in the register, comparing an element value of the next element with an element value of a previous element stored in the register, and retaining a row element with a larger element value; obtaining the first pooling data based on the row element.

4. The DPU-based model inference acceleration method of claim 3, wherein, The method of performing column-by-column maximum pooling on the first pooling data based on a row buffer to obtain second pooling data comprises: inputting two adjacent rows of elements in the first pooling data into the row buffer; storing the two adjacent rows of elements in a register based on a parallel output interface of the row buffer; comparing an element value of a first row of elements in the two rows of elements with an element value of a corresponding second row of elements in the two rows of elements based on the memory, and retaining a column element with a larger element value; obtaining the second pooling data based on the column element.

5. A DPU-based model inference acceleration device, comprising: The method comprises: The separation module is used to separate the colors of the image to be inferred, obtaining at least two single-channel images; An extraction module is used to input the feature data in the single-channel image line by line into a data buffer; and to map the feature data into a memory based on the parallel output interface of the data buffer. Based on the feature extraction computing nodes in the dual contraction array, the delay time for acquiring the feature data is determined; based on the delay time, the read address is continuously offset in the memory to obtain the target feature data; the target feature data is data that has undergone convolution operations with each convolution kernel; based on the processing units in the dual contraction array, the weights of each convolution kernel and the corresponding target feature data are convolved to obtain convolution sub-results; the convolution sub-results are passed to the next processing unit for accumulation to obtain at least two output results; wherein, one output result corresponds to one convolution kernel; based on the at least two output results and the bias corresponding to the output results, a first feature map in the single-channel image is determined; the first feature map is subjected to nonlinear transformation and quantization processing to obtain a second feature map; the second feature map is subjected to pooling processing to obtain the target feature map; the feature map is used by the model to infer the data image to be inferred; the dual contraction array is used to perform matrix multiplication operations required for feature extraction in parallel, and the dual contraction array is implemented by the DPU through a hardware description language.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the DPU-based model inference acceleration method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the DPU-based model inference acceleration method as described in any one of claims 1 to 4.

Images