Convolution optimization method and device based on DSP architecture, equipment and storage medium

Abstract

The invention particularly relates to a convolution optimization method and device based on a DSP framework, equipment and a storage medium, and the method comprises the steps: distributing a high-bandwidth programmable shared cache at a DSP end based on a preset distribution strategy, and obtaining the capacity of the high-bandwidth programmable shared cache distributed by each core; taking a pointer starting point of the current channel as a reference, dynamically segmenting a preset input feature map according to the number of cores and offsetting a first address to obtain a core data total load and a data processing starting point corresponding to each core; based on the high-bandwidth programmable shared cache capacity allocated to each core, calculating a query round needing to be executed when traversing the total core data load corresponding to each core; and selecting each query round, applying a preset declaration strategy and a declaration result to obtain the shape of a convolution kernel, obtaining the explained shape of the convolution kernel based on a preset column feature graph, and obtaining a final convolution optimization result based on a preset general matrix multiplication strategy. Thus, the efficiency of convolution is enhanced.

Description

Technical Field

[0001] The present application relates to the field of digital signal processing technology, and in particular to a convolution optimization method, apparatus, device and storage medium based on a DSP architecture. Background Art

[0002] Among the current optimization methods for convolution operators, the mainstream optimization technique is the im2col method, such as Figure 1 As shown, Figure 1 This is a schematic diagram of the standard im2col method in the related art; that is, the input tensor is expanded to columns according to the sliding window coverage area to become a two-dimensional matrix, and then the expanded input tensor (which is a 2D matrix) is multiplied by the convolution kernel expanded in the same way to obtain the convolution result; when processing larger matrices, the above im2col conversion method is also followed.

[0003] In related technologies, for data with more dimensions (such as 3D or higher dimensional tensors), it is possible to expand on the existing two-dimensional matrix. Specifically, for each additional dimension, it can be regarded as a series of independent two-dimensional matrices, and these matrices are sequentially appended to the existing structure. Take a 3x3 convolution kernel as an example, such as Figure 2 As shown, Figure 2 A schematic diagram of the im2col algorithm for a multi-dimensional 3x3 convolution kernel in related art. This process can be understood as applying the same im2col transformation to each depth slice of the input tensor, then stacking all the transformed two-dimensional matrices sequentially to form an expanded, large matrix suitable for efficient matrix operations. The im2col algorithm in related art pads data within the sliding window itself, sequentially reading each element in the current sliding window and then padding each element in the corresponding column of the result matrix to obtain the final result.

[0004] However, for Digital Signal Processor (DSP) devices, the original algorithm cannot take advantage of its local cache devices and high concurrency. In addition, due to the overlap between different sliding windows, the traditional algorithms in related technologies cannot cleanly divide the data. In any case, there will always be data falling in the overlapping area that needs to be read a second time and written into a new temporary space, which will make the operation efficiency lower and needs to be solved urgently. Summary of the Invention

[0005] The present application provides a convolution optimization method, apparatus, device and storage medium based on DSP architecture to solve the problems of high memory footprint, poor data continuity and low concurrency utilization in related technologies, and enhance the efficiency of convolution.

[0006] The first embodiment of the present application provides a convolution optimization method based on a DSP architecture, comprising the following steps:

[0007] Allocating the high-bandwidth programmable shared cache on the DSP device side to each core of the DSP device side based on a preset allocation strategy to obtain a high-bandwidth programmable shared cache capacity allocated to each core;

[0008] Determine a current channel in a preset input feature map, and dynamically divide the preset input feature map according to the number of cores on the DSP device end based on the starting point of the pointer of the current channel, and retain the whole row to obtain the total core data load corresponding to each core;

[0009] The preset input feature map is offset in the first address according to the number of cores of the DSP device, and based on the offset address, a data processing starting point corresponding to each core is obtained;

[0010] Based on the high-bandwidth programmable shared cache capacity allocated to each core, calculating the number of query rounds required to be executed when traversing the total core data load corresponding to each core, to obtain multiple query rounds;

[0011] Select each query round and apply the preset declaration strategy, and based on the declaration result, obtain the shape of the convolution kernel, and based on the preset columnized feature map, interpret the shape of the convolution kernel to obtain the shape of the interpreted convolution kernel, and based on the preset general matrix multiplication strategy, process the shape of the interpreted convolution kernel and the preset columnized feature map to obtain the final convolution optimization result.

[0012] Optionally, selecting each query round and applying a preset declaration strategy includes:

[0013] Determining a current query round from the multiple query rounds, and calculating a single data transfer amount based on the current query round;

[0014] Calculate the current local memory address offset value based on the current query round;

[0015] Determine the starting position of the preset input feature map according to the pointer starting point of the current channel, the total core data load corresponding to each core, the current local memory address offset value, and the overall starting pointer of the preset input feature map, and based on the starting position, move the data from the DRAM to the high-bandwidth programmable shared cache through the DMA BUS according to the amount of data transferred in a single time;

[0016] Selecting each row of data moved into the cache segment allocated on the high-bandwidth programmable shared cache, and calculating a corresponding row number in the preset input feature map based on the currently selected data row;

[0017] Calculating the number of repetitions of the corresponding repeated data sub-block on the columnized feature map based on the currently selected data row and the corresponding row number in the preset input feature map;

[0018] Select each repetition number, calculate the first address corresponding to the columnized feature map, and determine whether the current repeated data sub-block is included in the output based on the Stride parameter declared in the current convolution parameters;

[0019] Select each row in the current repeated data sub-block, shift the address on the high-bandwidth programmable shared cache to the current local memory address offset value by a preset position and a preset step size, and asynchronously transfer the Stride parameter via the DMABUS bus based on a preset transfer strategy. Optionally,

[0020] Optionally, the preset allocation strategy is:

[0021]

[0022] Among them, Core Cache The high-bandwidth programmable shared cache capacity allocated to each core, L Cache is the total capacity of the high-bandwidth programmable shared cache, N Core is the number of cores on the DSP device.

[0023] Optionally, the total core data load corresponding to each core is:

[0024]

[0025] Among them, Kernel Load Img is the total core data load corresponding to each core, Width is the input feature map width, Img Height is the input feature map height, N Core is the number of cores on the DSP device side, C i is the i-th core.

[0026] Optionally, the data processing starting point corresponding to each core is:

[0027]

[0028] Among them, Kernel Offset is the data processing starting point corresponding to each core, C i is the i-th core, Imgwidth is the input feature map width, Img Height is the input feature map height, N Core The number of cores on the DSP device.

[0029] A second embodiment of the present application provides a convolution optimization device based on a DSP architecture, comprising:

[0030] an allocation module, configured to allocate the high-bandwidth programmable shared cache of the DSP device end to each core of the DSP device end based on a preset allocation strategy, thereby obtaining the high-bandwidth programmable shared cache capacity allocated to each core;

[0031] A segmentation module is used to determine a current channel in a preset input feature map, and dynamically segment the preset input feature map according to the number of cores on the DSP device side based on the pointer starting point of the current channel, and retain the whole row to obtain the total core data load corresponding to each core;

[0032] An offset module is used to offset the first address of the preset input feature map according to the number of cores on the DSP device end, and obtain the data processing starting point corresponding to each core based on the offset address;

[0033] a calculation module, configured to calculate, based on the high-bandwidth programmable shared cache capacity allocated to each core, a number of query rounds required to be executed when traversing a total core data load corresponding to each core, to obtain a plurality of query rounds;

[0034] A determination module is used to select each query round and apply a preset declaration strategy, and based on the declaration result, obtain the shape of the convolution kernel, and based on the preset columnized feature map, interpret the shape of the convolution kernel to obtain the shape of the interpreted convolution kernel, and based on the preset general matrix multiplication strategy, process the shape of the interpreted convolution kernel and the preset columnized feature map to obtain the final convolution optimization result.

[0035] Optionally, the determining module is specifically configured to:

[0036] Determining a current query round from the multiple query rounds, and calculating a single data transfer amount based on the current query round;

[0037] Calculate the current local memory address offset value based on the current query round;

[0038] Determine the starting position of the preset input feature map according to the pointer starting point of the current channel, the total core data load corresponding to each core, the current local memory address offset value, and the overall starting pointer of the preset input feature map, and based on the starting position, move the data from the DRAM to the high-bandwidth programmable shared cache through the DMA BUS according to the amount of data transferred in a single time;

[0039] Selecting each row of data moved into the cache segment allocated on the high-bandwidth programmable shared cache, and calculating a corresponding row number in the preset input feature map based on the currently selected data row;

[0040] Calculating the number of repetitions of the corresponding repeated data sub-block on the columnized feature map based on the currently selected data row and the corresponding row number in the preset input feature map;

[0041] Select each repetition number, calculate the first address corresponding to the columnized feature map, and determine whether the current repeated data sub-block is included in the output based on the Stride parameter declared in the current convolution parameters;

[0042] Each row in the current repeated data sub-block is selected, the address on the high-bandwidth programmable shared cache is shifted backward by a preset position and a preset step size as the current local memory address offset value, and the Stride parameter is asynchronously transferred through the DMABUS bus based on a preset transfer strategy.

[0043] Optionally, the preset allocation strategy is:

[0044]

[0045] Among them, Core Cache The high-bandwidth programmable shared cache capacity allocated to each core, L Cache is the total capacity of the high-bandwidth programmable shared cache, N Core is the number of cores on the DSP device.

[0046] Optionally, the total core data load corresponding to each core is:

[0047]

[0048] Among them, Kernel Load Img is the total core data load corresponding to each core, Width is the input feature map width, Img Height is the input feature map height, N Core is the number of cores on the DSP device side, C i is the i-th core.

[0049] Optionally, the data processing starting point corresponding to each core is:

[0050]

[0051] Among them, Kernel offset is the data processing starting point corresponding to each core, C i is the i-th core, Img width is the input feature map width, Img Height is the input feature map height, N Core The number of cores on the DSP device.

[0052] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the convolution optimization method based on the DSP architecture as described in the above embodiment.

[0053] The fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored. The program is executed by a processor to implement the convolution optimization method based on the DSP architecture as described in the above embodiment.

[0054] The fifth embodiment of the present application provides a computer program product, which stores a computer program. When the program is executed by a processor, it implements the convolution optimization method based on the DSP architecture as described in the above embodiment.

[0055] Therefore, the embodiment of the present application allocates the high-bandwidth programmable shared cache on the DSP side based on the preset allocation strategy to obtain the high-bandwidth programmable shared cache capacity allocated to each core, takes the pointer starting point of the current channel as the benchmark, dynamically divides the preset input feature map according to the number of cores and the first address offset, obtains the total core data load and data processing starting point corresponding to each core, and then calculates the query rounds that need to be executed when traversing the total core data load corresponding to each core based on the high-bandwidth programmable shared cache capacity allocated to each core; selects each query round, and applies the preset declaration strategy and declaration result to obtain the shape of the convolution kernel, and obtains the shape of the interpreted convolution kernel based on the preset columnized feature map, and obtains the final convolution optimization result based on the preset general matrix multiplication strategy. Thus, the problems of high memory footprint, poor data continuity and low concurrency utilization in the related technology are solved, and the efficiency of convolution is enhanced.

[0056] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0058] Figure 1 Schematic diagram of the standard im2col method in the related art;

[0059] Figure 2 Schematic diagram of the im2col algorithm of the multi-dimensional 3x3 convolution kernel in the related art;

[0060] Figure 3 A flowchart of a convolution optimization method based on a DSP architecture provided according to an embodiment of the present application;

[0061] Figure 4 A schematic diagram of relevant parameters of the im2col method of a convolution optimization method based on a DSP architecture provided according to one embodiment of the present application;

[0062] Figure 5 This is a schematic diagram showing an overall optimization of the im2col method of a convolution optimization method based on a DSP architecture according to one embodiment of the present application;

[0063] Figure 6 A schematic diagram of a DMA device architecture for a convolution optimization method based on a DSP architecture according to one embodiment of the present application;

[0064] Figure 7 A schematic diagram of an im2col method based on a DSP device and a convolution optimization method based on a DSP architecture according to one embodiment of the present application;

[0065] Figure 8 Schematic diagram of a convolution optimization device based on a DSP architecture according to an embodiment of the present application;

[0066] Figure 9 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0067] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0068] The following describes the convolution optimization method, device, equipment and storage medium based on the DSP architecture of the present application embodiment with reference to the accompanying drawings. In response to the problems of high memory footprint, poor data continuity and low concurrent utilization in the related art mentioned in the above background technology, the present application provides a convolution optimization method based on the DSP architecture, in which the embodiment of the present application allocates the high-bandwidth programmable shared cache on the DSP side based on a preset allocation strategy to obtain the high-bandwidth programmable shared cache capacity allocated to each core, and takes the pointer starting point of the current channel as the benchmark, dynamically divides the preset input feature map according to the number of cores and the first address offset, obtains the total core data load and data processing starting point corresponding to each core, and then calculates the query rounds required to execute when traversing the total core data load corresponding to each core based on the high-bandwidth programmable shared cache capacity allocated to each core; selects each query round, and applies the preset declaration strategy and declaration result to obtain the shape of the convolution kernel, and obtains the shape of the interpreted convolution kernel based on the preset columnized feature map, and obtains the final convolution optimization result based on the preset general matrix multiplication strategy. Thus, the problems of high memory footprint, poor data continuity and low concurrent utilization in the related art are solved, and the efficiency of convolution is enhanced.

[0069] Specifically, Figure 3 A flowchart of a convolution optimization method based on DSP architecture provided in an embodiment of the present application.

[0070] like Figure 3 As shown, the convolution optimization method based on DSP architecture includes the following steps:

[0071] In step S301 , based on a preset allocation strategy, the high-bandwidth programmable shared cache on the DSP device side is allocated to each core on the DSP device side, thereby obtaining the high-bandwidth programmable shared cache capacity allocated to each core.

[0072] Among them, the high-bandwidth programmable shared cache capacity (High-Bandwidth Programmable Shared Cache Capacity, L_Cache) refers to the total storage space size of the high-bandwidth programmable shared cache in the DSP architecture.

[0073] Optionally, in some embodiments, the preset allocation strategy is:

[0074]

[0075] Among them, Core Cache High-bandwidth programmable shared cache capacity allocated to each core, L Cache is the total capacity of the high-bandwidth programmable shared cache, N Core The number of cores on the DSP device.

[0076] It is understandable that the High-Bandwidth Programmable Shared Cache (HBSC) on the DSP device side is divided based on the preset allocation strategy so that each DSP Core can be allocated to the Core Cache Equally, each core is allocated high-bandwidth programmable shared cache capacity.

[0077] In step S302, the current channel is determined in the preset input feature map, and based on the pointer starting point of the current channel, the preset input feature map is dynamically divided according to the number of cores on the DSP device side, and retained as a whole row to obtain the total core data load corresponding to each core.

[0078] Optionally, in some embodiments, the total core data load corresponding to each core is:

[0079]

[0080] Among them, Kernel Load The total core data load corresponding to each core, Img Width is the input feature map width, Img Height is the input feature map height, N Core is the number of cores on the DSP device side, C i is the i-th core.

[0081] It is understandable that the embodiment of the present application can simply calculate the starting point of the continuous cache space that each DSP Core can be allocated, which is defined as Cache Ptr Select a channel on the input feature map, and the subsequent steps are based on the selected current channel. i As a basis for explanation; the current channel pointer starting point Channel Offset As a benchmark, the Data Ptr As the overall starting pointer of the input feature map. The input feature map is dynamically divided according to the number of DSP Cores and retained as a whole row to obtain the total core data load corresponding to each DSP Core, which is defined as Kernel Load .

[0082] In step S303, the first address of the preset input feature map is offset according to the number of cores on the DSP device side, and based on the offset address, the data processing starting point corresponding to each core is obtained.

[0083] The first address offset refers to the displacement of the starting memory address of the input data block allocated to each processing core (DSP Core) relative to the global base address in DSP multi-core parallel computing.

[0084] Optionally, in some embodiments, the data processing starting point corresponding to each core is:

[0085]

[0086] Among them, Kernel Offset is the data processing starting point for each core, C i is the i-th core, Img Width is the input feature map width, Img Height is the input feature map height, N Core The number of cores on the DSP device.

[0087] It can be understood that the input feature map is offset according to the first address of each DSP Core so that the offset address reflects the starting point of the data that the current DSP Core needs to process, which is defined as Kernel offset ;

[0088] In step S304 , based on the high-bandwidth programmable shared cache capacity allocated to each core, the query rounds required to be executed when traversing the total core data load corresponding to each core are calculated to obtain multiple query rounds.

[0089] Specifically, the high-bandwidth programmable shared cache capacity allocated to each core is calculated based on the current Core Cache , complete the Kernel Load The number of query rounds that need to be executed during the traversal is defined as Kernel Pass .

[0090] In step S305, each query round is selected, and the preset declaration strategy is applied. Based on the declaration result, the shape of the convolution kernel is obtained, and based on the preset columnized feature map, the shape of the convolution kernel is interpreted to obtain the shape of the interpreted convolution kernel. Based on the preset general matrix multiplication strategy, the shape of the interpreted convolution kernel and the preset columnized feature map are processed to obtain the final convolution optimization result.

[0091] Optionally, in some embodiments, each query round is selected and a preset declaration strategy is applied, including: determining the current query round from multiple query rounds, and calculating the amount of data transferred in a single time based on the current query round; calculating the current local memory address offset value based on the current query round; determining the starting position of the preset input feature map based on the pointer starting point of the current channel, the total core data load corresponding to each core, the current local memory address offset value and the overall starting pointer of the preset input feature map, and based on the starting position, moving the data from the DRAM to the high-bandwidth programmable shared cache via DMABUS according to the amount of data transferred in a single time; selecting each cache segment allocated on the high-bandwidth programmable shared cache to be moved in. A row of data is selected, and based on the currently selected data row, the corresponding row number in the preset input feature map is calculated; based on the currently selected data row and the corresponding row number in the preset input feature map, the number of repetitions of the corresponding repeated data sub-block on the columnized feature map is calculated; each number of repetitions is selected, the head address corresponding to the columnized feature map is calculated, and based on the Stride parameter declared in the current convolution parameter, it is determined whether the current repeated data sub-block is included in the output; each row in the current repeated data sub-block is selected, the address on the high-bandwidth programmable shared cache is offset backward by a preset position and a preset step size as the current local memory address offset value, and based on the preset transmission strategy, the Stride parameter is asynchronously transmitted through the DMABUS bus.

[0092] Specifically, select each Kernel Pass , based on the current Kernel Pass Calculate the amount of data that can be moved at a time, defined as Kernel PassLoad , the calculation method is as follows:

[0093]

[0094] Among them, Kernel PassLoad The amount of data transferred in a single time, Core Cache The HBSC size allocated to each core, Img Width is the input feature map width, Kernel Load The total core data load corresponding to each DSP Core.

[0095] Based on the current Kernel Pass , calculate the local memory address offset value, defined as Local Shift The calculation process is relatively simple, multiplying the amount of data by the data size, which will not be described here. Based on the pointer starting point of the current channel, the total core data load corresponding to each core, the current local memory address offset value and the preset overall starting pointer of the input feature map, the input feature map is converted from Channel Offset +KernelOffset +Local Shift +Data Ptr Start, length is kernel PassLoad The continuous data is moved from DRAM (Dynamic Random Access Memory) to HBSC via DMA BUS (Direct Memory Access Bus) and then from Cache Ptr Starting from a continuous space with the same length as the data source, select each row of data moved into the cache segment allocated on the current HBSC and define its local row number on the HBSC (that is, starting from 0) as Local Row Based on the currently selected data row, calculate the corresponding row number in the input feature map, defined as Img Row , the calculation process is as follows:

[0096]

[0097] Among them, Img Row is the corresponding row number in the input feature map, Img Height is the input feature map height, N Core The number of cores on the DSP device side. ID Kernel is the kernel number. Pass For channel, Core Cache The HBSC size allocated to each core, Img Width is the input feature map width, Local Row is the local row offset.

[0098] Based on the currently selected data row and the corresponding row number in the feature map, the number of repetitions of the corresponding repeated data sub-block on the columnized feature map is calculated, which is defined as N Repeat , repeated data sub-blocks such as Figure 4 As shown, Figure 4 This is a schematic diagram of the relevant parameters of the im2col method of a convolution optimization method based on DSP architecture in one embodiment of the present application; the repeated data sub-block specifically refers to a fixed-size matrix block on the columnar feature map; the width Block Width Equal to the number of receptive fields in the horizontal direction of the current convolution kernel based on the current convolution parameters (including step size, padding and hole value, etc.). The receptive field of the embodiment of the present application refers to the total set of states or a subset thereof in the process from the starting state to the ending state when the convolution kernel slides in one (or more) specific directions on the input feature map based on the current convolution parameters; Height Block HeightEqual to the width of the convolution kernel currently participating in the convolution; the number of its distribution in the horizontal direction of the columnar feature map is equal to the number of receptive fields in the vertical direction of the current convolution kernel based on the current convolution parameters; the number of its distribution in the vertical direction of the columnar feature map is equal to the height of the convolution kernel currently participating in the convolution; where, Figure 4 Img Width Indicates the input feature map width, Img Height Indicates the input feature map height; Kernel Width Indicates the width of the convolution kernel; Kernel Height Indicates the height of the input feature map; Col Width Indicates the width of the feature map after columnization; Col Height Indicates the height of the feature map after columnization; blocl width Indicates the width of the sub-block after splitting. The definition and allocation method of the sub-block are described in detail below; blocl height Indicates the height of the sub-block after splitting. The definition and allocation method of the sub-block are described in detail below; Col Wwidth Indicates the number of sub-blocks in the horizontal direction after splitting; Col Wheight Represents the number of sub-blocks in the vertical direction after splitting. The calculation process of the number of repetitions of the corresponding repeated data sub-blocks on the columnar feature map is as follows:

[0099]

[0100] Among them, Kernel Width is the convolution kernel width, Img row is the image width, Img Height is the image height.

[0101] Select each repetition number and define it as i Repeat , calculate the current column feature map i RePeat The corresponding first address is defined as Col Ptr The first address here refers to the memory address with the smallest equivalent row and column coordinates on the column-based feature map stored in row-major order, or the memory address with the lowest value within the coverage range. The starting data address of the column-based feature map is specified as Col. Init , then Col Ptr The calculation process is as follows:

[0102]

[0103] Among them, Col Ptr i is the columnar feature map Repeat The corresponding first address, Col Init is the starting data address of the column feature map, Channel i is the i-th channel, Col HeightIndicates the height of the feature map after columnization, Col Width Indicates the width of the feature map after columnization, Img row is the image width, Colw Blocks Block for row direction, block width is the block width, i Repeat is the number of repetitions of the i-th block, block height For block height.

[0104] Determine whether the current repeated data sub-block should be included in the output based on the Stride parameter declared in the current convolution parameters, and calculate the result of the following formula:

[0105]

[0106] Among them, Colw Blocks For row-wise partitioning, Img row is the image width, i Repeat is the number of repetitions for the i-th block.

[0107] Img Width and compare it with 0. If the result of the operation is zero, it is included in the output; otherwise it should be ignored. Select each row in the current repeated data sub-block, defined as i Block ; Set the current HBSC address to Local Shift Backward offset i Block positions, length Img Width -block width The non-continuous data with a step size (specifically data interval in this application) of Stride is asynchronously transferred to the address Col via the DMA BUS. Ptr Backward offset i Block *Img Width locations, in consecutive memory locations of equal length.

[0108] Furthermore, the length of the visited PassLoad Data from Kernel Load The convolution kernel shape is reinterpreted to be the same as the columnized feature map without any change to the data. The preset general matrix multiplication strategy (GEMM) is performed on the columnized feature map and the interpreted convolution kernel shape to obtain the final convolution optimization result.

[0109] Therefore, the embodiment of the present application overcomes the defects of high memory footprint and memory as a performance bottleneck in the convolution calculation method using im2col in the related art by rethinking and exploring the DSP architecture. Through the efficient application of the DSP architecture, the number of memory reads and writes is reduced by more than 40%, and the defects of poor data continuity and poor general cache hit rate in traditional algorithms are overcome by integrating data. In terms of the advantages brought by the high concurrency of the DSP architecture, the embodiment of the present application overcomes the defects of high data coupling, difficult slicing or the need to use additional computing resources and storage resources in traditional algorithms by re-dividing and slicing the data, thereby improving the independence between input data and ensuring that the DSP device can fully utilize the high concurrency advantages brought by its architecture as much as possible. On the DMA BUS, the embodiment of the present application utilizes the advantage of allowing DMA to transmit discontinuous (i.e., with step) data in the DSP architecture to overcome the defect of reduced memory read and write performance caused by discontinuous step data in traditional algorithms, ensuring that the subsequent calculation steps implemented based on the embodiment of the present application can reduce memory read / write delays and alleviate memory bottleneck defects.

[0110] To facilitate those skilled in the art to further understand the convolution optimization method based on DSP architecture in the embodiment of the present application, the following is a detailed description of the convolution optimization method based on DSP architecture in combination with the embodiment of the present application. Figures 5 to 7 The illustrated embodiment is described in detail.

[0111] Specifically, if Figure 5 As shown, Figure 5 This is a schematic diagram illustrating the overall optimization of the im2col method for a DSP-based convolution optimization method according to one embodiment of the present application. This embodiment of the present application designs an im2col algorithm specifically optimized for general-purpose DSP devices, aiming to maximize the utilization of DSP device bandwidth resources. By optimizing the data flow during data conversion and matrix operations, it ensures efficient performance when processing input data of varying sizes and dimensions. This optimization not only speeds up convolution operations but also enhances the overall efficiency of DSP devices when performing complex neural network computations.

[0112] Furthermore, if Figure 6 and Figure 7 As shown, Figure 6 This is a schematic diagram of a DMA device architecture of a convolution optimization method based on a DSP architecture according to an embodiment of the present application. Figure 7 This is a schematic diagram of an im2col method based on a DSP device for a convolution optimization method based on a DSP architecture according to an embodiment of the present application; wherein, N Core Indicates the number of DSP Cores; L Cache represents the total size of HBSC; Indicates the bus bandwidth from the DSP device end to the DRAM for the DSP Core with serial number i corresponding to DMA; B DRAM Indicates the bus bandwidth from the CPU to the DRAM. The embodiment of the present application realizes high-speed convolution calculation based on the DSP architecture by utilizing the high-bandwidth programmable shared cache (HBSC) and multiple physical cores (DSP Core) of the DSP device. The DSP architecture of the embodiment of the present application includes Figure 6 The digital signal processor or equivalent integrated circuit of the architecture shown includes a high-bandwidth programmable shared cache (HBSC) or its equivalent, multiple programmable cores with direct memory access (DMA) sub-chips or equivalent control circuits, independent DMA bus widths equal to or exceeding the number of programmable cores, and a bidirectional high-speed interconnect bus from the DSPCore in the chip to the HBSC. Among them, the local cache of the programmable core, the externally integrated or mounted central processing unit (CPU), and the CPU-DRAM bus are only introduced to facilitate the introduction of the method of the present invention and do not exist as part of the DSP architecture shown in this application. And among the variables mentioned in the im2col method shown, Core Cache Indicates the size of HBSC allocated to each core, N Line Indicates the number of rows from the input feature map assigned to each core.

[0113] Therefore, the embodiments of the present application relate to a convolution calculation optimization method based on a specific DSP platform and architecture, including optimization of reducing DRAM footprint by utilizing high-speed shared cache or its equivalent integrated circuit in the DSP architecture, convolution acceleration by efficiently utilizing DMA BUS by utilizing high-concurrency physical core or its equivalent integrated circuit in the DSP architecture, acceleration of general convolution calculation by utilizing HBSC blocking and data transfer in the DSP architecture, and improvement of special heterogeneous platforms of the traditional im2col method.

[0114] According to the convolution optimization method based on DSP architecture proposed in the embodiment of the present application, the embodiment of the present application allocates the high-bandwidth programmable shared cache on the DSP side based on a preset allocation strategy to obtain the high-bandwidth programmable shared cache capacity allocated to each core, and takes the pointer starting point of the current channel as the benchmark, dynamically divides the preset input feature map according to the number of cores and the first address offset, obtains the total core data load and data processing starting point corresponding to each core, and then calculates the query rounds that need to be executed when traversing the total core data load corresponding to each core based on the high-bandwidth programmable shared cache capacity allocated to each core; selects each query round, and applies the preset declaration strategy and declaration result to obtain the shape of the convolution kernel, and obtains the shape of the interpreted convolution kernel based on the preset columnized feature map, and obtains the final convolution optimization result based on the preset general matrix multiplication strategy. Thus, the problems of high memory footprint, poor data continuity and low concurrency utilization in the related technology are solved, and the efficiency of convolution is enhanced.

[0115] Next, a convolution optimization device based on a DSP architecture proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.

[0116] Figure 8 It is a block diagram of a convolution optimization device based on DSP architecture according to an embodiment of the present application.

[0117] like Figure 8 As shown, the convolution optimization device 10 based on DSP architecture includes: an allocation module 100, a segmentation module 200, an offset module 300, a calculation module 400 and a determination module 500.

[0118] The allocation module 100 is configured to allocate the high-bandwidth programmable shared cache on the DSP device side to each core on the DSP device side based on a preset allocation strategy, thereby obtaining the high-bandwidth programmable shared cache capacity allocated to each core.

[0119] The segmentation module 200 is used to determine the current channel in the preset input feature map, and dynamically segment the preset input feature map according to the number of cores on the DSP device side based on the pointer starting point of the current channel, and retain the entire row to obtain the total core data load corresponding to each core;

[0120] An offset module 300 is used to offset the first address of the preset input feature map according to the number of cores on the DSP device, and obtain the data processing starting point corresponding to each core based on the offset address;

[0121] A calculation module 400 is configured to calculate, based on the high-bandwidth programmable shared cache capacity allocated to each core, a number of query rounds required to be executed when traversing the total core data load corresponding to each core, thereby obtaining a plurality of query rounds;

[0122] Determine module 500, which is used to select each query round and apply a preset declaration strategy, and obtain the shape of the convolution kernel based on the declaration result, and interpret the shape of the convolution kernel based on the preset columnized feature map to obtain the shape of the interpreted convolution kernel, and process the shape of the interpreted convolution kernel and the preset columnized feature map based on the preset general matrix multiplication strategy to obtain the final convolution optimization result.

[0123] Optionally, the determination module 500 is specifically configured to: determine a current query round from multiple query rounds, and calculate the amount of data transferred in a single transaction based on the current query round; calculate a current local memory address offset value based on the current query round; determine a starting position of a preset input feature map based on a pointer starting point of the current channel, a total core data load corresponding to each core, a current local memory address offset value, and an overall starting pointer of a preset input feature map; and based on the starting position, transfer the data from the DRAM to the high-bandwidth programmable shared cache via the DMABUS according to the amount of data transferred in a single transaction; select each row of data transferred into a cache segment allocated on the high-bandwidth programmable shared cache, and calculate a corresponding row number in the preset input feature map based on the currently selected data row; and calculate a number of repetitions of a corresponding repeated data sub-block on the columnized feature map based on the currently selected data row and the corresponding row number in the preset input feature map;

[0124] Select each number of repetitions, calculate the first address corresponding to the columnized feature map, and determine whether the current repeated data sub-block is included in the output based on the Stride parameter declared in the current convolution parameters; select each row in the current repeated data sub-block, offset the address on the high-bandwidth programmable shared cache to the current local memory address offset value backward by a preset position and a preset step size, and based on the preset transfer strategy, asynchronously transfer the Stride parameter through the DMA BUS bus.

[0125] Optionally, the preset allocation strategy is:

[0126]

[0127] Among them, Core Cache High-bandwidth programmable shared cache capacity allocated to each core, L Cache is the total capacity of the high-bandwidth programmable shared cache, N Core The number of cores on the DSP device.

[0128] Optionally, the total core data load corresponding to each core is:

[0129]

[0130] Among them, Kernel Load The total core data load corresponding to each core, ImgWidth is the input feature map width, Img Height is the input feature map height, N Core is the number of cores on the DSP device side, C i is the i-th core.

[0131] Optionally, the data processing starting point for each core is:

[0132]

[0133] Among them, Kernel Offset is the data processing starting point for each core, C i is the i-th core, Img Width is the input feature map width, Img Height is the input feature map height, N Core The number of cores on the DSP device.

[0134] It should be noted that the above explanation of the embodiment of the convolution optimization method based on DSP architecture is also applicable to the convolution optimization device based on DSP architecture in this embodiment, and will not be repeated here.

[0135] According to the convolution optimization device based on DSP architecture proposed in the embodiment of the present application, the embodiment of the present application allocates the high-bandwidth programmable shared cache on the DSP side based on a preset allocation strategy to obtain the high-bandwidth programmable shared cache capacity allocated to each core, takes the pointer starting point of the current channel as the benchmark, dynamically divides the preset input feature map according to the number of cores and the first address offset, obtains the total core data load and data processing starting point corresponding to each core, and then calculates the query rounds that need to be executed when traversing the total core data load corresponding to each core based on the high-bandwidth programmable shared cache capacity allocated to each core; selects each query round, and applies the preset declaration strategy and declaration result to obtain the shape of the convolution kernel, and obtains the shape of the interpreted convolution kernel based on the preset columnized feature map, and obtains the final convolution optimization result based on the preset general matrix multiplication strategy. Thus, the problems of high memory footprint, poor data continuity and low concurrency utilization in the related technology are solved, and the efficiency of convolution is enhanced.

[0136] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0137] A memory 901 , a processor 902 , and a computer program stored in the memory 901 and executable on the processor 902 .

[0138] When the processor 902 executes the program, the convolution optimization method based on the DSP architecture provided in the above embodiment is implemented.

[0139] Furthermore, the electronic device further includes:

[0140] The communication interface 903 is used for communication between the memory 901 and the processor 902 .

[0141] The memory 901 is used to store computer programs that can be run on the processor 902 .

[0142] The memory 901 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0143] If the memory 901, processor 902, and communication interface 903 are implemented independently, the communication interface 903, memory 901, and processor 902 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0144] Optionally, in a specific implementation, if the memory 901, the processor 902 and the communication interface 903 are integrated on a chip, the memory 901, the processor 902 and the communication interface 903 can communicate with each other through an internal interface.

[0145] The processor 902 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0146] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned convolution optimization method based on DSP architecture.

[0147] An embodiment of the present application also provides a computer program product, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned convolution optimization method based on DSP architecture.

[0148] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0149] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0150] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0151] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0152] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

Claims

1. A convolution optimization method based on DSP architecture, characterized in that: The following steps are involved: Allocating the high-bandwidth programmable shared cache on the DSP device side to each core of the DSP device side based on a preset allocation strategy to obtain a high-bandwidth programmable shared cache capacity allocated to each core; Determine a current channel in a preset input feature map, and dynamically divide the preset input feature map according to the number of cores on the DSP device end based on the starting point of the pointer of the current channel, and retain the whole row to obtain the total core data load corresponding to each core; The preset input feature map is offset in the first address according to the number of cores of the DSP device, and based on the offset address, a data processing starting point corresponding to each core is obtained; Based on the high-bandwidth programmable shared cache capacity allocated to each core, calculating the number of query rounds required to be executed when traversing the total core data load corresponding to each core, to obtain multiple query rounds; Select each query round and apply the preset declaration strategy, and based on the declaration result, obtain the shape of the convolution kernel, and based on the preset columnized feature map, interpret the shape of the convolution kernel to obtain the shape of the interpreted convolution kernel, and based on the preset general matrix multiplication strategy, process the shape of the interpreted convolution kernel and the preset columnized feature map to obtain the final convolution optimization result.

2. The method according to claim 1, characterized in that The selection process for each query round applies the preset declaration strategy, including: Determining a current query round from the multiple query rounds, and calculating a single data transfer amount based on the current query round; Calculate the current local memory address offset value based on the current query round; Determine the starting position of the preset input feature map according to the pointer starting point of the current channel, the total core data load corresponding to each core, the current local memory address offset value, and the overall starting pointer of the preset input feature map, and based on the starting position, move the data from the DRAM to the high-bandwidth programmable shared cache through the DMA BUS according to the amount of data transferred in a single time; Selecting each row of data moved into the cache segment allocated on the high-bandwidth programmable shared cache, and calculating a corresponding row number in the preset input feature map based on the currently selected data row; Calculating the number of repetitions of the corresponding repeated data sub-block on the columnized feature map based on the currently selected data row and the corresponding row number in the preset input feature map; Select each repetition number, calculate the first address corresponding to the columnized feature map, and determine whether the current repeated data sub-block is included in the output based on the Stride parameter declared in the current convolution parameters; Each row in the current repeated data sub-block is selected, the address on the high-bandwidth programmable shared cache is shifted backward by a preset position and a preset step size as the current local memory address offset value, and the Stride parameter is asynchronously transferred through the DMA BUS based on a preset transfer strategy.

3. The method according to claim 1, characterized in that The preset allocation strategy is: Among them, Core Cache The high-bandwidth programmable shared cache capacity allocated to each core, L Cache is the total capacity of the high-bandwidth programmable shared cache, N Core is the number of cores on the DSP device.

4. The method according to claim 1, wherein The total core data load corresponding to each core is: Among them, Kernel Load Img is the total core data load corresponding to each core, width is the input feature map width, Img Height is the input feature map height, N Core is the number of cores on the DSP device side, C i is the i-th core.

5. The method according to claim 1, wherein The data processing starting point corresponding to each core is: Among them, Kernel Offset is the data processing starting point corresponding to each core, C i is the i-th core, Img Width is the input feature map width, Img Height is the input feature map height, N Core The number of cores on the DSP device.

6. A convolution optimization device based on DSP architecture, characterized in that: include: an allocation module, configured to allocate the high-bandwidth programmable shared cache of the DSP device end to each core of the DSP device end based on a preset allocation strategy, thereby obtaining the high-bandwidth programmable shared cache capacity allocated to each core; A segmentation module is used to determine a current channel in a preset input feature map, and dynamically segment the preset input feature map according to the number of cores on the DSP device side based on the pointer starting point of the current channel, and retain the whole row to obtain the total core data load corresponding to each core; An offset module is used to offset the first address of the preset input feature map according to the number of cores on the DSP device end, and obtain the data processing starting point corresponding to each core based on the offset address; a calculation module, configured to calculate, based on the high-bandwidth programmable shared cache capacity allocated to each core, a number of query rounds required to be executed when traversing a total core data load corresponding to each core, to obtain a plurality of query rounds; A determination module is used to select each query round and apply a preset declaration strategy, and based on the declaration result, obtain the shape of the convolution kernel, and based on the preset columnized feature map, interpret the shape of the convolution kernel to obtain the shape of the interpreted convolution kernel, and based on the preset general matrix multiplication strategy, process the shape of the interpreted convolution kernel and the preset columnized feature map to obtain the final convolution optimization result.

7. The device according to claim 6, characterized in that The determining module is specifically configured to: Determining a current query round from the multiple query rounds, and calculating a single data transfer amount based on the current query round; Calculate the current local memory address offset value based on the current query round; Determine the starting position of the preset input feature map according to the pointer starting point of the current channel, the total core data load corresponding to each core, the current local memory address offset value, and the overall starting pointer of the preset input feature map, and based on the starting position, move the data from the DRAM to the high-bandwidth programmable shared cache through the DMA BUS according to the amount of data transferred in a single time; Selecting each row of data moved into the cache segment allocated on the high-bandwidth programmable shared cache, and calculating a corresponding row number in the preset input feature map based on the currently selected data row; Calculating the number of repetitions of the corresponding repeated data sub-block on the columnized feature map based on the currently selected data row and the corresponding row number in the preset input feature map; Select each repetition number, calculate the first address corresponding to the columnized feature map, and determine whether the current repeated data sub-block is included in the output based on the Stride parameter declared in the current convolution parameters; Each row in the current repeated data sub-block is selected, the address on the high-bandwidth programmable shared cache is shifted backward by a preset position and a preset step size as the current local memory address offset value, and the Stride parameter is asynchronously transferred through the DMA BUS based on a preset transfer strategy.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the convolution optimization method based on the DSP architecture according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the convolution optimization method based on DSP architecture as described in any one of claims 1 to 5.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the convolution optimization method based on the DSP architecture according to any one of claims 1 to 5 is implemented.

Images