Chip structure and its size transformation accelerator

Abstract

The present invention provides a chip structure and a size transformation accelerator thereof. The size transformation accelerator comprises: a control module, comprising an interface register list and an output feature map coordinate control module, the interface register list comprising a first register and a second register, the output feature map coordinate control module being used to traverse the coordinates of the output feature map, and generating a size transformation calculation parameter based on the interface register list according to the coordinates of the output feature map; a feature map input control module being used to connect to an on-chip bus; a feature map multiplexing control module being used to cache the image data of the input feature map, and controlling the writing and reading of the cached image data of the input feature map; a size transformation multiplication-addition array, which transforms the image data of the input feature map into the image data of the output feature map according to the size transformation calculation parameters in a pipeline manner; and a feature map output control module being used to connect to the on-chip bus. The present invention realizes size transformation acceleration.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a chip structure and a size transformation accelerator thereof. Background Art

[0002] In the field of image processing, especially in deep learning network models of images, it is usually necessary to resize and scale the input image. Therefore, resize operators are widely used in various deep learning algorithms.

[0003] The commonly used algorithm for size transformation is the bilinear interpolation algorithm. There are currently some accelerated calculation methods for the bilinear interpolation algorithm. For example, the announcement number CN104869284A, named a high-efficiency FPGA (Field-Programmable Gate Array) of a bilinear interpolation amplification algorithm, implementation method and device, mainly through two consecutive output pixel points, respectively provide a set of calculation weights, and express the relationship between the two sets of calculation weights by a mathematical formula. The patent aims to reduce the use of multipliers through the FPGA implementation of the relationship formula, so that FPGAs with small-scale resources can also support bilinear interpolation. Another example is the announcement number CN111429346A, named a real-time video image method based on FPGA, which describes the use of a 2x2 register array. Each calculation first updates two numbers in the row direction, and then updates two numbers in the column direction, which is completely consistent with the original algorithm.

[0004] It can be seen that the current acceleration improvements for the bilinear interpolation algorithm mainly lie in the improvement of the algorithm and the hardware implementation of the size transformation. None of them consider how to achieve the calculation acceleration of the size transformation from the direction of the hardware reading, writing and caching of the bilinear interpolation algorithm.

[0005] It can be seen that how to improve the data reading, writing and caching of the chip to achieve computational acceleration of size transformation is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0006] In order to overcome the defects of the above-mentioned prior art, the present invention provides a chip structure and a size transformation accelerator thereof to improve the data reading and writing and cache of the chip to achieve the calculation acceleration of size transformation.

[0007] According to one aspect of the present invention, there is provided a size transformation accelerator, comprising:

[0008] A control module, comprising an interface register list and an output feature map coordinate control module, wherein the interface register list comprises a first register for configuring a scaling ratio of a feature map in a first direction and a second register for configuring a scaling ratio of a feature map in a second direction, wherein the first direction is perpendicular to the second direction, and the output feature map coordinate control module is used to traverse the coordinates of the output feature map, and generate a calculation parameter for size transformation based on the interface register list according to the coordinates of the output feature map;

[0009] A feature map input control module, used to connect to the on-chip bus to obtain image data of the input feature map;

[0010] A feature map multiplexing control module, used to cache image data of the input feature map and control the writing and reading of the cached image data of the input feature map;

[0011] a size conversion multiplication-addition array for performing multiplication-addition calculations on the image data of the input feature map read by the feature map multiplexing control module and the calculation parameters of the size conversion in a pipeline manner, so as to convert the image data of the input feature map into the image data of the output feature map according to the calculation parameters of the size conversion; and

[0012] The feature map output control module is used to connect to the on-chip bus to send the image data of the output feature map to the on-chip bus.

[0013] In some embodiments of the present application, the output feature map coordinate control module includes a first output feature map coordinate control module and a second output feature map coordinate control module to traverse the coordinates of the first direction and the second direction of the output feature map respectively.

[0014] In some embodiments of the present application, the feature map multiplexing control module includes:

[0015] The input feature map cache module includes multiple pairs of cache units, each pair of cache units includes a first cache unit and a second cache unit, which are used to cache two rows of input feature map data for calculating one row of data of the current output feature map.

[0016] In some embodiments of the present application, the feature map input control module includes a direct storage access module, and the feature map input control module is configured to:

[0017] Determine along the first traversal direction whether the rows of the input feature map to be cached by the first cache unit / the second cache unit of the plurality of pairs of cache units of the input feature map cache module are continuous and the same;

[0018] If so, the direct storage access module is activated to read the row of data of the input feature map once and cache it in multiple consecutive and identical first cache units / second cache units.

[0019] In some embodiments of the present application, the feature map input control module includes a direct storage access module, and the feature map input control module is configured to:

[0020] Determine whether there are multiple second cache units having the same rows of input features to be cached by the first cache unit of the multiple pairs of cache units of the input feature map cache module;

[0021] If so, the direct storage access module is started to read the row of data of the input feature map once and cache it in the corresponding first cache unit and second cache unit.

[0022] In some embodiments of the present application, the input feature map cache module includes a first input feature map cache module and a second input feature map cache module, the first input feature map cache module and the second input feature map cache module both include multiple pairs of cache units, and the first input feature map cache module and / or the second input feature map cache module are configured as follows:

[0023] Determine whether the row of the input feature map to be cached in the multiple pairs of cache units of the input feature map cache module is repeated in the row of the input feature map cached in the multiple pairs of cache units of another input feature map cache module;

[0024] If so, directly read the rows of the input feature map cached in multiple pairs of cache units of another input feature map cache module.

[0025] In some embodiments of the present application, the first input feature map cache module and the second input feature map cache module respectively include a first data reading interface and a second data reading interface, the first data reading interface respectively transmits data with the feature map input control module and the feature map output control module, and the second data reading interface is used to perform data transmission between the first input feature map cache module and the second input feature map cache module.

[0026] In some embodiments of the present application, the first cache unit and the second cache unit include an odd-numbered column cache unit and an even-numbered column cache unit, respectively.

[0027] In the first quantization mode, the bit width of each column of the input feature map is N bits, the interface bit width of the odd column cache unit and the even column cache unit is 2N bits, the odd column cache unit caches the spliced ​​data of two adjacent odd columns of the input image, and the even column cache unit caches the spliced ​​data of two adjacent even columns of the input image;

[0028] In the second quantization mode, the bit width of each column of the input feature map is 2N bits, the interface bit width of the odd column cache unit and the even column cache unit is 2N bits, the odd column cache unit caches data of an odd column of the input image, and the even column cache unit caches data of an even column of the input image,

[0029] Wherein, N is an integer greater than 0.

[0030] In some embodiments of the present application, the control module is also used to slice the input feature map and the output feature map, and the slicing direction is the row direction of the feature map.

[0031] According to another aspect of the present application, a chip structure is further provided, comprising:

[0032] A size-changing accelerator as described above;

[0033] On-chip bus; and

[0034] An on-chip processor is used to configure an interface register list of the size change accelerator.

[0035] Compared with the prior art, the advantages of the present invention are:

[0036] 1) Through the control module of the size transformation accelerator, the coordinates of the output feature map are traversed, and the calculation parameters of the size transformation are generated based on the interface register list according to the coordinates of the output feature map, thereby eliminating the need to provide the calculation parameters of the size transformation externally, reducing the data reading and writing steps, and the interface register provides scaling parameter settings in two directions to adapt to different scaling sizes;

[0037] 2) A feature map input control module and a feature map reuse control module are used to cache image data of the input feature map and control the writing and reading of the cached image data of the input feature map, thereby facilitating data reuse of multiple dimensions of the feature map and further reducing the time consumption required for data reading;

[0038] Therefore, the present application realizes the computational acceleration of size transformation by improving the data reading and writing and caching of the chip. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other features and advantages of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings.

[0040] Figure 1 A module diagram showing a chip structure according to an embodiment of the present invention;

[0041] Figure 2 A module diagram of a size transformation accelerator according to an embodiment of the present invention is shown;

[0042] Figure 3 A schematic diagram of an input / output feature map according to an embodiment of the present invention is shown;

[0043] Figure 4 A schematic diagram of an input feature map cache module according to an embodiment of the present invention is shown;

[0044] Figure 5 A schematic diagram showing a first acceleration dimension of an input feature map cache module according to an embodiment of the present invention is shown;

[0045] Figure 6 A schematic diagram showing a second acceleration dimension of an input feature map cache module according to an embodiment of the present invention is shown;

[0046] Figure 7 A schematic diagram showing a third acceleration dimension of an input feature map cache module according to an embodiment of the present invention is shown;

[0047] Figure 8 A schematic diagram of a fourth acceleration dimension of an input feature map cache module according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0048] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0049] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0050] This application is mainly used for the reasoning calculation of the resize operator in the neural network, focusing on the efficient reuse of the original data to be scaled that is cached in the chip, using a multi-dimensional reuse idea, combined with a ping-pong cache, and a pipeline method to provide high-performance computing capabilities. At the same time, the corresponding on-chip parallel computing resources can be matched according to different DRAM (Dynamic Random Access Memory) bandwidths, so that the accelerator DRAM interface bandwidth and accelerator computing power are optional and adaptable in the chip design stage. Compared with the direct use of the x86 or ARM architecture CPU, this application implements the accelerator on the FPGA after cache acceleration. Based on measured data, compared with the x86 architecture CPU, under common image sizes, the computing performance of the operator can be improved by 100 times.

[0051] See first Figure 1 , Figure 1 A module diagram of a chip structure according to an embodiment of the present invention is shown. The chip structure includes a size transformation accelerator 141, an on-chip bus system 120, and an on-chip processor (general processor) 110. Specifically, the size transformation accelerator 141 can be set in the deep learning algorithm accelerator 140 to serve as one of the multiple acceleration IP cores (intellectual property cores) of the deep learning algorithm accelerator 140. The on-chip bus system 120 can be, for example, an AXI (Advanced Extensible Interface) bus system, but the present application is not limited to this. The on-chip processor 110 can configure the characteristic parameters of the size transformation accelerator 141 according to the requirements of the deep learning network model. Figure 1 Also shown is an off-chip DRAM access interface 130, which is used to read and write off-chip storage.

[0052] The structure of the size-changing accelerator can be found in Figure 2 The resizing accelerator includes a control module 210 , a feature map input control module 220 , a feature map multiplexing control module 230 , a resizing multiplication-addition array 240 , and a feature map output control module 250 .

[0053] The control module 220 includes an interface register list and an output feature map coordinate control module. The interface register list includes a first register for configuring the scaling ratio of the first direction of the feature map (e.g., the row direction of the feature map) and a second register for configuring the scaling ratio of the second direction of the feature map (e.g., the column direction of the feature map), wherein the first direction is perpendicular to the second direction. Specifically, the first register and the second register are both 32-bit interface registers for storing size scaling parameters. The present application may also set registers other than the first and second registers to store other parameters, thereby realizing the increase and decrease of configuration functions, and the present application is not limited thereto.

[0054] The output feature map coordinate control module is used to traverse the coordinates of the output feature map, and generate the calculation parameters of the size transformation based on the interface register list according to the coordinates of the output feature map. Specifically, the output feature map coordinate control module may include a first output feature map coordinate control module and a second output feature map coordinate control module to traverse the coordinates of the first direction and the second direction of the output feature map respectively.

[0055] The feature map input control module 220 is used to connect to the on-chip bus 120 to obtain image data of the input feature map.

[0056] The feature map multiplexing control module 230 is used to cache the image data of the input feature map and control the writing and reading of the cached image data of the input feature map.

[0057] The size transformation multiplication-addition array 240 performs multiplication-addition calculations on the image data of the input feature map read by the feature map multiplexing control module and the calculation parameters of the size transformation in a pipeline manner, so as to transform the image data of the input feature map into the image data of the output feature map according to the calculation parameters of the size transformation.

[0058] The feature map output control module 250 is used to connect to the on-chip bus 120 to send the image data of the output feature map to the on-chip bus.

[0059] In the chip structure and size transformation accelerator thereof provided in the present application, the coordinates of the output feature map are traversed through the control module of the size transformation accelerator, and the calculation parameters of the size transformation are generated based on the interface register list according to the coordinates of the output feature map. Therefore, there is no need to provide the calculation parameters of the size transformation externally, reducing the data reading and writing steps. At the same time, the interface register provides scaling parameter settings in two directions to adapt to different scaling sizes; through the feature map input control module and the feature map multiplexing control module, the image data of the input feature map is cached, and the writing and reading of the cached input feature map image data are controlled, so as to facilitate the execution of data multiplexing of multiple dimensions of the feature map, and further reduce the time consumption required for data reading; therefore, the present application realizes the calculation acceleration of size transformation by improving the data reading and writing and caching of the chip.

[0060] Specifically, the control module 210 can also be used to slice and cut the input feature map and the output feature map, and the slicing direction is the row direction of the feature map. Furthermore, the control module 210 can slice the output feature map along the row direction of the feature map, and not slice the column direction of the feature map. Then in each slice, the calculation of n rows of complete output feature map data requires the preparation of 2n rows of complete input feature map data (n is an integer greater than 0). In the process of alternating calculations of each slice, the present application can automatically parse the scaling characteristics and automatically enable the reuse of input feature map data in multiple dimensions.

[0061] Specifically, the control module 210 can control each slice of the feature map to be calculated in this way, and thereby control the direct memory access module (DMA) of the feature map input control module 220 and the feature map multiplexing control module 230, as well as obtain or input the corresponding starting address of the feature map from the off-chip DRAM.

[0062] Specifically, the present application is applied to scenarios where a bilinear interpolation algorithm is used to achieve size transformation. In the bilinear interpolation algorithm, for each pixel point of the output feature map, a set of parameters is required for its bilinear interpolation calculation, and the parameters required for each pixel point are different. If the output feature map size is large, the amount of parameter data is large. The present application controls the row direction and column direction of the output feature map respectively through the first output feature map coordinate control module and the second output feature map coordinate control module, so that the corresponding calculation parameters can be generated in real time based on the coordinate control module while traversing the output coordinates. The calculation parameters can enter the size transformation multiplication and addition array 240 in a pipeline manner, and match the input feature map entering the size transformation multiplication and addition array 240 in real time. Therefore, the size transformation accelerator no longer needs to obtain relevant calculation parameters from the on-chip bus interface.

[0063] In other words, the coefficients directly multiplied with the image data to be scaled in the size transformation multiplication and addition array 240 are automatically generated by the first output feature map coordinate control module and the second output feature map coordinate control module in the chip, and do not need to be provided by the accelerator outside. Compared with the solution in which the host computer writes the coefficients calculated in advance externally into the accelerator, on the one hand, there is no need to write the coefficients in advance, which avoids the problem that the accelerator can only work under the configuration of a fixed size scaling ratio. Once the scaling ratio is changed, the coefficients need to be rewritten, resulting in the accelerator being unable to adapt to the rapid switching of scene tasks in real time; on the other hand, there is no need to use a large area of ​​static random access memory (SRAM) to store related calculation coefficients, avoiding excessive consumption of on-chip area.

[0064] In addition, through the above method, the present application only needs to provide the original feature map data to be scaled, and the scaling ratios of the row direction and the column direction to perform accelerated calculation. The scaling ratios of the row direction and the column direction can be configured in the interface register list.

[0065] See below Figure 3 , Figure 3 A schematic diagram of an input / output feature map according to an embodiment of the present invention is shown. Along the horizontal direction of the feature map, it can be defined as the feature map column direction, wherein, starting from the 0th column and the 1st column, they are defined as even columns and odd columns, respectively. Even columns and odd columns appear alternately. Along the vertical direction of the feature map, it is defined as the feature map row direction. The following embodiments of the present application are described and illustrated with the definition of the row direction and column direction.

[0066] Specifically, the open source code for calculating the row and column coordinates of the feature map of the bilinear interpolation algorithm is:

[0067]

[0068] The above source code shows the calculation process of the row and column coordinates of the input feature map and the output feature map, and shows the relationship between the row coordinates of the output feature map and the row coordinates of the input feature map.

[0069] Among them, src_h represents the maximum row coordinate of the input feature map. dst_h represents the maximum row coordinate of the output feature map. scale_y represents the ratio of the input feature map to the output feature map in the row direction. dst_index represents the row coordinate of the output feature map. Assuming that the image is enlarged, scale_y is less than 1. The final src_y_0 and src_y_1 represent the two rows on the input feature map that need to be provided for the output feature map of a certain dst_index row to complete the relevant calculation. Generally speaking, src_y_1 is 1 greater than src_y_0, but at the edge of the image, they may be equal.

[0070] According to the above source code analysis, the calculation of each row in the output feature map requires two rows of data on the input feature map, and the required rows can be obtained according to the above source code calculation. Therefore, the feature map reuse control module 230 of the present application may include input feature map cache modules 233 and 234. The input feature map cache modules 233 and 234 include multiple pairs of cache units, each pair of cache units includes a first cache unit 235A and a second cache unit 235B, to respectively cache two rows of data of the input feature map used to calculate one row of data of the current output feature map.

[0071] Further, see Figure 4 , Figure 4 A schematic diagram of an input feature map cache module according to an embodiment of the present invention is shown. In this embodiment, the input feature map cache module includes a first input feature map cache module 233 and a second input feature map cache module 234.

[0072] Among them, the first input feature map cache module 233 and the second input feature map cache module 234 can have the same structure. The first input feature map cache module 233 and the second input feature map cache module 234 can both be SRAM, so that the SRAM resources consumed by the first input feature map cache module 233 and the second input feature map cache module 234 are the same, which can also be called a ping-pong cache architecture.

[0073] like Figure 4 , for calculating one line of output feature map, the required two lines of input feature maps src_y_0 and src_y_1 will be stored in the first cache unit 235A and the second cache unit 235B respectively. The first cache unit 235A and the second cache unit 235B can be used as SRAM basic units, which can be built based on the features of the SRAM IP library provided by the chip manufacturer, or based on the minimum size monolithic SRAM provided by the FPGA manufacturer. Among them, src_y_0 is stored in the first cache unit 235A, and src_y_1 is stored in the second cache unit 235B.

[0074] The input feature map is input to the first input feature map cache module 233 or the second input feature map cache module 234 for cache. The first input feature map cache module 233 / the second input feature map cache module 234 stores at most input feature rows for calculating n rows of output feature maps. The first input feature map cache module 233 and the second input feature map cache module 234 cache 2nx2 rows of input feature data in total. Here n is a variable variable. In the architecture design, the value of n can be comprehensively considered according to the design performance requirements and the area and cost of the chip. n represents the parallel computing acceleration capability of the output feature map.

[0075] The following example is continued with n=6.

[0076] Assume that an input feature map is non-integer-scaled, assuming that it is scaled up by 2.6 times along h, i.e., the row direction.

[0077] For the first 12 rows of output feature maps, according to the calculation of the aforementioned source code, it can be concluded that src_y_0 and src_y_1 of these 12 rows are:

[0078] dst_index: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11

[0079] src_y_0: 0, 0, 0, 0, 1, 1, 2, 2, 2, 3, 3, 3

[0080] src_y_1: 1, 1, 1, 1, 2, 2, 3, 3, 3, 4, 4, 4

[0081] Within the maximum range of input feature map size and output feature map size, this application can flexibly support the configuration of any integer size, and the ratio of different size changes also brings about irregular changes in the src_y_0, src_y_1 sequence. The design starting point and the most difficult design challenge of this application is to design a detection logic that can automatically generate dst_index, src_y_0, src_y_1 sequence based only on the feature map size and scaling ratio configured by the accelerator interface register, and secondly, it can automatically parse the coordinate relationship between any src_y_0, src_y_1 sequence, and at the same time realize the multiplexing of input feature maps within the chip.

[0082] The following will combine Figures 5 to 8 Describe various on-chip multiplexing methods of this application.

[0083] Figure 5 A schematic diagram of a first acceleration dimension of an input feature map cache module according to an embodiment of the present invention is shown. In this embodiment, the feature map input control module includes a direct storage access module. The feature map input control module can be configured to: determine whether the rows of the input feature map to be cached by the first cache unit / second cache unit of multiple pairs of cache units of the input feature map cache module 233 are continuously the same along the first traversal direction; if so, when the rows of the input feature map to be cached are different (that is, when the continuous sameness ends), start the direct storage access module to read the row data of the input feature map once, and cache it in multiple consecutively identical first cache units / second cache units.

[0084] like Figure 5In the aforementioned embodiment (src_y_0: 0, 0, 0, 0, 1, 1, 2, 2, 2, 3, 3, 3; src_y_1: 1, 1, 1, 1, 2, 2, 3, 3, 3, 4, 4, 4), it is determined that the first cache unit of the first pair of cache units of the input feature map cache module 233 is to store the 0th row of data of the input feature map, and the first pair of cache units are arranged along the first traversal direction (such as Figure 5 In the downward direction), it is determined that the rows of the input feature graph to be cached by the first cache units of the plurality of pairs of cache units of the input feature graph cache module 233 are the same until the first cache units of the fourth pair of cache units are the same. The same first cache units can be marked, so that the feature graph input control module starts the direct storage access module once, so that in the start-up, the required row data of the input feature graph is read and cached in the plurality of consecutive and identical first cache units (the marked plurality of first cache units).

[0085] In this example, the rows of input feature maps required by the first four first cache units in the input feature map cache module 233 are the same. According to the above DMA data multiplexing method, the DMA of one row of data can achieve on-chip cache of four rows of calculation data, and the DMA efficiency is improved by 400%. However, common accelerators will repeatedly read the same row 0 from DDR four times, which greatly affects the efficiency of the bus and DMA.

[0086] The above multiplexing method is not only implemented between the first cache units in the input feature map cache module 233 / 234, but also implemented between the second cache units. The multiplexing method of this embodiment is implemented between the first cache units or between the second cache units in the input feature map cache module 233 / 234. Figure 5 The dotted arrow in can also adopt this multiplexing method to start the second DMA.

[0087] See below Figure 6 , Figure 6 A schematic diagram of a second acceleration dimension of an input feature map cache module according to an embodiment of the present invention is shown.

[0088] In this embodiment, the feature map input control module includes a direct storage access module, and the feature map input control module is configured to determine whether there are multiple second cache units whose rows of input features are the same as the row of input feature maps to be cached by the first cache unit of multiple pairs of cache units of the input feature map cache module; if so, the direct storage access module is started to read the row of data of the input feature map once, and cache it in the corresponding first cache unit and second cache unit.

[0089] When combined with the above-mentioned first multiplexing method, if after the above-mentioned first multiplexing method, the detection logic finds that the current valid DMA input feature row is located in a first cache unit module, and the data row needs to exist in several second cache units at the same time, then the search for the second cache unit will continue until the last second cache unit in the minimum coordinate row is detected along the second traversal direction, and then the DMA of the input feature data row will be started.

[0090] like Figure 6 , input feature line 1 exists in the first cache unit corresponding to the coordinates of output feature lines 4, 5, and also exists in the second cache unit corresponding to the coordinates of output feature lines 3, 2, 1, 0. When the detection of the last second cache unit in the direction of the arrow is completed, the DMA of input feature line 1 is started, and the data of this line is simultaneously written into the SRAM of the first cache unit and the second cache unit. One line of data completes the required cache of six lines, and the efficiency is improved by 600%.

[0091] The above-mentioned multiplexing method is performed between the first cache unit and the second cache unit within each of the input feature map cache modules 233 / 234.

[0092] See below Figure 7 , Figure 7 A schematic diagram of the third acceleration dimension of the input feature map cache module according to an embodiment of the present invention is shown. In this embodiment, the input feature map cache module includes a first input feature map cache module 233 and a second input feature map cache module 234, the first input feature map cache module 233 and the second input feature map cache module 234 both include multiple pairs of cache units, and the first input feature map cache module 233 and / or the second input feature map cache module 234 are configured to: determine whether the rows of the input feature map to be cached in the multiple pairs of cache units of the input feature map cache module are repeated in the rows of the input feature map cached in the multiple pairs of cache units of another input feature map cache module; if so, directly read the rows of the input feature map cached in the multiple pairs of cache units of another input feature map cache module.

[0093] This multiplexing method takes into account the possibility of data duplication and the usual sequence characteristics between the first input feature map cache module 233 and the second input feature map cache module 234. Therefore, when detecting the first cache unit of any input feature map cache module, if the input feature in the first cache unit also exists in the second cache unit of another input feature map cache module, then the row of input features is directly taken out from the second read data interface in the other input feature map cache module, and no DDR DMA operation is performed, thereby reducing the resource usage of the interface bandwidth.

[0094] Furthermore, the feature here is that the SRAM arrays in the two cache modules, the first input feature map cache module 233 and the second input feature map cache module 234, can be implemented in the form of true dual-port SRAM (True Dual Port RAM), so one SRAM can simultaneously support two read interfaces to work independently, thereby ensuring that the direct data movement between the first input feature map cache module 233 and the second input feature map cache module 234 does not affect the use of the first read data interface of the first input feature map cache module 233 and the second input feature map cache module 234 of the next-level multiplication and addition array module.

[0095] like Figure 7 , input feature row 2 exists in the second cache unit corresponding to the coordinates of output feature row 5, and also exists in the first cache unit corresponding to the coordinates of output feature rows 6, 7, and 8. When the detection of the last first cache unit in the direction of the arrow is completed, the on-chip reading and writing of input feature row 2 is started, and it is read from the first input feature map cache module 233 and written to the second input feature map cache module 234. The row of data is simultaneously written into the SRAM of the above-mentioned first cache unit. One row of data completes the cache of three rows of requirements, with an efficiency improvement of 300% and without any DDR interface access.

[0096] Therefore, the third multiplexing method mentioned above is implemented between the first input feature map cache module 233 and the second input feature map cache module 234.

[0097] See below Figure 8 , Figure 8 A schematic diagram of the fourth acceleration dimension of the input feature map cache module according to an embodiment of the present invention is shown. In this embodiment, the first cache unit and the second cache unit respectively include an odd column cache unit (such as an odd column SRAM) and an even column cache unit (such as an even column SRAM). In the first quantization mode, the data bit width of each column of the input feature map is N bits, and the interface bit width of the odd column cache unit and the even column cache unit is 2N bits. The odd column cache unit caches the spliced ​​data of two adjacent odd columns of the input image, and the even column cache unit caches the spliced ​​data of two adjacent even columns of the input image; in the second quantization mode, the data bit width of each column of the input feature map is 2N bits, and the interface bit width of the odd column cache unit and the even column cache unit is 2N bits. The odd column cache unit caches the data of an odd column of the input image, and the even column cache unit caches the data of an even column of the input image, wherein N is an integer greater than 0.

[0098] like Figure 8, corresponding to any cache module in the first input feature map cache module 233 and the second input feature map cache module 234, for a group of cache units corresponding to any output feature row, the feature of this architecture is that an SRAM cache module in a first cache unit or a second cache unit is composed of two basic SRAM units. And a row of input feature data is stored in the odd column SRAM and the even column SRAM respectively according to the numbering of the odd column and the even column. This embodiment supports both the first data quantization mode (such as 8-bit quantization) and the second quantization mode (16-bit quantization), so the read and write data interface bit width of the SRAM basic unit in this embodiment can be 16 bits.

[0099] based on Figure 8 The following table shows the parity SRAM storage architecture in which data is stored in the parity SRAM address in two scenarios: when a row of feature map data is 8-bit quantized data or 16-bit quantized data:

[0100] Addr(address) Even-numbered SRAM Odd-numbered SRAM 0 0,2 1,3 1 4,6 5,7 2 8,10 9,11 3 12,14 13,15 4 16,18 17,19 5 20,22 21,23 6 24,26 25,27 7 28,30 29,31

[0101] 8-bit quantized data

[0102] Addr(address) Even-numbered SRAM Odd-numbered SRAM 0 0 1 1 2 3 2 4 5 3 6 7 4 8 9 5 10 11 6 12 13 7 14 15

[0103] 16-bit quantized data

[0104] As shown in the table above, in 8-bit quantization mode, the width of each column data is 8 bits. The data of the 0th and 2nd columns are spliced ​​into 1 16-bit data and stored in the even column SRAM address 0; the data of the 1st and 3rd columns are spliced ​​into 1 16-bit data and stored in the odd column SRAM address 0. According to this rule, every time the address increases by 1 position, the four columns of data stored are based on the coordinates of 0, 2, 1, and 3, and each column coordinate increases by 4. For example, in address 1, the spliced ​​data of the even columns 4 and 6 are stored, and the spliced ​​data of the odd columns 5 and 7 are stored.

[0105] In 16-bit quantization mode, the width of each column of data is 16 bits. For the same group of odd-even SRAM arrays, in each address, starting from 0, each time the address increases by 1, the data coordinates stored are the even column coordinates plus 2, and the odd column coordinates plus 2. For example, in address 0, the data of even column 0 is stored, and the data of odd column 1 is stored. In address 1, the data of even column 2 is stored, and the data of odd column 3 is stored.

[0106] Furthermore, the depth of the above-mentioned parity column SRAM represents the maximum cache size in the column direction of the input feature map that can be supported by the present application. The depth can be specifically defined according to the indicators in the design and the chip performance and cost, and the total area consumption of the SRAM can be obtained.

[0107] Therefore, the same group of two SRAM arrays can support two quantized data structures.

[0108] Generally speaking, the coordinates of the two columns of input features required for calculating a column of output features usually differ by 1, and the coordinates may be equal at the boundary of the image. The parity cache architecture in this design ensures that any two adjacent parity columns of data can be read out at the same time in any clock cycle, so as to ensure that the two columns of input feature columns required for the calculation of each column of pixels on each output feature map can be read out at the same time in each clock cycle, thereby improving the computational pipeline efficiency of the multiplication and addition engine of the next-level module. The use of the above parity cache architecture can solve the problem of ensuring that any two adjacent columns of data are taken out from the SRAM array at the same time when there is no obvious regular column coordinate sequence.

[0109] For example, suppose that the first four columns of a row of output feature maps are 0, 1, 2, and 3 according to the column numbering; if the scene is a reduction of the image, it is possible that the input feature columns required for the calculation of these four columns of output features are grouped in order: 0 and 1; 3 and 4; 8 and 9; 13 and 14; it can be found that the group of 3 and 4 is quite special. Since a piece of SRAM can only access one address per clock cycle, the odd-even SRAM array in this application can ensure that the data required by the above feature sequence is provided at the same time in each clock cycle. When the 3rd and 4th columns of data are to be retrieved within a cycle, the even-numbered column SRAM read address is controlled to be written to 1, and the odd-numbered column SRAM read address is controlled to be written to 0. Then, in the next cycle of this cycle, the 3rd and 4th columns of data will appear on the data output signals of the odd-numbered column SRAM and the even-numbered column SRAM, respectively, at the same time.

[0110] Therefore, a pair of cache units in this embodiment requires at least four SRAMs, thereby realizing the fourth data multiplexing method of the present application.

[0111] The above-mentioned multiplexing methods can be implemented independently or in combination, and the present application is not limited thereto.

[0112] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A size transformation accelerator, characterized in that: include: A control module, comprising an interface register list and an output feature map coordinate control module, wherein the interface register list comprises a first register for configuring a scaling ratio of a feature map in a first direction and a second register for configuring a scaling ratio of a feature map in a second direction, wherein the first direction is perpendicular to the second direction, and the output feature map coordinate control module is used to traverse the coordinates of the output feature map, and generate a calculation parameter for size transformation based on the interface register list according to the coordinates of the output feature map; A feature map input control module, used to connect to the on-chip bus to obtain image data of the input feature map; A feature map multiplexing control module is used to cache image data of an input feature map and control the writing and reading of the cached image data of the input feature map, the feature map multiplexing control module comprising: at least one input feature map cache module, each of the input feature map cache modules comprising a plurality of pairs of cache units, each pair of cache units comprising a first cache unit and a second cache unit, to respectively cache two rows of data of the input feature map used to calculate one row of data of the current output feature map; a size conversion multiplication-addition array for performing multiplication-addition calculations on the image data of the input feature map read by the feature map multiplexing control module and the calculation parameters of the size conversion in a pipeline manner, so as to convert the image data of the input feature map into the image data of the output feature map according to the calculation parameters of the size conversion; and The feature map output control module is used to connect to the on-chip bus to send the image data of the output feature map to the on-chip bus.

2. The size-changing accelerator according to claim 1, wherein: The output feature map coordinate control module includes a first output feature map coordinate control module and a second output feature map coordinate control module to traverse the coordinates of the first direction and the second direction of the output feature map respectively.

3. The size-changing accelerator according to claim 1, wherein: The feature map input control module includes a direct storage access module, and the feature map input control module is configured to: Determine along the first traversal direction whether the rows of the input feature map to be cached by the first cache unit / the second cache unit of the plurality of pairs of cache units of the input feature map cache module are continuous and the same; If so, the direct storage access module is activated to read the row of data of the input feature map once and cache it in multiple consecutive and identical first cache units / second cache units.

4. The size-changing accelerator according to claim 1, wherein: The feature map input control module includes a direct storage access module, and the feature map input control module is configured to: Determine whether there are multiple second cache units having the same rows of input features to be cached by the first cache unit of the multiple pairs of cache units of the input feature map cache module; If so, the direct storage access module is started to read the row of data of the input feature map once and cache it in the corresponding first cache unit and second cache unit.

5. The size-changing accelerator according to claim 1, wherein: The input feature map cache module includes a first input feature map cache module and a second input feature map cache module, the first input feature map cache module and the second input feature map cache module both include a plurality of pairs of cache units, and the first input feature map cache module and / or the second input feature map cache module are configured as follows: Determine whether the row of the input feature map to be cached in the multiple pairs of cache units of the input feature map cache module is repeated in the row of the input feature map cached in the multiple pairs of cache units of another input feature map cache module; If so, directly read the rows of the input feature map cached in multiple pairs of cache units of another input feature map cache module.

6. The size-changing accelerator according to claim 5, characterized in that: The first input feature map cache module and the second input feature map cache module respectively include a first data reading interface and a second data reading interface, the first data reading interface respectively transmits data with the feature map input control module and the feature map output control module, and the second data reading interface is used to perform data transmission between the first input feature map cache module and the second input feature map cache module.

7. The size-changing accelerator according to claim 1, wherein: The first cache unit and the second cache unit include an odd-numbered column cache unit and an even-numbered column cache unit, respectively. In the first quantization mode, the data bit width of each column of the input feature map is N bits, the interface bit width of the odd column cache unit and the even column cache unit is 2N bits, the odd column cache unit caches the spliced ​​data of two adjacent odd columns of the input image, and the even column cache unit caches the spliced ​​data of two adjacent even columns of the input image; In the second quantization mode, the bit width of each column of the input feature map is 2N bits, the interface bit width of the odd column cache unit and the even column cache unit is 2N bits, the odd column cache unit caches data of an odd column of the input image, and the even column cache unit caches data of an even column of the input image, Wherein, N is an integer greater than 0.

8. The size-changing accelerator according to claim 1, wherein: The control module is also used to slice the input feature map and the output feature map, and the slicing direction is the row direction of the feature map.

9. A chip structure, characterized in that: include: The size-changing accelerator according to any one of claims 1 to 8; On-chip bus; as well as An on-chip processor is used to configure an interface register list of the size change accelerator.

Images