Convolution calculation method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses a convolution calculation method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: segmenting feature map data, and obtaining a plurality of sub-image blocks; segmenting the weight data in the plurality of groups of weight data groups to obtain sub weight blocks; obtaining a corresponding first target element and a corresponding second target element based on the clock period; broadcasting the first target element to all the computing units of the corresponding row in the corresponding computing array, and broadcasting the second target element to all the computing units of the corresponding column in the computing array; and outputting a target feature map based on the current convolution calculation result. The feature map data is segmented according to the parameters of the calculation array, so that the convolution calculation results of the image blocks in the same feature map direction are accumulated in the calculation unit, and extra intermediate result storage resources and chip resources for processing the stored intermediate results do not need to be consumed; and the area of the chip can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a convolution calculation method, device, electronic device and computer-readable storage medium. Background Art

[0002] In recent years, AI (Artificial Intelligence) technology has developed rapidly and achieved remarkable results, especially in the fields of image detection and recognition and language recognition, where the recognition rate of AI exceeds that of humans. Neural network processing is an important processing technology for realizing AI. At present, the convolution calculation process is to perform corresponding multiplication and accumulation operations on the convolution kernel weights and the elements in the input feature map to obtain an element of the target feature map, and then move the calculation window according to the stride, and repeat the above convolution calculation operation until all elements of the target feature map are calculated.

[0003] Due to the large amount of data required for convolution calculations and the limitations of hardware circuits, the input feature map and convolution kernel weights are split or converted into matrices and then split into blocks that match the hardware computing power for cyclic calculations. However, the intermediate results of each convolution calculation need to be stored in order to accumulate and obtain the convolution calculation results, which consumes a lot of area resources.

[0004] For AI chips, area equals computing power and area equals cost. Low-area chip design has always been one of the most important considerations in chip design. Summary of the invention

[0005] The main purpose of this application is to provide a convolution calculation method, device, electronic device and computer-readable storage medium, aiming to solve the technical problem of how to reduce the chip area of ​​convolution calculation.

[0006] To achieve the above purpose, the present application provides a convolution calculation method, including:

[0007] Based on the parameters of the calculation array and the feature map direction in the feature map data, the feature map data is segmented to obtain a plurality of image blocks and a plurality of sub-image blocks corresponding to each image block;

[0008] Based on the direction of the feature map, the weight data in the plurality of weight data groups are divided to obtain sub-weight blocks of each weight data in the weight data group;

[0009] Based on the clock cycle, a corresponding first target element is obtained in each sub-image block along the direction of the feature map, and a corresponding second target element is obtained in each sub-weight block along the direction of the feature map;

[0010] Broadcasting the first target element to all computing units in the corresponding row in the corresponding computing array, and broadcasting the second target element to all computing units in the corresponding column in the computing array, wherein the computing unit performs convolution calculation and accumulates the calculation result with the convolution calculation result of the previous cycle to obtain the current convolution calculation result;

[0011] Based on the current convolution calculation result, a target feature map is output.

[0012] In one embodiment, the step of acquiring the corresponding first target element in each sub-image block along the feature map direction based on the clock cycle, and acquiring the corresponding second target element in each sub-weight block along the feature map direction includes:

[0013] Acquire a target weight data group from the weight data group at intervals of a first preset clock cycle;

[0014] Determine the target image blocks in sequence according to the segmentation order of the image blocks at intervals of a second preset clock cycle, and obtain the target sub-weight blocks corresponding to the target image blocks in the target weight data group;

[0015] In each clock cycle, a corresponding first target element is obtained in each target sub-image block of the target image block along the direction of the feature map, and a second target element is obtained in the target sub-weight block along the direction of the feature map.

[0016] In one embodiment, the first preset clock period is obtained according to the number of the image blocks and the convolution kernel parameters in the parameters, the second preset clock period is obtained according to the convolution kernel parameters, and the first preset clock period is greater than the second preset clock period.

[0017] In one embodiment, the step of acquiring the corresponding first target element in each target sub-image block of the target image block along the feature map direction, and acquiring the second target element in the target sub-weight block along the feature map direction includes:

[0018] Obtain the corresponding first target element in the target sub-image block in sequence along the feature map direction according to the element order, wherein the element order is the order of elements in the first feature map belonging to the same sub-image block, and the element order corresponding to the sub-image block is the same as the element order corresponding to the sub-weight block;

[0019] Obtain the corresponding second target element in the target sub-weight block in the element order along the feature map direction.

[0020] In one embodiment, the step of outputting a target feature map based on the current convolution calculation result includes:

[0021] If the accumulated number of cycles corresponding to the current clock cycle reaches a preset threshold, the sub-image block corresponding to the target feature map is output based on the current convolution calculation result.

[0022] In one embodiment, the computing unit includes a multiplication-accumulation circuit, an adder, and an output register.

[0023] The multiplication and accumulation circuit is used to perform convolution calculation to obtain a calculation result, and transmit the calculation result to the adder;

[0024] A feedback channel is set between the adder and the output register, and the adder is used to obtain the convolution calculation result of the previous cycle through the feedback channel, and accumulate the calculation result with the convolution calculation result of the previous cycle to obtain the current convolution calculation result;

[0025] The output register is used to store the current convolution calculation result of the adder.

[0026] In one embodiment, if the accumulated number of cycles corresponding to the current clock cycle reaches a preset threshold, the current convolution calculation result is output through the output register of each computing unit, the sub-image block corresponding to the target feature map is obtained, and the current convolution calculation result stored in each of the computing units is cleared.

[0027] In one embodiment, the step of segmenting the feature map data based on the parameters of the calculation array and the feature map direction in the feature map data to obtain a plurality of image blocks and a plurality of sub-image blocks corresponding to each image block includes:

[0028] Based on the number of rows of the calculation array, the calculation unit parameters, the convolution kernel parameters, and the step size of the convolution calculation in the parameters, the feature map data is segmented according to the feature map direction to obtain multiple image blocks;

[0029] Based on the convolution kernel parameters and the step size, each of the image blocks is segmented to obtain a plurality of sub-image blocks corresponding to each image block.

[0030] In one embodiment, the step of dividing the feature map data according to the feature map direction based on the number of rows of the calculation array, the calculation unit parameters, the convolution kernel parameters and the step size of the convolution calculation in the parameters to obtain multiple image blocks includes:

[0031] Determine the number of first elements of the feature map direction corresponding to the first direction based on the calculation unit parameter;

[0032] Determine the number of second elements corresponding to the second direction based on the number of rows and the step size, and determine the number of third elements corresponding to the third direction based on the convolution kernel parameters;

[0033] Based on the first element quantity, the second element quantity and the third element quantity, the feature map data is segmented according to the feature map direction to obtain multiple image blocks.

[0034] In one embodiment, the step of segmenting the weight data in the plurality of weight data groups based on the feature map direction to obtain sub-weight blocks of each weight data in the weight data group comprises:

[0035] Based on the number of columns of the calculation array in the parameter and the amount of the weight data, the weight data are grouped to obtain a plurality of weight data groups;

[0036] The weight data in the weight data group is segmented based on the calculation unit parameters in the parameters to obtain the sub-weight blocks.

[0037] In addition, to achieve the above purpose, the present application also provides a convolution computing device, the convolution computing device comprising:

[0038] A first segmentation module is used to segment the feature map data based on the parameters of the calculation array and the feature map direction in the feature map data to obtain multiple image blocks and multiple sub-image blocks corresponding to each image block;

[0039] A second segmentation module is used to segment the weight data in the plurality of weight data groups based on the feature map direction to obtain sub-weight blocks of each weight data in the weight data group;

[0040] An acquisition module, configured to acquire the corresponding first target element in each sub-image block along the direction of the feature map based on a clock cycle, and acquire the corresponding second target element in each sub-weight block along the direction of the feature map;

[0041] A broadcasting module, used to broadcast the first target element to all computing units of the corresponding row in the corresponding computing array, and broadcast the second target element to all computing units of the corresponding column in the computing array, wherein the computing unit performs convolution calculation and accumulates the calculation result with the convolution calculation result of the previous cycle to obtain the current convolution calculation result;

[0042] The output module is used to output a target feature map based on the current convolution calculation result.

[0043] In addition, to achieve the above-mentioned purpose, the present application also provides an electronic device, which includes: a memory, a processor, and a convolution calculation program stored in the memory and executable on the processor, wherein the convolution calculation program implements the steps of the aforementioned convolution calculation method when executed by the processor.

[0044] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, on which a convolution calculation program is stored, and when the convolution calculation program is executed by a processor, the steps of the aforementioned convolution calculation method are implemented.

[0045] The present application divides the feature map data based on the parameters of the calculation array and the feature map direction in the feature map data to obtain multiple image blocks and multiple sub-image blocks corresponding to each image block; then divides the weight data in the multiple groups of weight data groups based on the feature map direction to obtain sub-weight blocks of each weight data in the weight data group; then based on the current clock cycle, obtains the corresponding first target element in each sub-image block along the feature map direction, and obtains the corresponding second target element in each sub-weight block along the feature map direction; then broadcasts the first target element to all the calculation units of the corresponding row in the corresponding calculation array, and broadcasts the second target element to all the calculation units of the corresponding column in the calculation array, wherein the calculation unit performs convolution calculation and accumulates the calculation result with the convolution calculation result of the previous cycle. Add to obtain the current convolution calculation result; finally, based on the current convolution calculation result, output the target feature map, by dividing the feature map data according to the parameters of the calculation array, the convolution calculation between each image block and the corresponding weight data in the same feature map direction can be given priority during the convolution calculation. During the convolution calculation process of each image block in the same feature map direction, each calculation unit in the calculation array accumulates the current calculation result with the convolution calculation result of the previous cycle to achieve the accumulation of the convolution calculation results of each image block in the same feature map direction within the calculation unit. The intermediate result (the current calculation result) does not need to be transmitted out of the calculation array for storage, and thus there is no need to consume additional intermediate result storage resources and chip resources for processing the stored intermediate results, which can effectively reduce the chip area.

[0046] Furthermore, by dividing the feature map data according to the parameters of the computing array, the convolution calculation process can fully utilize every computing unit in the computing array, thereby improving the area utilization rate of the computing array in the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of the first embodiment of the convolution calculation method of the present application;

[0048] Figure 2A schematic diagram of a possible data structure of feature map data in the convolution calculation method of this application;

[0049] Figure 3 A schematic diagram of a possible data structure of a sub-image block in an embodiment of the present application;

[0050] Figure 4 A possible data structure diagram of weight data in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of a possible structure of a computing array in an embodiment of the present application;

[0052] Figure 6 This is a schematic diagram of the structure of the calculation unit in the embodiment of the present application;

[0053] Figure 7 A schematic diagram of a possible data structure of a target feature map in an embodiment of the present application;

[0054] Figure 8 This is a schematic diagram of the module structure of the convolution computing device of this application;

[0055] Fig. 9 Schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0056] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0059] The main solution of the embodiment of the present application is: based on the parameters of the calculation array and the feature map direction in the feature map data, the feature map data is divided to obtain multiple image blocks and multiple sub-image blocks corresponding to each image block; based on the feature map direction, the weight data in the multiple groups of weight data groups is divided to obtain sub-weight blocks of each weight data in the weight data group; based on the current clock cycle, the corresponding first target element is obtained in each sub-image block along the feature map direction, and the corresponding second target element is obtained in each sub-weight block along the feature map direction; the first target element is broadcast to all the calculation units of the corresponding row in the corresponding calculation array, and the second target element is broadcast to all the calculation units of the corresponding column in the calculation array, wherein the calculation unit performs convolution calculation and accumulates the calculation result with the convolution calculation result of the previous cycle to obtain the current convolution calculation result; based on the current convolution calculation result, the target feature map is output.

[0060] At present, due to the large amount of data in convolution calculation and the limitation of hardware circuit, the input feature map (feature map data) and convolution kernel weight (weight data) are split or converted into a matrix and then split into blocks that match the hardware computing power for cyclic calculation. However, the intermediate results of each convolution calculation need to be stored for accumulation to obtain the convolution calculation results, which consumes a lot of chip area resources.

[0061] The present application divides the feature map data and the weight data respectively according to the parameters of the computing array in the chip and the feature map direction in the feature map data. During the convolution calculation, the convolution calculation between each image block in the same feature map direction and the corresponding weight data is given priority. During the convolution calculation process of each image block in the same feature map direction, each computing unit in the computing array accumulates the current calculation result with the convolution calculation result of the previous cycle to obtain the current convolution calculation result, and then accumulates the convolution calculation results of each image block in the same feature map direction in the computing unit. When the convolution calculation of each image block in the same feature map direction is completed, the current convolution calculation result corresponding to the feature map direction, that is, the data of the corresponding point in the target feature map, is output. The intermediate result (the current calculation result) does not need to be transmitted out of the computing array for storage, and thus there is no need to consume additional intermediate result storage resources and chip resources for processing the stored intermediate results, which can effectively reduce the area of ​​the chip.

[0062] It should be noted that the execution subject of this embodiment can be an electronic device, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, etc., or an AI chip capable of realizing the above functions, etc., and this embodiment does not specifically limit this. The following takes the electronic device as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0063] Based on this, this application proposes a convolution calculation method of the first embodiment, please refer to Figure 1 , the convolution calculation method includes steps S10 to S11:

[0064] Step S10, based on the parameters of the calculation array and the feature map direction in the feature map data, the feature map data is segmented to obtain a plurality of image blocks and a plurality of sub-image blocks corresponding to each image block;

[0065] It should be noted that the computing array of the electronic device is formed by multiple computing units. The parameters of the computing array include the number of rows m of the computing array, the number of columns n of the computing array, the parameters of each computing unit in the computing array, the convolution kernel parameters corresponding to the computing unit, and the step size of the convolution calculation. The number of rows refers to the number of rows of computing units in the computing array, the number of columns refers to the number of columns of computing units in the computing array, the computing unit parameters refer to the number of multiplication and accumulation units in the computing unit, the computing unit parameters of each computing unit in the computing array are the same, the convolution kernel parameter is kernel_size, and the convolution kernel parameter kernel_size indicates that the number of elements in each convolution calculation window is kernel_size*kernel_size.

[0066] The electronic device obtains feature map data (input feature map input feature map), such as Figure 2 As shown, the feature map data is a three-dimensional structure with three directions: w, h, and c. Figure 2 The sizes of the feature map data in the three directions of w, h, and c are (iw, ih, ic) respectively. The feature map data stores iw*ih*ic (* represents multiplication) elements, where the c direction is the feature map direction, and ic is the number of feature maps in the feature map direction, that is, the feature map data includes ic feature maps.

[0067] The electronic device divides the feature map data based on the parameters of the calculation array and the feature map direction in the feature map data to obtain multiple image blocks and multiple sub-image blocks corresponding to each image block, that is, the electronic device obtains the number of rows of the calculation array, the calculation unit parameters, the convolution kernel parameters and the step size of the convolution calculation in the parameters, and divides the feature map data according to the feature map direction to obtain multiple image blocks; specifically, the standard size of the image block ifm_tile can be determined as (w0, h0, c0) based on the number of rows m, the calculation unit parameters, the convolution kernel parameters kernel_size and the step size stride of the convolution calculation. Next, the electronic device divides the feature map data according to the standard size with the feature map direction as the first dividing order to obtain multiple image blocks. Specifically, the dividing order is first the c direction, then the h direction, and finally the w direction, or the dividing order is first the c direction, then the w direction, and finally the h direction, such as Figure 2 As shown, Figure 2 The order of segmentation is first in the c direction, then in the w direction, and finally in the h direction. If the last ifm_tile in the current feature map direction is inconsistent with the standard size, it is filled with 0 to supplement the standard size (w0, h0, c0). The size of each image block ifm_tile is the same, which is the standard size (w0, h0, c0).

[0068] After the image blocks are segmented, for each image block, the image blocks are segmented based on the convolution kernel parameters and the stride to obtain multiple sub-image blocks corresponding to each image block. For example, the convolution kernel parameter kernel_size is 3, the stride is 1, and the number of rows in the calculation array is m=16. At this time, Figure 2 As shown, if the order of segmentation is first c direction, then w direction, and finally h direction, that is, the first direction is c direction, the second direction is w direction, and the third direction is h direction, w0=18, the image block is segmented based on the kernel_size and stride to obtain 16 sub-image blocks ifm_block of each image block, that is Figure 2 Among ifm_block_1, ifm_block_2, and ifm_block_m, the sizes of the sub-image blocks are all kernel_size*kernel_size=3*3.

[0069] Reference Figure 3 , Figure 3 Schematic diagram of a possible sub-image block, the size of the sub-image block is (h0, h0, c0). For example, when kernel_size=3, the sub-image block is (3, 3, c0), that is, each side of the sub-image block in the direction of the feature map includes 3*3=9 elements.

[0070] By dividing the feature map data according to the parameters of the calculation array and the feature map direction, the image blocks in the same feature map direction can be calculated according to the feature map direction during the convolution calculation, so that the convolution calculation results of each image block in the same feature map direction can be accumulated in the calculation unit, so that the intermediate results of the convolution calculation do not need to be transmitted outside the calculation array for storage, and there is no need to consume additional intermediate result storage resources and chip resources for processing the stored intermediate results, which can effectively reduce the chip area.

[0071] Step S20, dividing the weight data in the plurality of weight data groups based on the direction of the feature map to obtain sub-weight blocks of each weight data in the weight data group;

[0072] After obtaining the weight data, the electronic device groups the weight data based on the number of columns of the calculation array in the parameter and the number of weight data to obtain multiple groups of weight data. The sizes of each weight data weight in the three directions of w, h, and c are respectively (kernel_size, kernel_size, ic). The sizes of weight in the two directions of w and h are the same. When the feature map data is segmented in the order of first the c direction, then the w direction, and finally the h direction, as shown in FIG. Figure 4 As shown, the size of the weight data is (h0, h0, ic).

[0073] The electronic device groups the weight data based on the number of columns of the calculation array and the number of weight data in the parameter to obtain multiple weight data groups. For the last weight data group, if the number of weight data is less than n, 0 can be filled to supplement n weight data. For each weight data in any weight data group, the electronic device divides the weight data in the weight data group based on the calculation unit parameter in the parameter to obtain a sub-weight block, that is, the size of the sub-weight block is (h0, h0, c0), such as Figure 4 As shown, Figure 4 In the example, the first weight data weight_1 can be divided into weight_block_1, weight_block_2, and so on.

[0074] Since the structure of the sub-weight block is exactly the same as that of the sub-data block, Figure 3 It can also be a possible schematic diagram of a sub-weight block, where the size of the sub-weight block is (h0, h0, c0). For example, when kernel_size = 3, the sub-weight block is (3, 3, c0), that is, each side of the sub-weight block in the direction of the feature map includes 3*3=9 elements.

[0075] Step S30, based on the current clock cycle, obtaining the corresponding first target element in each sub-image block along the direction of the feature map, and obtaining the corresponding second target element in each sub-weight block along the direction of the feature map;

[0076] After obtaining the sub-image blocks and sub-weight blocks, when performing convolution calculations, each clock cycle is based on the current clock cycle to obtain the corresponding first target element in each sub-image block along the direction of the feature map, and to obtain the corresponding second target element in each sub-weight block along the direction of the feature map.

[0077] Specifically, when performing convolution calculation, the first preset clock cycle can be determined in advance according to the number of image blocks and the convolution kernel parameter kernel_size, and the second preset clock cycle can be determined according to the convolution kernel parameter kernel_size, wherein the first preset clock cycle = the number of image blocks * kernel_size * kernel_size, the second preset clock cycle = kernel_size * kernel_size, taking the convolution kernel parameter kernel_size = 3 as an example, the first preset clock cycle = the number of image blocks * 9, the second preset clock cycle = 9, that is, 9 clock cycles can complete the convolution calculation of an image block, then, the electronic device obtains the target weight data group in the weight data group at intervals of the first preset clock cycle; determines the target image block in turn according to the segmentation order of the image block at intervals of the second preset clock cycle, and obtains the target sub-weight block corresponding to the target image block in the target weight data group; in each clock cycle, obtains the corresponding first target element in each target sub-image block of the target image block along the direction of the feature map, and obtains the second target element in the target sub-weight block along the direction of the feature map.

[0078] Step S40, broadcasting the first target element to all computing units in the corresponding row in the corresponding computing array, and broadcasting the second target element to all computing units in the corresponding column in the computing array, wherein the computing unit performs convolution calculation and accumulates the calculation result with the convolution calculation result of the previous cycle to obtain the current convolution calculation result;

[0079] After acquiring the first target element, the electronic device broadcasts the first target element to all computing units in the corresponding row in the corresponding computing array, and broadcasts the second target element to all computing units in the corresponding column in the computing array, that is, the electronic device broadcasts the elements of each sub-image block in the first target element to all computing units in the corresponding row in the computing array, and at the same time, the electronic device broadcasts the elements of each sub-weight block in the second target element to all computing units in the corresponding column in the computing array.

[0080] Specifically, Figure 5 As shown, taking the first image block ifm_tile and the first group of weight data as an example, the first image block ifm_tile includes ifm_block_1 to ifm_block_m, and the first group of weight data includes n weight data weight_1 to weight_n, as shown in FIG. Figure 3As shown, the convolution kernel parameter kernel_size is h0, the first target element can be the c0 element in the feature map direction corresponding to the first element in ifm_block_1 to ifm_block_m, and the second target element is the c0 element in the feature map direction corresponding to the first element of the first sub-weight block weight_block in weight_1 to weight_n. The electronic device broadcasts the c0 element in the feature map direction corresponding to the first element in ifm_block_1 to all computing units in the first row of the corresponding computing array, and broadcasts the c0 element in the feature map direction corresponding to the first element in ifm_block_2 to all computing units in the second row of the corresponding computing array, and so on, until ifm_block_n is The c0 element in the feature map direction corresponding to the first element in lock_m is broadcast to all computing units in the mth row of the corresponding computing array. At the same time, the electronic device broadcasts the c0 element in the feature map direction corresponding to the first element of the first sub-weight block weight_block in weight_1 to all computing units in the first column of the computing array, and broadcasts the c0 element in the feature map direction corresponding to the first element of the first sub-weight block weight_block in weight_2 to all computing units in the second column of the computing array, and so on, until the c0 element in the feature map direction corresponding to the first element of the first sub-weight block weight_block in weight_n is broadcast to all computing units in the nth column of the computing array.

[0081] In a possible implementation, Figure 6 As shown, the calculation unit includes a multiplication and accumulation circuit, an adder and an output register, and a feedback channel is set between the adder and the output register.

[0082] The multiplication-accumulation circuit is used to perform convolution calculation to obtain a calculation result, and transmit the calculation result to the adder, that is, the multiplication-accumulation circuit performs corresponding multiplication-accumulation calculations according to the c0 elements in the received first target element and the c0 elements in the second target element to obtain the calculation result, and transmits the calculation result to the adder, for example, the calculation result is first transmitted to the intermediate result register, and then transmitted to the adder.

[0083] The adder is used to obtain the convolution calculation result of the previous cycle through the feedback channel, accumulate the calculation result with the convolution calculation result of the previous cycle, and obtain the current convolution calculation result. In each clock cycle, the output register will return the convolution calculation result of the previous cycle to the adder through the feedback channel, and the adder will add the calculation result of the current clock cycle with the convolution calculation result of the previous cycle to obtain the current convolution calculation result.

[0084] The output register is used to store the current convolution calculation result of the adder. After the adder obtains the current convolution calculation result, the current convolution calculation result is transmitted to the output register. The output register stores the current convolution calculation result and can transmit the stored current convolution calculation result as the convolution calculation result of the previous cycle to the adder through the return channel in the next clock cycle.

[0085] By setting up a return channel to return and accumulate the convolution calculation results of the previous cycle, the convolution calculation results of each image block in the same feature map direction can be accumulated in the computing unit. When the convolution calculation of each image block in the same feature map direction is completed, the data of the corresponding point in the target feature map is output. The intermediate result (the current convolution calculation result) does not need to be transmitted out of the computing array for storage, and there is no need to consume additional intermediate result storage resources and chip resources for processing the stored intermediate results, which can effectively reduce the chip area.

[0086] Step S50: output a target feature map based on the current convolution calculation result.

[0087] After each computing unit obtains the current convolution calculation result, it determines whether data output is required based on the current clock cycle, that is, whether the preset output conditions of the target feature map are met. If so, a sub-image block of the target feature map is output according to the current convolution calculation result. That is, according to the preset output rule, the current convolution calculation result of each computing unit is used as a point in the sub-image block in the target feature map, and the current convolution calculation results of all computing units form a sub-image block of the target feature map.

[0088] In a feasible implementation manner, step S50 includes:

[0089] Step S51, if the accumulated number of cycles corresponding to the current clock cycle reaches a preset threshold, the sub-image block corresponding to the target feature map is output based on the current convolution calculation result.

[0090] In each clock cycle, the electronic device accumulates the accumulated number of clock cycles and determines whether the accumulated number of cycles reaches a preset threshold, wherein the preset threshold can be determined according to the convolution kernel parameter, the calculation unit parameter and the number of feature maps in the feature map data, the number of weight data is the same as the number of feature maps in the target feature map, the convolution kernel parameter is kernel_size, the calculation unit parameter is the number of multiplication and accumulation units in the calculation unit c0 and the number of feature maps in the feature map data is ic, and the preset To round up, is the number of image blocks in each feature map direction.

[0091] If the accumulated number of cycles corresponding to the current clock cycle reaches a preset threshold, the electronic device outputs a sub-image block of the target feature map according to the current convolution calculation result, that is, according to the preset output rule, the current convolution calculation result of each computing unit is used as a point of the sub-image block in the target feature map, and the current convolution calculation result of the computing unit forms a sub-image block of the target feature map.

[0092] In a feasible implementation, if the accumulated number of cycles corresponding to the current clock cycle reaches a preset threshold, the current convolution calculation result is output through the output register of each computing unit, the sub-image block corresponding to the target feature map is obtained, and the current convolution calculation result stored in each of the computing units is cleared.

[0093] When the accumulated number of cycles corresponding to the current clock cycle reaches a preset threshold, the electronic device outputs the current convolution calculation result through the output register of each computing unit to obtain the sub-image block corresponding to the target feature map. At this time, the electronic device clears the current convolution calculation result stored in the computing unit to facilitate the subsequent convolution calculation between other image blocks and weight data.

[0094] By dividing the feature map data based on the parameters of the calculation array and the feature map direction in the feature map data, a plurality of image blocks and a plurality of sub-image blocks corresponding to each image block are obtained; then the weight data in the plurality of weight data groups are divided based on the feature map direction to obtain a sub-weight block of each weight data in the weight data group; then based on the current clock cycle, the corresponding first target element is obtained in each sub-image block along the feature map direction, and the corresponding second target element is obtained in each sub-weight block along the feature map direction; then the first target element is broadcast to all the calculation units of the corresponding row in the corresponding calculation array, and the second target element is broadcast to all the calculation units of the corresponding column in the calculation array, wherein the calculation unit performs convolution calculation and accumulates the calculation result with the convolution calculation result of the previous cycle , obtain the current convolution calculation result; finally, based on the current convolution calculation result, output the target feature map, by segmenting the feature map data according to the parameters of the calculation array, the convolution calculation between each image block and the corresponding weight data in the same feature map direction can be given priority during the convolution calculation. During the convolution calculation process of each image block in the same feature map direction, each calculation unit in the calculation array accumulates the current calculation result with the convolution calculation result of the previous cycle to achieve the accumulation of the convolution calculation results of each image block in the same feature map direction within the calculation unit. The intermediate result (the current calculation result) does not need to be transmitted out of the calculation array for storage, and thus there is no need to consume additional intermediate result storage resources and chip resources for processing the stored intermediate results, which can effectively reduce the chip area.

[0095] Furthermore, by dividing the feature map data according to the parameters of the computing array, the convolution calculation process can fully utilize every computing unit in the computing array, thereby improving the area utilization rate of the computing array in the electronic device.

[0096] In a feasible implementation, step S30 may include steps S31 to S33:

[0097] Step S31, obtaining a target weight data group from the weight data group at intervals of a first preset clock cycle;

[0098] Step S33, determining the target image block in sequence according to the segmentation order of the image block at intervals of the second preset clock cycle, and obtaining the target sub-weight block corresponding to the target image block in the target weight data group;

[0099] Step S33, in each clock cycle, the corresponding first target element is obtained in each target sub-image block of the target image block along the direction of the feature map, and the second target element is obtained in the target sub-weight block along the direction of the feature map.

[0100] Among them, the first preset clock cycle is obtained according to the number of the image blocks and the convolution kernel parameters in the parameters, the second preset clock cycle is obtained according to the convolution kernel parameters, and the first preset clock cycle is greater than the second preset clock cycle.

[0101] When performing convolution calculations, the first preset clock cycle can be determined in advance based on the number of image blocks and the convolution kernel parameter kernel_size, and the second preset clock cycle can be determined based on the convolution kernel parameter kernel_size, where the first preset clock cycle = the number of image blocks * kernel_size * kernel_size, the second preset clock cycle = kernel_size * kernel_size, taking the convolution kernel parameter kernel_size = 3 as an example, the first preset clock cycle = the number of image blocks * 9.

[0102] The electronic device obtains a target weight data group from the weight data group at an interval of a first preset clock cycle. Specifically, when the weight data are grouped, the weight data groups can be sorted to obtain a third order. The electronic device obtains the target weight data group from the weight data group in sequence according to the third order at an interval of the first preset clock cycle, so that the currently obtained target weight data group is different from the target weight data group obtained in the previous first preset clock cycle, and the currently obtained target weight data group is weight data without convolution calculation.

[0103] Then, the electronic device determines the target image blocks in sequence according to the segmentation order of the image blocks at intervals of the second preset clock cycle. For example, when segmenting the image blocks, the image blocks can be grouped according to the feature map direction to obtain multiple image block groups, each image block group includes multiple image blocks in the same feature map direction, and the image blocks are sorted according to the segmentation order to obtain a first order. For each image block group, the image blocks in the image block group are sorted according to the segmentation order in the feature map direction to obtain a second order. The electronic device determines the target image block group in sequence according to the first order at intervals of the second preset clock cycle, and sequentially obtains the target image blocks in the target image block group in sequence according to the second order. Alternatively, the electronic device determines the target image block group in sequence according to the first order at intervals of the second preset clock cycle, and sequentially obtains the target image blocks in the target image block group. The target image block is an image block that is not currently convolved with the target weight data group. Since all elements in the feature map direction (c direction) in the target image block group are multiplied by the corresponding weights, they will be accumulated, so it is not necessary to obtain the target image block in the target image block in sequence.

[0104] At the same time, the electronic device obtains the target sub-weight block corresponding to the target image block in the target weight data group at an interval of a second preset clock cycle, wherein the segmentation order of the target sub-weight block in the c direction is the same as the segmentation order of the target image block in the c direction to ensure the accuracy of the convolution calculation.

[0105] After acquiring the target image block and the target sub-weight block, in each clock cycle, the electronic device acquires the corresponding first target element in each target sub-image block of the target image block along the direction of the feature map, and acquires the second target element in the target sub-weight block along the direction of the feature map, wherein the first target element is an uncalculated element in each target sub-image block along the direction of the feature map, the elements of the second target column are uncalculated elements in each target sub-weight block along the direction of the feature map, and the order of the first target elements in the target sub-image block is the same as the order of the second target elements in the target sub-weight block.

[0106] In a feasible implementation manner, step S33 may include steps S331 to S332:

[0107] Step S331, sequentially obtaining the corresponding first target element in the target sub-image block along the feature map direction according to the element order, wherein the element order is the order of elements in the first feature map belonging to the same sub-image block, and the element order corresponding to the sub-image block is the same as the element order corresponding to the sub-weight block;

[0108] Step S332, obtaining the corresponding second target element in the target sub-weight block in the direction of the feature map in element order.

[0109] It should be noted that the element order can be set in advance. For example, for the elements of the planes corresponding to the w direction and the h direction in the sub-image block, different orders are set according to kernel_size*kernel_size. The element order is the order of the elements in the first feature map belonging to the same sub-image block. The element order corresponding to the sub-image block is the same as the element order corresponding to the sub-weight block, and the element order of the elements at the same position in each sub-image block is the same.

[0110] The electronic device sequentially obtains the corresponding first target element in the target sub-image block along the feature map direction according to the element order, such as Figure 3 As shown, the convolution kernel parameter kernel_size is h0. If Figure 3 is the data structure of the sub-image block, for example, Figure 3 The sub-image block is ifm_block_1, which belongs to the first ifm_tile (image block). In the first clock cycle, the c0 elements of the first element (element 1) in the ifm_block_1 in the feature map direction (c direction) are obtained as the elements of the sub-image block in the first target element. And so on, the c0 elements of ifm_block_2 to ifm_block_n in the feature map direction (c direction) in the first ifm_tile are obtained respectively, and then the first target element is obtained.

[0111] At the same time, the electronic device sequentially obtains the corresponding second target element in the target sub-weight block along the feature map direction according to the element order, such as Figure 3 As shown, Figure 3 It can also be a data structure of a sub-weight block, for example, Figure 3 The neutron weight block is the first weight_block of weight_1 in the first group of weight data groups. In the first clock cycle, c0 elements of the first element (element 1) in the weight_block in the direction of the characteristic graph (c direction) are obtained. Similarly, the electronic device simultaneously obtains c0 elements of the first element (element 1) in the first weight_block of weight_2 to weight_n in the first group of weight data groups in the direction of the characteristic graph (c direction), and then obtains the second target element.

[0112] By acquiring the first target element and the second target element in element order, the acquired first target element and the second target element are both uncalculated elements, and the first target element corresponds to the second target element one-to-one, thereby improving the accuracy of the convolution calculation and improving the efficiency of the convolution calculation.

[0113] A target weight data group is obtained in the weight data group at intervals of a first preset clock cycle; then, the target image blocks are determined in turn according to the segmentation order of the image blocks at intervals of a second preset clock cycle, and the target sub-weight blocks corresponding to the target image blocks are obtained in the target weight data group; and then in each clock cycle, the corresponding first target elements are obtained in each target sub-image block of the target image block along the feature map direction, and the second target elements are obtained in the target sub-weight block along the feature map direction, and the image blocks and weight blocks are obtained according to the preset clock cycle to realize the convolution calculation between each image block and the corresponding weight data in the same feature map direction within the preset clock cycle, and then the convolution calculation results of each image block in the same feature map direction are accumulated in the calculation unit, and the intermediate results (current calculation results) do not need to be transmitted out of the calculation array for storage, and thus there is no need to consume additional intermediate result storage resources and chip resources for processing the stored intermediate results, which can effectively reduce the chip area.

[0114] Reference Figures 2 to 7 , the following takes the convolution calculation starting from the first clock cycle as an example:

[0115] In the first clock cycle, the c0*m elements corresponding to the first element in ifm_block_1 to ifm_block_m cut from the first ifm_tile in the c direction are sent to the m rows of the calculation array, and the c0 elements of each ifm_block are broadcast to all the calculation units in each row; at the same time, the c0*n elements corresponding to the first element in the n weight_block_1 cut from the first group of n weights in the c direction are sent to the n columns of the calculation array, and the c0 elements of each weight_block_1 are broadcast to all the calculation units in each column. The calculation unit performs the corresponding convolution calculation and stores the result in the output register, where the data returned to the adder from the output register in the first clock cycle is 0.

[0116] In the second clock cycle, the c0*m elements corresponding to the second element in ifm_block_1 to ifm_block_m cut from the first ifm_tile in the c direction are sent to the m rows of the calculation array, and the c0 elements of each ifm_block are broadcast to all the calculation units in each row; at the same time, the c0*n elements corresponding to the second element in the n weight_block_1 cut from the first group of n weights in the c direction are sent to the n columns of the calculation array, and the c0 elements of each weight_block_1 are broadcast to all the calculation units in each column, and the calculation units perform corresponding convolution calculations to obtain calculation results, add the calculation results to the calculation results of the previous clock cycle through the adder, and store the added results (current convolution calculation results) in the output register.

[0117] By analogy, in the h0*h0th clock cycle, the c0*m elements corresponding to the h0*h0th elements in the ifm_block_1 to ifm_block_m cut from the first ifm_tile are sent to the m rows of the calculation array, and the c0 elements of each ifm_block are broadcast to all the calculation units in each row; at the same time, the c0*n elements corresponding to the h0*h0th elements in the n weight_block_1 cut from the first group of n weights are sent to the n columns of the calculation array in the c direction, and the c0 elements of each weight_block_1 are broadcast to all the calculation units in each column, and the calculation units perform corresponding convolution calculations to obtain calculation results, add the calculation results to the calculation results of the previous clock cycle through the adder, and store the added results (current convolution calculation results) in the output register.

[0118] In the h0*h0+1th clock cycle, the c0*m elements corresponding to the first element in ifm_block_1 to ifm_block_m cut from the second ifm_tile are sent to the m rows of the calculation array, and the c0 elements of each ifm_block are broadcast to all the calculation units in each row; at the same time, the c0*n elements corresponding to the first element in the n weight_block_2 cut from the first group of n weights are sent to the n columns of the calculation array in the c direction, and the c0 elements of each weight_block_2 are broadcast to all the calculation units in each column. The calculation units perform corresponding convolution calculations to obtain calculation results, add the calculation results to the calculation results of the previous clock cycle through the adder, and store the added results (current convolution calculation results) in the output register.

[0119] In the h0*h0+2th clock cycle, the c0*m elements corresponding to the second element in ifm_block_1 to ifm_block_m cut from the second ifm_tile are sent to the m rows of the calculation array, and the c0 elements of each ifm_block are broadcast to all the calculation units in each row; at the same time, the c0*n elements corresponding to the second element in the n weight_block_2 cut from the first group of n weights are sent to the n columns of the calculation array in the c direction, and the c0 elements of each weight_block_2 are broadcast to all the calculation units in each column. The calculation units perform corresponding convolution calculations to obtain calculation results, add the calculation results to the calculation results of the previous clock cycle through the adder, and store the added results (current convolution calculation results) in the output register.

[0120] Similarly, after all ifm_tile calculations in the c direction corresponding to the first ifm_tile are completed, the current convolution calculation result is output to form a sub-image block of the output feature map, such as Figure 7 The ofm_block_1 shown in . Then, along the w direction, get the ifm_tile adjacent to the first ifm_tile, and repeat the convolution calculation with the first group of n weights as in the previous steps. After the calculation, the result is output as ofm_block_2. And so on, until all image blocks are convolved with the first group of n weights, the last ofm_block is output, and an image block of the output feature map is formed, as shown in Figure 7 Then change the weight, and use the second group of n weights and all image groups to calculate based on the above convolution calculation method, and output the second image block of the feature map, as shown in Figure 7 The second ofm_tile is shown in . This cycle is repeated to complete the calculation of all weights, and finally the final target feature map (output featuremap) is obtained, with the sizes of ow, oh, and oc in the w, h, and c directions.

[0121] In a feasible implementation, step S10 may include steps S11 to S13:

[0122] Step S11, based on the number of rows of the calculation array, the calculation unit parameters, the convolution kernel parameters and the step size of the convolution calculation in the parameters, the feature map data is segmented according to the feature map direction to obtain multiple image blocks;

[0123] Step S12: based on the convolution kernel parameters and the step size, each of the image blocks is segmented to obtain a plurality of sub-image blocks corresponding to each image block.

[0124] It should be noted that the parameters of the calculation array include the number of rows m of the calculation array, the number of columns n of the calculation array, the parameters of each calculation unit in the calculation array, the convolution kernel parameters corresponding to the calculation unit, and the step size of the convolution calculation. The number of rows refers to the number of rows of calculation units in the calculation array, the number of columns refers to the number of columns of calculation units in the calculation array, the calculation unit parameters refer to the number of multiplication and accumulation units in the calculation unit, the calculation unit parameters of each calculation unit in the calculation array are the same, the convolution kernel parameter is kernel_size, and the convolution kernel parameter kernel_size indicates that the number of elements in each convolution calculation window is kernel_size*kernel_size.

[0125] The electronic device obtains feature map data (input feature map input feature map), such as Figure 2 As shown, the feature map data is a three-dimensional structure with three directions: w, h, and c. Figure 2 The sizes of the feature map data in the three directions of w, h, and c are (iw, ih, ic) respectively. The feature map data stores iw*ih*ic (* represents multiplication) elements, where the c direction is the feature map direction, and ic is the number of feature maps in the feature map direction, that is, the feature map data includes ic feature maps.

[0126] After obtaining the feature map data and weight data, the electronic device obtains the number of rows of the calculation array, the calculation unit parameters, the convolution kernel parameters and the step size of the convolution calculation in the parameters, and divides the feature map data according to the feature map direction to obtain multiple image blocks; specifically, the standard size of the image block ifm_tile can be determined as (w0, h0, c0) based on the number of rows m, the calculation unit parameters, the convolution kernel parameters kernel_size and the step size stride of the convolution calculation. Further, in a feasible implementation, step S11 can include steps S111 to S113:

[0127] S111, determining the number of first elements of the feature map direction corresponding to a first direction based on the calculation unit parameter;

[0128] S113, determining the number of second elements corresponding to the second direction based on the number of rows and the step size, and determining the number of third elements corresponding to the third direction based on the convolution kernel parameters;

[0129] S113: Based on the first number of elements, the second number of elements, and the third number of elements, the feature map data is segmented along the feature map direction to obtain a plurality of image blocks.

[0130] In order to reduce the off-chip storage of intermediate data during the convolution calculation process, when the feature map data is segmented, it is first segmented according to the feature map direction, so that the convolution calculation of the data in the same feature map direction is completed first during the convolution calculation. Then, the electronic device determines the number of first elements corresponding to the first direction of the feature map direction based on the calculation unit parameters, referring to Figure 2 As shown, the first direction, that is, the feature map direction, is the c direction, and the first element number c0 = calculation unit parameter.

[0131] Next, the electronic device determines the number of second elements corresponding to the second direction based on the number of rows and the step size, and determines the number of third elements corresponding to the third direction based on the convolution kernel parameter. The number of second elements corresponds to the number of rows and the step size, so that the image block can be divided into m sub-image blocks with the number of rows according to the step size. If the order of division is first the c direction, then the h direction, and finally the w direction, that is, the first direction is the c direction, the second direction is the h direction, and the third direction is the w direction, then c0 = calculation unit parameter, w0 = kernel_size, and h0 is determined according to the number of rows m, the step size stride, and the convolution kernel parameter kernel_size. Figure 2 As shown, if the order of segmentation is first c direction, then w direction, and finally h direction, that is, the first direction is c direction, the second direction is w direction, and the third direction is h direction, then c0 = calculation unit parameter, h0 = kernel_size, and w0 is determined according to the number of rows m, step size stride, and convolution kernel parameter kernel_size.

[0132] After obtaining the first number of elements, the second number of elements, and the third number of elements, that is, obtaining the standard size of the image block ifm_tile as (w0, h0, c0), the feature map data is segmented according to the feature map direction based on the first number of elements, the second number of elements, and the third number of elements to obtain multiple image blocks. Specifically, based on the first number of elements, the second number of elements, and the third number of elements, the data of the feature map direction in the feature map data is segmented from the starting position, such as Figure 2 As shown, Figure 2 The order of segmentation is first in the c direction, then in the w direction, and finally in the h direction. The electronic device takes the first element in the upper left corner of the feature map data as the starting point, and segments the feature map data in the feature map direction according to the standard size to obtain the first ifm_tile, the second ifm_tile, and so on, until the segmentation of ic input feature maps in the feature map direction is completed. If the last ifm_tile in the current feature map direction is inconsistent with the standard size, fill it with 0 to supplement it to the standard size (w0, h0, c0).

[0133] When the data segmentation corresponding to the starting position in the feature map data is completed, the first segmentation position is determined based on the second direction, the second element number and the starting position, the first segmentation position is used as the starting position, and the step of segmenting the data of the feature map direction pair in the feature map data from the starting position based on the first element number, the second element number and the third element number is returned; Figure 2 As shown, the first segmentation position is the position in the feature map data adjacent to the previously segmented ifm_tile, that is, the position in the feature map data adjacent to the first ifm_tile along the w direction. The feature map data is segmented according to the feature map direction according to the standard size. After the data segmentation in the current feature map direction is completed, the first segmentation position is determined again, and the segmentation is performed cyclically until the data segmentation in the current w direction is completed.

[0134] When the data corresponding to the second direction in the feature map data is segmented, the second segmentation position is determined based on the second direction, the number of third elements and the starting position, the second segmentation position is used as the starting position, and the step of segmenting the data of the feature map direction in the feature map data from the starting position based on the number of first elements, the number of second elements and the number of third elements is returned to execute until the segmentation of the feature map data is completed to obtain multiple image blocks.

[0135] like Figure 2 As shown, the second segmentation position is the position adjacent to the first ifm_tile in the feature map data along the h direction ( Figure 2 ), firstly, along the h direction, at a position adjacent to the first ifm_tile of the feature map data, the feature map data is segmented according to the feature map direction according to the standard size, and then the second segmentation position is determined again along the w direction, that is, the second segmentation position is determined along the w direction at a position adjacent to the currently segmented ifm_tile, and a loop segmentation is performed until the data segmentation in the current w direction is completed, and when the data segmentation corresponding to the second direction in the feature map data is completed, the loop is executed again, and the second segmentation position is determined based on the second direction, the number of third elements and the starting position until the feature map data segmentation is completed.

[0136] Among them, the size of each image block ifm_tile is the same, which is a standard size (w0, h0, c0).

[0137] After the image blocks are segmented, for each image block, the image blocks are segmented based on the convolution kernel parameters and the stride to obtain multiple sub-image blocks corresponding to each image block. For example, the convolution kernel parameter kernel_size is 3, the stride is 1, and the number of rows in the calculation array is m=16. At this time, Figure 2As shown, if the order of segmentation is first c direction, then w direction, and finally h direction, that is, the first direction is c direction, the second direction is w direction, and the third direction is h direction, if w0=18, the image block is segmented based on the kernel_size and stride to obtain 16 sub-image blocks ifm_block of each image block, that is Figure 2 Among ifm_block_1, ifm_block_2, and ifm_block_m, the sizes of the sub-image blocks are all kernel_size*kernel_size=3*3.

[0138] Reference Figure 3 , Figure 3 Schematic diagram of a possible sub-image block, the size of the sub-image block is (h0, h0, c0). For example, when kernel_size=3, the sub-image block is (3, 3, c0), that is, each side of the sub-image block in the direction of the feature map includes 3*3=9 elements.

[0139] It should be noted that when segmenting the image blocks, the image blocks can be grouped according to the direction of the feature map to obtain multiple image block groups, each image block group includes multiple image blocks in the same feature map direction, and the image blocks are sorted according to the segmentation order to obtain a first order. For each image block group, the image blocks in the image block group are sorted according to the segmentation order in the feature map direction to obtain a second order, so as to accurately obtain the sub-image block that needs to be calculated when performing convolution calculations.

[0140] By dividing the feature map data according to the feature map direction based on the number of rows of the calculation array, the calculation unit parameters, the convolution kernel parameters and the step size of the convolution calculation in the parameters, a plurality of image blocks are obtained, and then each of the image blocks is divided based on the convolution kernel parameters and the step size to obtain a plurality of sub-image blocks corresponding to each image block. By dividing the feature map data according to the parameters of the calculation array and the feature map direction, the image blocks in the same feature map direction can be calculated according to the feature map direction during the convolution calculation, so that the convolution calculation results of each image block in the same feature map direction can be accumulated in the calculation unit, so that the intermediate results of the convolution calculation do not need to be transmitted outside the calculation array for storage, and thus there is no need to consume additional intermediate result storage resources and chip resources for processing the stored intermediate results, which can effectively reduce the chip area.

[0141] Furthermore, by dividing the feature map data according to the number of rows of the computing array, the convolution calculation process can fully utilize every computing unit in the computing array, thereby improving the area utilization rate of the computing array in the electronic device.

[0142] In a feasible implementation, step S20 may include steps S21 to S23:

[0143] Step S21, grouping the weight data based on the number of columns of the calculation array in the parameter and the number of the weight data to obtain a plurality of weight data groups;

[0144] Step S22, dividing the weight data in the weight data group based on the calculation unit parameters in the parameters to obtain the sub-weight blocks.

[0145] After obtaining the weight data, the electronic device groups the weight data based on the number of columns of the calculation array in the parameter and the number of weight data to obtain multiple groups of weight data. The sizes of each weight data weight in the three directions of w, h, and c are respectively (kernel_size, kernel_size, ic). The sizes of weight in the two directions of w and h are the same. When the feature map data is segmented in the order of first the c direction, then the w direction, and finally the h direction, as shown in FIG. Figure 4 As shown, the size of the weight data is (h0, h0, ic).

[0146] Since the size of the calculation array is m*n, that is, m rows and n columns, when performing convolution calculations, the calculation array can process n columns of data in each clock cycle. Therefore, based on the number of columns in the calculation array and the number of weight data in the parameters, the weight data is grouped to obtain multiple groups of weight data, such as Figure 4 As shown, the number of weight data is oc, oc is the number of feature maps in the target feature map, and the weight data group includes That is, oc / n is rounded up to get the number of weight data groups. For the last group of weight data, if the number of its weight data is less than n, 0 can be filled in to supplement the number of weight data to n.

[0147] For each weight data in any group of weight data, the weight data in the weight data group is segmented based on the calculation unit parameters in the parameters to obtain sub-weight blocks, that is, the size of the sub-weight blocks is (h0, h0, c0). When the weight data is segmented, it is still segmented according to the feature map direction, so that the number of sub-weight blocks in each weight data is the same as the number of sub-image blocks in the corresponding feature map direction, so as to facilitate convolution calculation. Figure 4 As shown, Figure 4 In the example, the first weight data weight_1 can be divided into weight_block_1, weight_block_2, and so on.

[0148] It should be noted that when grouping weight data, the weight data groups can be sorted to obtain a third order, and each sub-weight block of weight data can be sorted according to the segmentation order in the feature map direction to obtain a fourth order, so as to obtain the efficiency of the sub-weight block in subsequent convolution calculations.

[0149] Since the structure of the sub-weight block is exactly the same as that of the sub-data block, Figure 3 It can also be a possible schematic diagram of a sub-weight block, where the size of the sub-weight block is (h0, h0, c0). For example, when kernel_size = 3, the sub-weight block is (3, 3, c0), that is, each side of the sub-weight block in the direction of the feature map includes 3*3=9 elements.

[0150] The weight data is grouped based on the number of columns of the calculation array in the parameters and the number of the weight data to obtain a plurality of weight data groups; then the weight data in the weight data group is divided based on the calculation unit parameters in the parameters to obtain the sub-weight blocks. By dividing the weight data according to the calculation unit parameters, it is convenient to perform convolution calculations with image blocks, so that during the convolution calculation, the image blocks and sub-weight blocks in the same feature map direction can be calculated according to the feature map direction, so that the convolution calculation results corresponding to the same feature map direction can be accumulated in the calculation unit, so that the intermediate results of the convolution calculation do not need to be transmitted out of the calculation array for storage, and thus there is no need to consume additional intermediate result storage resources and chip resources for processing the stored intermediate results, which can effectively reduce the chip area.

[0151] Furthermore, by dividing the weight data according to the number of columns of the computing array, the convolution calculation process can fully utilize every computing unit in the computing array, thereby improving the area utilization rate of the computing array in the electronic device.

[0152] The present application also provides a convolution computing device, please refer to Figure 8 , the convolution computing device comprises:

[0153] A first segmentation module 10 is used to segment the feature map data based on the parameters of the calculation array and the feature map direction in the feature map data to obtain a plurality of image blocks and a plurality of sub-image blocks corresponding to each image block;

[0154] A second segmentation module 20 is used to segment the weight data in the plurality of weight data groups based on the feature map direction to obtain sub-weight blocks of each weight data in the weight data group;

[0155] An acquisition module 30, configured to acquire a corresponding first target element in each sub-image block along the direction of the feature map based on a clock cycle, and acquire a corresponding second target element in each sub-weight block along the direction of the feature map;

[0156] A broadcast module 40 is used to broadcast the first target element to all computing units in the corresponding row in the corresponding computing array, and broadcast the second target element to all computing units in the corresponding column in the computing array, wherein the computing unit performs convolution calculation and accumulates the calculation result with the convolution calculation result of the previous cycle to obtain the current convolution calculation result;

[0157] The output module 50 is used to output a target feature map based on the current convolution calculation result.

[0158] The convolution computing device provided in the embodiment of the present application adopts the convolution computing method in the above embodiment, which can solve the technical problem of how to effectively reduce the chip area. Compared with the prior art, the beneficial effects of the convolution computing device provided in the embodiment of the present application are the same as the beneficial effects of the convolution computing method provided in the above embodiment, and the other technical features in the convolution computing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0159] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the convolution calculation method in the above-mentioned embodiment one.

[0160] Reference below Fig. 9 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0161] like Fig. 9As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0162] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0163] The electronic device provided by the present application adopts the convolution calculation method in the above embodiment, which can solve the technical problem of effectively reducing the chip area of ​​the convolution calculation. Compared with the prior art, the beneficial effects of the electronic device provided by the present application are the same as the beneficial effects of the convolution calculation method provided by the above embodiment, and the other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0164] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0165] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0166] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the convolution calculation method in the above-mentioned embodiment.

[0167] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0168] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0169] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device: based on the parameters of the calculation array and the feature map direction in the feature map data, divides the feature map data to obtain multiple image blocks and multiple sub-image blocks corresponding to each image block; divides the weight data in multiple groups of weight data groups based on the feature map direction to obtain sub-weight blocks of each weight data in the weight data group; based on the clock cycle, obtains the corresponding first target element in each sub-image block along the feature map direction, and obtains the corresponding second target element in each sub-weight block along the feature map direction; broadcasts the first target element to all calculation units of the corresponding row in the corresponding calculation array, and broadcasts the second target element to all calculation units of the corresponding column in the calculation array, wherein the calculation unit performs convolution calculation and accumulates the calculation result with the convolution calculation result of the previous cycle to obtain the current convolution calculation result; outputs the target feature map based on the current convolution calculation result.

[0170] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0171] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0172] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0173] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned convolution calculation method, and can solve the technical problem of effectively reducing the chip area of ​​the convolution calculation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the convolution calculation method provided in the above-mentioned embodiment, and will not be repeated here.

[0174] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the above-mentioned convolution calculation method when executed by a processor.

[0175] The computer program product provided in this application can solve the technical problem of how to effectively reduce the chip area of ​​convolution calculation. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiment of this application are the same as the beneficial effects of the convolution calculation method provided in the above embodiment, which will not be repeated here.

[0176] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A convolution calculation method, characterized in that: include: Based on the parameters of the calculation array and the feature map direction in the feature map data, the feature map data is segmented to obtain a plurality of image blocks and a plurality of sub-image blocks corresponding to each image block; Based on the direction of the feature map, the weight data in the plurality of weight data groups are divided to obtain sub-weight blocks of each weight data in the weight data group; Based on the clock cycle, a corresponding first target element is obtained in each sub-image block along the direction of the feature map, and a corresponding second target element is obtained in each sub-weight block along the direction of the feature map; Broadcasting the first target element to all computing units in the corresponding row in the corresponding computing array, and broadcasting the second target element to all computing units in the corresponding column in the computing array, wherein the computing unit performs convolution calculation and accumulates the calculation result with the convolution calculation result of the previous cycle to obtain the current convolution calculation result; Based on the current convolution calculation result, a target feature map is output.

2. The convolution calculation method according to claim 1, characterized in that: The step of acquiring the corresponding first target element in each sub-image block along the feature map direction based on the clock cycle, and acquiring the corresponding second target element in each sub-weight block along the feature map direction comprises: Acquire a target weight data group from the weight data group at intervals of a first preset clock cycle; Determine the target image blocks in sequence according to the segmentation order of the image blocks at intervals of a second preset clock cycle, and obtain the target sub-weight blocks corresponding to the target image blocks in the target weight data group; In each clock cycle, a corresponding first target element is obtained in each target sub-image block of the target image block along the direction of the feature map, and a second target element is obtained in the target sub-weight block along the direction of the feature map.

3. The convolution calculation method according to claim 2, characterized in that: The first preset clock cycle is obtained according to the number of the image blocks and the convolution kernel parameters in the parameters, the second preset clock cycle is obtained according to the convolution kernel parameters, and the first preset clock cycle is greater than the second preset clock cycle.

4. The convolution calculation method according to claim 2, characterized in that: The step of acquiring the corresponding first target element in each target sub-image block of the target image block along the direction of the feature map, and acquiring the second target element in the target sub-weight block along the direction of the feature map comprises: Obtain the corresponding first target element in the target sub-image block in sequence along the feature map direction, wherein the element order is the order of elements in the first feature map belonging to the same sub-image block, and the element order corresponding to the sub-image block is the same as the element order corresponding to the sub-weight block; Obtain the corresponding second target element in the target sub-weight block in the element order along the feature map direction.

5. The convolution calculation method according to claim 1, wherein: The step of outputting a target feature map based on the current convolution calculation result comprises: If the accumulated number of cycles corresponding to the current clock cycle reaches a preset threshold, the sub-image block corresponding to the target feature map is output based on the current convolution calculation result.

6. The convolution calculation method according to claim 1, wherein: The computing unit includes a multiplication and accumulation circuit, an adder and an output register. The multiplication and accumulation circuit is used to perform convolution calculation to obtain a calculation result, and transmit the calculation result to the adder; A feedback channel is set between the adder and the output register, and the adder is used to obtain the convolution calculation result of the previous cycle through the feedback channel, and accumulate the calculation result with the convolution calculation result of the previous cycle to obtain the current convolution calculation result; The output register is used to store the current convolution calculation result of the adder.

7. The convolution calculation method according to claim 6, characterized in that: If the accumulated number of cycles corresponding to the current clock cycle reaches a preset threshold, the current convolution calculation result is output through the output register of each computing unit, the sub-image block corresponding to the target feature map is obtained, and the current convolution calculation result stored in each of the computing units is cleared.

8. The convolution calculation method according to claim 1, characterized in that: The step of dividing the feature map data based on the parameters of the calculation array and the feature map direction in the feature map data to obtain a plurality of image blocks and a plurality of sub-image blocks corresponding to each image block comprises: Based on the number of rows of the calculation array, the calculation unit parameters, the convolution kernel parameters, and the step size of the convolution calculation in the parameters, the feature map data is segmented according to the feature map direction to obtain multiple image blocks; Based on the convolution kernel parameters and the step size, each of the image blocks is segmented to obtain a plurality of sub-image blocks corresponding to each image block.

9. The convolution calculation method according to claim 8, characterized in that: The step of dividing the feature map data according to the feature map direction based on the number of rows of the calculation array, the calculation unit parameters, the convolution kernel parameters and the step size of the convolution calculation in the parameters to obtain multiple image blocks includes: Determine the number of first elements of the characteristic map direction corresponding to the first direction based on the calculation unit parameter; Determine the number of second elements corresponding to the second direction based on the number of rows and the step size, and determine the number of third elements corresponding to the third direction based on the convolution kernel parameters; Based on the first element quantity, the second element quantity and the third element quantity, the feature map data is segmented according to the feature map direction to obtain multiple image blocks.

10. The convolution calculation method according to any one of claims 1 to 9, characterized in that: The step of dividing the weight data in the plurality of weight data groups based on the feature map direction to obtain sub-weight blocks of each weight data in the weight data group comprises: Based on the number of columns of the calculation array in the parameter and the amount of the weight data, the weight data are grouped to obtain a plurality of weight data groups; The weight data in the weight data group is segmented based on the calculation unit parameters in the parameters to obtain the sub-weight blocks.

11. A convolution computing device, characterized in that: The convolution computing device comprises: A first segmentation module is used to segment the feature map data based on the parameters of the calculation array and the feature map direction in the feature map data to obtain multiple image blocks and multiple sub-image blocks corresponding to each image block; A second segmentation module is used to segment the weight data in the plurality of weight data groups based on the feature map direction to obtain sub-weight blocks of each weight data in the weight data group; An acquisition module, configured to acquire the corresponding first target element in each sub-image block along the direction of the feature map based on a clock cycle, and acquire the corresponding second target element in each sub-weight block along the direction of the feature map; A broadcasting module, used to broadcast the first target element to all computing units of the corresponding row in the corresponding computing array, and broadcast the second target element to all computing units of the corresponding column in the computing array, wherein the computing unit performs convolution calculation and accumulates the calculation result with the convolution calculation result of the previous cycle to obtain the current convolution calculation result; The output module is used to output a target feature map based on the current convolution calculation result.

12. An electronic device, characterized in that: The electronic device comprises: a memory, a processor, and a convolution calculation program stored in the memory and executable on the processor, wherein the convolution calculation program implements the steps of the convolution calculation method as described in any one of claims 1 to 10 when executed by the processor.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a convolution calculation program, and when the convolution calculation program is executed by a processor, the steps of the convolution calculation method as described in any one of claims 1 to 10 are implemented.

Images