A full connection processing system, method and terminal device based on convolution hardware

Abstract

The application provides a full connection processing system and method based on convolution hardware and a terminal device. The technical problem of processing full connection calculation by using convolution calculation hardware is solved. The convolution hardware comprises a feature data input interface and a weight parameter input interface. The system is configured to: if the size of the feature data is less than or equal to a size threshold, directly cross input, that is, input the feature data at the weight parameter input interface; and input the weight parameter at the feature data input interface; if the size of the feature data is greater than the size threshold, split the feature data and then cross input; wherein the weight parameter is data obtained by splicing original weight parameters according to the output channel of the convolution, the product of the height and width of the weight parameter is the product of the output channel number, height and width of the original weight parameter, and the input channel number of the weight parameter is equal to the input channel number of the original weight parameter. The adaptability of the convolution hardware when processing full connection is improved, and the processing performance is further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of convolutional neural network, and particularly relates to a full connection processing system and method based on convolutional hardware and a terminal device. BACKGROUND

[0002] A mainstream artificial intelligence solution is implemented by using a convolutional neural network (CNN). The CNN mainly comprises a convolution layer, a pooling layer, an activation function and a full connection layer, etc. More than 90% of the operation amount of the CNN is concentrated in the convolution layer and the full connection layer, so the CNN model can be briefly summarized as two parts of convolution operation and full connection operation. Compared with the convolution operation, the full connection operation has a very large amount of data of feature data and weight parameters, and usually has a large amount of operation. In addition, the full connection layer has a high requirement for data precision and a high requirement for external storage speed, so the optimization of the full connection layer operation needs to be solved first to realize the CNN acceleration.

[0003] At present, the optimization of the full connection layer operation is mainly implemented by using parameter compression and parallel operation. The optimization of the parallel operation mainly includes: using GPU multi-thread calculation to realize acceleration, using FPGA and other programmable logic circuits to realize acceleration by expanding hardware resources for speed, and using a deep learning dedicated processor or an ASIC to realize acceleration.

[0004] However, the above acceleration methods can realize CNN acceleration to a certain extent, but the acceleration effect is realized by directly increasing hardware, so that the terminal applying the convolutional neural network increases the hardware burden. SUMMARY

[0005] The present application provides a full connection processing system and method based on convolutional hardware and a terminal device, which solves the technical problem of processing full connection calculation by using convolutional computing hardware.

[0006] In a first aspect, the present application provides a full connection processing system based on convolutional hardware, wherein the convolutional hardware comprises a feature data input interface and a weight parameter input interface, and the system is configured to: input feature data through the weight parameter input interface; and input weight parameters through the feature data input interface.

[0007] In this way, by inputting the feature data at the weight parameter input interface and inputting the weight parameter at the feature data input interface, cross input is realized, the convolution hardware performs full connection calculation, the effective calculation rate of full connection is improved without increasing hardware, and the system power consumption and burden are reduced. In addition, in the case that there is a difference between convolution calculation and full connection calculation, cross calculation can overcome the problem of low calculation efficiency when convolution hardware is directly reused for full connection calculation. The full connection processing is realized by using the convolution hardware, the full connection calculation output value is obtained, and the utilization rate of the convolution hardware is improved.

[0008] In some embodiments, the weight parameter is data obtained by splicing the original weight parameter according to the output channel of convolution, the product of the height and width of the weight parameter is the product of the output channel number, the height and the width of the original weight parameter, and the input channel number of the weight parameter is equal to the input channel number of the original weight parameter.

[0009] In this way, by splicing the weight parameter, the problem that the weight parameter is not suitable for convolution parallel computing architecture when directly performing cross input can be solved.

[0010] In some embodiments, the feature data includes first feature data; before inputting the feature data at the weight parameter input interface, the system is further configured to: if the size of the first feature data is greater than a size threshold, split the first feature data into a plurality of feature maps that meet a preset condition; the plurality of feature maps include all data of the first feature data.

[0011] In this way, the feature data exceeding the size is split according to specific needs, and then the above full connection processing is performed, and some restrictions of using convolution hardware to realize full connection processing are eliminated, the adaptability of reusing convolution hardware to process full connection is further improved, and the performance of reusing convolution hardware to process full connection is improved.

[0012] In some embodiments, the size of the feature map is NIX new ×NIY new ; NIX new = NKX max ; NIY new = NKY max ; wherein NKX max and NKY max are size thresholds; and the number of the feature maps NIF new is: wherein NIX×NIY is the size of the first feature data, NIF is the input channel number of the first feature data, and CEIL is a rounding up function. The rounding up function ensures that the plurality of feature maps include all data of the first feature data.

[0013] In some embodiments, the original weight parameters are spliced according to the output channels of the convolution; before that, the system is further configured to: according to the size threshold, split the weight parameters.

[0014] In a second aspect, the present application also provides a full connection processing method based on convolution hardware, comprising inputting feature data at a weight parameter input interface of the convolution hardware, and inputting weight parameters at a feature data input interface of the convolution hardware, so as to realize full connection calculation.

[0015] In some embodiments, the weight parameters are data obtained by splicing original weight parameters according to the output channels of the convolution, the number of input channels of the weight parameters is equal to the number of input channels of the original weight parameters, and the product of the height and width of the weight parameters is the product of the number of output channels, the height and the width of the original weight parameters.

[0016] In a third aspect, the present application also provides a terminal device comprising the full connection processing system based on convolution hardware. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiments will be briefly introduced as follows: obviously, other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0018] Figure 1 A framework diagram of an embodiment of a full connection processing system based on convolution hardware provided by the present application embodiment;

[0019] Figure 2 A framework diagram of an embodiment of a full connection processing system based on convolution hardware provided by the present application embodiment;

[0020] Figure 3 A process diagram of parallel calculation of convolution using convolution hardware;

[0021] Figure 4 A process diagram of an embodiment of full connection calculation using convolution hardware;

[0022] Figure 5 A framework diagram of an embodiment of a full connection processing system based on convolution hardware provided by the present application embodiment;

[0023] Figure 6 A process diagram of an embodiment of splicing original weight parameters provided by the present application embodiment;

[0024] Figure 7 A process diagram of an implementation of splicing original weight parameters provided by the present application embodiment;

[0025] Figure 8a A schematic diagram illustrating one implementation method for splicing original weight village parameters provided in an embodiment of this application;

[0026] Figure 8b A schematic diagram illustrating one implementation method for splitting the original weight parameters provided in an embodiment of this application;

[0027] Figure 9 This is a schematic diagram illustrating one embodiment of feature data splitting provided in this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be described completely and clearly below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0029] Figure 1 This is a schematic diagram illustrating the framework of one embodiment of a fully connected processing system based on convolutional hardware, provided in this application. Figure 1 As shown, this application implements fully connected computation using convolutional hardware, enabling terminal devices to achieve hardware simplification while still supporting convolutional computation functionality. For example, this application inputs the feature data and weight parameters for fully connected computation into the convolutional hardware to obtain the output result of the fully connected computation. It should be noted that in some scenarios, the weight parameters can be pre-processed offline. Offline processing of weight parameters includes pre-training a model to obtain weight parameters and storing these weight parameters in the terminal's processing and computation module. When the terminal inputs feature data, the aforementioned weight parameters are invoked to perform convolutional and fully connected computations. For example, in an access control system, weight parameters are obtained through pre-training a model and stored in the computation module of the access control system's convolutional hardware. When a human facial feature is input, the weight parameters are invoked to perform convolutional and fully connected computations, and the computation result is used to determine whether the input human facial can pass through the access control system.

[0030] However, due to the differences between convolutional and fully connected computations, if the feature data for fully connected computations is directly input into the feature data input interface for convolutional computations, and the original weight parameters for fully connected computations are directly input into the weight parameter input interface for convolutional computations, the parallel computing architecture of convolutional hardware determines that its computational efficiency is very low when performing fully connected computations.

[0031] To solve the problem that the architecture of the convolution hardware determines low computing efficiency, the application provides an input cross method for a full connection processing system based on convolution hardware. Figure 2 A framework schematic diagram of an embodiment of a full connection processing system based on convolution hardware is provided in the embodiments of the application. As shown in Figure 2 In some embodiments, the feature data is input at the weight parameter input interface of the convolution hardware, and the weight parameter is input at the feature data input interface of the convolution hardware.

[0032] However, direct cross input may cause the problem that the weight parameter is not suitable for the convolution parallel computing architecture, i.e., the convolution computing unit may not be able to perform efficient parallel computing in the convolution hardware due to the too small size of the weight parameter input.

[0033] The foregoing problem will be described below by taking a MAC unit as an example of a convolution computing unit and in combination with the accompanying drawings. It should be noted that the following description is based on an example of an implementation method of the convolution hardware (sliding window implementation of convolution computing) and is applicable to the implementation of GEMM (general matrix multiplication) convolution computing. The convolution computing unit implemented by the sliding window is referred to as a MAC (multiply-accumulate unit).

[0034] Figure 3 A process schematic diagram of parallel convolution computing using convolution hardware (the convolution hardware is composed of a plurality of MAC units) is shown in Figure 3 As shown in the figure, taking 64 (i.e., 8x8) MAC units, 16 feature data input channels for convolution computing, a 1x1 convolution kernel, and a step of 1 as an example. Each MAC unit performs 16 (i.e., 1x1x16) multiply-accumulate operations per round of parallel computing, and outputs one convolution computing result. The convolution kernel is slid horizontally 8 times and vertically 8 times, and the obtained feature data is input into the MAC unit. Each round of parallel computing outputs 64 convolution computing results. As can be seen, when the parallel computing size is 8x8 feature data, 64 MAC units are all performing effective computation. The input characteristics of the convolution computing are that the size of the feature data is large, and the size of the convolution kernel is small. Therefore, the minimum parallel computing unit (e.g., 8x8 MAC unit) of the convolution hardware is used to perform parallel computing on a certain amount of feature data, and the same output channel weight parameter is reused to complete the convolution computing.

[0035] Figure 4 A process schematic diagram of an embodiment of full connection computing using convolution hardware is shown in Figure 4As shown, the feature data to be subjected to full connection calculation is directly input into the feature data input interface of convolution calculation, and the original weight parameter to be subjected to full connection calculation is directly input into the weight parameter input interface of convolution calculation, taking the full connection calculation with the input feature data of 128x1x1 (i.e., input channel numberxheightxwidth, represented as ChannelxHighxWidth), the convolution kernel of 1x1, and the output channel number of 512 as an example, each MAC unit needs to perform 128x1x1=128 times of multiplication and accumulation to output one calculation result, and the 8x8 MAC unit needs to perform 512 rounds to complete the full connection calculation, only one MAC in the 64 MAC units is performing effective calculation, and the utilization rate is extremely low.

[0036] To solve the problem of low calculation efficiency caused by low utilization rate of the MAC unit, the application provides a method for splicing original weight parameters for a full connection processing system based on convolution hardware, which can be completed in an offline manner. Figure 5 An embodiment of a full connection processing system based on convolution hardware is provided in the application, and a framework schematic diagram of the embodiment is shown in FIG. 1. Figure 5 As shown, the feature data is input into the weight parameter input interface, and the weight parameter is input into the feature data input interface; wherein the product of the height and width of the weight parameter is the product of the output channel number, the height, and the width of the original weight parameter, and the input channel number of the weight parameter is equal to the input channel number of the original weight parameter.

[0037] Figure 6 An embodiment of a process for splicing original weight parameters is provided in the application, and a process schematic diagram of the embodiment is shown in FIG. 3. Figure 6 As shown, the size of the original weight parameter is 1x1, the input channel number of the original weight parameter is 128, and the input channel number of the spliced weight parameter is also 128; the output channel number of the original weight parameter is 512, and therefore the product of the height and width of the spliced weight parameter is 512x1x1=512.

[0038] Figure 7The process schematic diagram of one implementation manner of splicing the original weight parameters provided by the embodiment of the application is shown in FIG. 1. The feature data size of the convolution (pooling) output is 4x4, the weight parameter size is 4x4, the input channel number is 8, the output channel number is 32, and the maximum convolution kernel size supported by the convolution hardware is 7x7. For convenience of description, the weight parameters are numbered as XX-Y, wherein XX refers to the output channel number, and Y refers to the input channel number. The original weight parameters include 01-1, 01-2, 01-3, 01-4, 01-5, 01-6, 01-7, 01-8, 02-1, 02-2, 02-3, 02-4, 02-5, 02-6, 02-7, 02-8, 03-1, 03-2, 03-3, 03-4, 03-5, 03-6, 03-7, 03-8, 04-1, 04-2, 04-3, 04-4, 04-5, 04-6, 04-7, 04-8, ……, 30-1, 30-2, 30-3, 30-4, 30-5, 30-6, 30-7, 30-8, 31-1, 31-2, 31-3, 31-4, 31-5, 31-6, 31-7, 31-8, 32-1, 32-2, 32-3, 32-4, 32-5, 32-6, 32-7, 32-8. Splicing the original weight data means splicing 01-1, 02-1, 03-1, 04-1, ……, 30-1, 31-1, 32-1 into one, splicing 01-2, 02-2, 03-2, 04-2, ……, 30-2, 31-2, 32-2 into one, splicing 01-3, 02-3, 03-3, 04-3, ……, 30-3, 31-3, 32-3 into one, ……, and splicing 01-8, 02-8, 03-8, 04-8, ……, 30-8, 31-8, 32-8 into one, totally 8.

[0039] In the above implementation manner, the weight parameters do not need to be split. When the weight parameter size is greater than the maximum convolution kernel size supported by the convolution hardware, the weight parameters are first split and then spliced.

[0040] The splitting and splicing of the weight parameters will be described below with reference to the accompanying drawings Figure 8a and Figure 8b

[0041] Figure 8a The process schematic diagram of one implementation manner of splicing the original weight parameters provided by the embodiment of the application is shown in FIG. 1. As Figure 8a ​As shown, the feature data size of the convolution (pooling) output is 9x9, the weight parameter size is 9x9, the input channel number is 8, the output channel number is 32, full connection processing is performed, and the maximum convolution kernel size supported by the convolution hardware is 7x7. For the convenience of description, the weight parameters are numbered as XX-Y, where XX refers to the output channel number and Y refers to the input channel number. Then the original weight parameters include 01-1, 01-2, 01-3, 01-4, 01-5, 01-6, 01-7, 01-8, 02-1, 02-2, 02-3, 02-4, 02-5, 02-6, 02-7, 02-8, 03-1, 03-2, 03-3, 03-4, 03-5, 03-6, 03-7, 03-8, 04-1, 04-2, 04-3, 04-4, 04-5, 04-6, 04-7, 04-8, …, 30-1, 30-2, 30-3, 30-4, 30-5, 30-6, 30-7, 30-8, 31-1, 31-2, 31-3, 31-4, 31-5, 31-6, 31-7, 31-8, 32-1, 32-2, 32-3, 32-4, 32-5, 32-6, 32-7, 32-8.

[0042] In some embodiments, the original weight parameters are split (the same splitting method is used for the feature data and the weight parameters), and the size of the split weight parameters is NIX new ×NIY new , where NIX new = NKX max , NIY new = NKY max , NKX max and NKY max are size thresholds, i.e., the maximum convolution kernel size supported by the convolution hardware, which is 7x7 here, and the input channel number NIF new of the split weight parameters is:

[0043]

[0044] where NIX×NIY is the size of the original weight parameters, NIF is the input channel number of the original weight parameters, and CEIL is the ceiling function. The splitting formula is: where CEIL is the ceiling. That is, (9x9x8) / (7x7) = 13.22, CEIL(13.22) = 14, and the redundant bit invalid data caused by the ceiling is that the input channel number of the split weight parameters is 14.

[0045] The weighted parameters after splitting include 01-1', 01-2', 01-3', 01-4', 01-5', 01-6', 01-7', 01-8', ..., 01-11, 01-12, 01-13, 01-14, ..., 32-1', 32-2', 32-3', 32-4', 32-5', 32-6', 32-7', 32-8', ..., 32-11, 32-12, 32-13, 32-14.

[0046] To illustrate the weight parameter decomposition process in detail, let's take the example of decomposing "01-1, 01-2, 01-3, 01-4, 01-5, 01-6, 01-7, 01-8" into "01-1', 01-2', 01-3', 01-4', 01-5', 01-6', 01-7', 01-8', ..., 01-11, 01-12, 01-13, 01-14". Figure 8b As shown, the splitting of 01-1 and 01-2 involves the following: 01-1 has a size of 9×9 and a total of 81 data points. When splitting 01-1, first arbitrarily select 49 data points as 01-1'. When arbitrarily selecting the next 49 data points, you can select them row by row, column by column, or as follows: Figure 8b The selection method, which involves 7 rows and 7 columns, can also be random, continuous, or random, and this application does not impose any limitations on it. At this point, 32 data points remain unselected in 01-1, and these 32 unselected data points form 01-2'. 01-2' is also 7×7, therefore 17 data points need to be selected from 01-2 to supplement 01-2'. Then, 49 data points are selected from 01-2 to form 01-3'. At this point, 15 data points remain unselected in 01-2, and these data points, along with the 34 data points from 01-3, will form 01-4'. The process of selecting 34 data points from 01-3 to complete 01-4' and splitting "01-3, 01-4, 01-5, 01-6, 01-7, 01-8" into "01-5', 01-6', 01-7', 01-8', ..., 01-11, 01-12, 01-13" follows the same logic and will not be elaborated here. It should be noted that because of the rounding up (i.e., 13.22 rounded up to 14), 38 positions in 01-14 are invalid and can be placed using "0". This is understandable. Figure 8b The 01-4' shown only displays "15 data points split from 01-2", and does not show the data that was filled in by selecting 34 data points from 01-3. In fact, 01-4' is also 7×7.

[0047] The splicing original weight parameters refers to splicing 01-1', 02-1', 03-1', 04-1',..., 30-1', 31-1', 32-1' into one, splicing 01-2', 02-2', 03-2', 04-2',..., 30-2', 31-2', 32-2' into one, splicing 01-3', 02-3', 03-3', 04-3',..., 30-3', 31-3', 32-3' into one,..., splicing 01-14, 02-14, 03-14, 04-14,..., 30-14, 31-14, 32-14 into one, and a total of 14.

[0048] It should be noted that, in order to show the splitting process of the weight parameters, the data in 01-1 and 01-2 are distinguished as shown in the table, which is only to show the data correspondence before and after the splitting of 01-1, 01-2 into 01-1', 01-2', 01-3', 01-4', and does not constitute a limitation of the scheme itself. Figure 8b

[0049] In this way, after splicing, the cross input into the MAC unit can fully utilize the computing power of the parallel MAC unit.

[0050] In some scenarios, the size of the input feature data can be greater than the maximum convolution kernel size supported by the convolution hardware, or can be within the range of the convolution kernel size supported by the convolution hardware. For this case, it needs to be processed respectively.

[0051] Therefore, in some embodiments, as shown in the table, Figure 5 If the size of the feature data is less than or equal to the size threshold, the feature data is input at the weight parameter input interface, and the weight parameter is input at the feature data input interface; wherein the weight parameter is data obtained by splicing the original weight parameter according to the output channel of the convolution, the product of the height and width of the weight parameter is the product of the output channel number, height and width of the original weight parameter, and the input channel number of the weight parameter is equal to the input channel number of the original weight parameter. If the size of the feature data is greater than the size threshold, the feature data and the weight parameter also need to be split.

[0052] In some other embodiments, as shown in the table, Figure 5 ​As shown, if the size of the feature data is greater than a size threshold, the input feature data is split, and the weight parameters are also split. Then, the feature data is input into the weight parameter input interface; and the weight parameters are input into the feature data input interface. For example, taking a 3-channel input as an example, the fully connected input feature data is 9×9, and the maximum convolution kernel supported by the convolution hardware is 7×7. The technical solution of this application embodiment is to first split the feature data into 7×7 segments. As the feature data is split, the number of input channels will increase accordingly. At the same time, the weight parameters need to be processed offline to adapt to the splitting of the feature data.

[0053] In some embodiments, the feature map is sized NIX new ×NIY new For: NIX new =NKX max NIY new =NKY max Among them, NKX max and NKY max The size threshold; the number of feature maps NIF new for:

[0054] Where NIX×NIY is the size of the first feature data, NIF is the number of input channels of the first feature data, and CEIL is the floor function. Floor function ensures that multiple feature maps include all data from the first feature data.

[0055] Figure 9 This is a schematic diagram illustrating one embodiment of feature data splitting provided in this application. For example... Figure 9 As shown, taking an example with 3 input channels, 9×9 input feature data, and a maximum convolution kernel size of 7×7 supported by the convolution hardware, the 3 input channels are split into 5 input channels. The feature data of each input channel is 7×7, and the last input channel contains 2 invalid data.

[0056] For the specific splitting process, please refer to the aforementioned splitting process for weight parameters. The splitting principles are the same, so they will not be repeated here.

[0057] The present application has been described in detail above with reference to specific embodiments and exemplary examples, enabling those skilled in the art to understand or implement the present application. However, these descriptions should not be construed as limiting the present application. Those skilled in the art will understand that various equivalent substitutions, modifications, or improvements can be made to the technical solutions and implementation methods of the present application without departing from the spirit and scope of the present application, and all such modifications and improvements fall within the scope of the present application. The scope of protection of the present application is determined by the appended claims.

Claims

1. A fully connected processing system based on convolution hardware, the convolution hardware comprising a feature data input interface and a weight parameter input interface, characterized in that, The system is configured to: input feature data at the weight parameter input interface; and input weight parameters at the feature data input interface to implement full connection calculation; The weight parameters are data obtained by concatenating original weight parameters according to the output channels of convolution, the number of input channels of the weight parameters is equal to the number of input channels of the original weight parameters, and the product of the height and the width of the weight parameters is the product of the number of output channels, the height and the width of the original weight parameters; The feature data includes first feature data; before the feature data is input at the weight parameter input interface, the system is further configured to: the size NIX of the feature map new ×NIY new is: NIX new = NKX max ; NIY new = NKY max ; wherein NKX max and NKY max are size thresholds; The number of feature maps NIF new is: if the size of the first feature data is greater than a size threshold, split the first feature data into a plurality of feature maps that meet a preset condition; the plurality of feature maps include all data of the first feature data; wherein NIX×NIY is the size of the first feature data, NIF is the number of input channels of the first feature data, and CEIL is a rounding up function; Before the original weight parameters are concatenated according to the output channels of convolution, the system is further configured to:

2. A method for fully connected processing based on convolution hardware, characterized in that, split the weight parameters according to the size threshold. The full connection processing method includes: input feature data at the weight parameter input interface of the convolution hardware, and simultaneously input weight parameters at the feature data input interface of the convolution hardware to implement full connection calculation; the size NIX of the feature map new NIY new is: NIX new = NKX max ; NIY new = NKY max ; wherein NKX max and NKY max are size thresholds; The number of feature maps NIF new is: The weight parameters are data obtained by concatenating original weight parameters according to the output channels of convolution, the number of input channels of the weight parameters is equal to the number of input channels of the original weight parameters, and the product of the height and the width of the weight parameters is the product of the number of output channels, the height and the width of the original weight parameters; wherein NIX×NIY is the size of the first feature data, NIF is the number of input channels of the first feature data, and CEIL is a rounding up function; Before the original weight parameters are concatenated according to the output channels of convolution, the system is further configured to:

3. A terminal device, characterized by comprising: split the weight parameters according to the size threshold. The full connection processing system based on convolution hardware includes the full connection processing system according to claim 1.

Images