Data processing method, convolutional neural network training method, device and FPGA

Abstract

The embodiment of the present application provides a data processing method, a convolutional neural network training method, a device and an FPGA, which can obtain a picture to be processed and store it in a preset storage space. The FPGA can obtain the picture to be processed from the preset storage space, and process the picture to be processed based on the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network through the convolutional layer operation kernel to obtain the final feature map, which is stored in the preset storage space; the elements in the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network correspond one-to-one to the convolution kernel in the convolutional layer; the convolution kernel in the convolutional layer includes: a compressed convolution kernel and an uncompressed convolution kernel, and the size of the compressed convolution kernel is smaller than that of the uncompressed convolution kernel. Furthermore, the final feature map can be obtained from the preset storage space. In this way, the efficiency of data processing based on convolutional neural networks can be improved.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a data processing method, a convolutional neural network training method, a device and an FPGA. Background Art

[0002] With the rapid development of Convolutional Neural Networks (CNN) in the field of computer vision, running Convolutional Neural Networks to process images based on the Central Processing Unit (CPU) is increasingly used in image processing tasks.

[0003] However, a convolutional neural network usually includes more convolutional layers, and each convolutional layer includes multiple convolutional kernels, which makes the number of network parameters of the convolutional neural network larger. Furthermore, if the convolutional kernel in the convolutional layer is larger, for example, the size of the convolutional kernel is 3×3 or 5×5, the number of network parameters of the convolutional neural network will be further increased.

[0004] The number of network parameters of the convolutional neural network is large, which will increase the amount of calculation. In addition, the data processing based on the CPU running the convolutional neural network is a serial processing method, that is, the convolution processing is performed based on each convolution kernel in turn, which will reduce the efficiency of data processing based on the convolutional neural network. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a data processing method, a convolutional neural network training method, a device and an FPGA to improve the efficiency of data processing based on a convolutional neural network. The specific technical solution is as follows:

[0006] In a first aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application discloses a data processing method, the method comprising:

[0007] Get the image to be processed;

[0008] The image to be processed is stored in a preset storage space, so that the FPGA (Field Programmable Gate Array) obtains the image to be processed from the preset storage space, and processes the image to be processed through the convolution layer operation kernel based on the target parameter matrix corresponding to each convolution layer in the target convolutional neural network to obtain a final feature map, and stores it in the preset storage space;

[0009] Among them, the elements in the target parameter matrix corresponding to each convolution layer in the target convolution neural network correspond to the convolution kernels in the convolution layer one by one; the convolution kernels in the convolution layer include: compressed convolution kernels and uncompressed convolution kernels, and the size of the compressed convolution kernels is smaller than that of the uncompressed convolution kernels; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolution layer, respectively; the target convolution neural network is: for each convolution layer in the initial convolution neural network, the specified part of the convolution kernels in the convolution layer is compressed to obtain the compressed convolution kernels, and the model is trained; the specified part of the convolution kernels is set based on the number of input channels and the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation kernel of the FPGA;

[0010] The final feature map is obtained from the preset storage space.

[0011] Optionally, the elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network are divided into multiple element groups; the number of rows of the target parameter matrix is ​​the number of output channels of the convolution layer, and the number of columns of the target parameter matrix is ​​the number of input channels of the convolution layer; each element group contains M×N elements; each element group corresponds to a group of compressed convolution kernels or to a group of uncompressed convolution kernels; the convolution kernels of at least two element groups in the multiple element groups are of the same size, and the convolution kernels of at least two element groups are of different sizes; M is the input parallelism of the convolution layer operation kernel, and N is the output parallelism of the convolution layer operation kernel, so that the FPGA performs convolution processing in parallel through the convolution layer operation kernel based on the target parameter matrix corresponding to the convolution layer.

[0012] Optionally, the process of acquiring the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network includes:

[0013] For each convolutional layer in the initial convolutional neural network, based on the number of input channels and the number of output channels of the convolutional layer, and the input parallelism and output parallelism of the convolutional layer operation kernel, determine the compression step size for compressing the convolutional layer; wherein the compression step size is used to indicate: when compressing the convolutional layer, the proportion of the convolutional kernels that need to be compressed in the convolutional layer;

[0014] According to the determined compression step size, a portion of the convolution kernels specified in the convolution layer are compressed into target convolution kernels to obtain an alternative convolution neural network;

[0015] Training the candidate convolutional neural network to obtain a target convolutional neural network;

[0016] Generate a target parameter matrix corresponding to each convolutional layer in the target convolutional neural network.

[0017] Optionally, for each convolutional layer in the initial convolutional neural network, determining a compression step size for compressing the convolutional layer based on the number of input channels and the number of output channels of the convolutional layer, and the input parallelism and output parallelism of the convolutional layer operation kernel includes:

[0018] For each convolutional layer in the initial convolutional neural network, calculating the ratio of the number of input channels of the convolutional layer to the input parallelism of the convolutional layer operation core as a first ratio, and calculating the ratio of the number of output channels of the convolutional layer to the output parallelism of the convolutional layer operation core as a second ratio;

[0019] Based on the first ratio and the second ratio, determine a first compression step and a second compression step when compressing the convolution layer; wherein the first compression step is used to indicate: when compressing the convolution layer, the proportion of the convolution kernels corresponding to each output channel that need to be compressed; the second compression step is used to indicate: when compressing the convolution layer, the proportion of the convolution kernels corresponding to each input channel that need to be compressed.

[0020] Optionally, determining a first compression step size and a second compression step size when compressing the convolution layer based on the first ratio and the second ratio includes:

[0021] If the input parallelism of the convolutional layer operation kernel is equal to the output parallelism, determining a first compression step size and a second compression step size when compressing the convolutional layer based on the smaller value of the first ratio and the second ratio;

[0022] If the input parallelism of the convolution layer operation kernel is not equal to the output parallelism, a first compression step size when compressing the convolution layer is determined based on the first ratio, and a second compression step size when compressing the convolution layer is determined based on the second ratio.

[0023] Optionally, compressing a specified portion of the convolution kernels in the convolution layer into a target convolution kernel according to the determined compression step size to obtain an alternative convolution neural network includes:

[0024] For each convolution kernel corresponding to each output channel in the convolution layer, the convolution kernels are evenly divided into a plurality of first convolution kernel groups according to the order of the corresponding input channels; wherein the number of convolution kernels contained in each first convolution kernel group is the first compression step size; for each first convolution kernel group, the convolution kernels other than the convolution kernel at the specified position in the first convolution kernel group are determined as the first convolution kernels to be processed; the first convolution kernels to be processed in each first convolution kernel group are compressed into the target convolution kernel to obtain the candidate convolution neural network; or,

[0025] For each convolution kernel corresponding to each input channel in the convolution layer, they are evenly divided into multiple second convolution kernel groups according to the order of the corresponding output channels; wherein the number of convolution kernels contained in each second convolution kernel group is the second compression step size; for each second convolution kernel group, the other convolution kernels in the second convolution kernel group except the convolution kernel at the specified position are determined as the second convolution kernels to be processed; the second convolution kernels to be processed in each second convolution kernel group are compressed into the target convolution kernel to obtain an alternative convolution neural network.

[0026] Optionally, generating a target parameter matrix corresponding to each convolutional layer in the target convolutional neural network includes:

[0027] For each convolution layer in the target convolutional neural network, an initial parameter matrix corresponding to the convolution layer is obtained; wherein the elements in the initial parameter matrix correspond one-to-one to the convolution kernels in the convolution layer; and the elements in the initial parameter matrix are arranged in the order of the input channels and output channels to which the corresponding convolution kernels belong;

[0028] Based on the first compression step size and the second compression step size, the positions of the rows and columns in the initial parameter matrix are adjusted to obtain a target parameter matrix corresponding to the convolution layer.

[0029] Optionally, adjusting the positions of rows and columns in the initial parameter matrix based on the first compression step size and the second compression step size to obtain a target parameter matrix corresponding to the convolution layer includes:

[0030] For each candidate row in the initial parameter matrix, determine the element corresponding to the unreplaced convolution kernel in the candidate row as the first element; wherein the candidate row includes: the row where the element corresponding to the unreplaced convolution kernel in the first column of the initial parameter matrix is ​​located;

[0031] Move the columns of each adjacent element of the first compression step length in the first element to adjacent positions;

[0032] For each candidate column in the initial parameter matrix after the move, determine the element corresponding to the unreplaced convolution kernel in the candidate column as the second element; wherein the candidate column includes: the column where the element corresponding to the unreplaced convolution kernel in the first row of the initial parameter matrix is ​​located;

[0033] Move each adjacent row of the second compression step length elements in the second element to an adjacent position to obtain a target parameter matrix corresponding to the convolution layer.

[0034] In a second aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application discloses a data processing method, which is applied to a field programmable gate array FPGA, and the method includes:

[0035] Get the image to be processed from the preset storage space;

[0036] Through the convolution layer operation kernel, based on the target parameter matrix of each convolution layer in the target convolution neural network, the image to be processed is processed to obtain a final feature map;

[0037] Among them, the elements in the target parameter matrix corresponding to each convolution layer in the target convolution neural network correspond to the convolution kernels in the convolution layer one by one; the convolution kernels in the convolution layer include: compressed convolution kernels and uncompressed convolution kernels, and the size of the compressed convolution kernels is smaller than that of the uncompressed convolution kernels; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolution layer, respectively; the target convolution neural network is: for each convolution layer in the initial convolution neural network, the specified part of the convolution kernels in the convolution layer is compressed to obtain the compressed convolution kernels, and the model is trained; the specified part of the convolution kernels is set based on the number of input channels and the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation kernel of the FPGA;

[0038] The final feature map is stored in the preset storage space.

[0039] Optionally, the elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network are divided into multiple element groups; the number of rows of the target parameter matrix is ​​the number of output channels of the convolution layer, and the number of columns of the target parameter matrix is ​​the number of input channels of the convolution layer; each element group contains M×N elements; each element group corresponds to a group of compressed convolution kernels or to a group of uncompressed convolution kernels; the convolution kernels of at least two element groups in the multiple element groups are of the same size, and the convolution kernels of at least two element groups are of different sizes; M is the input parallelism of the convolution layer operation kernel, and N is the output parallelism of the convolution layer operation kernel;

[0040] The process of processing the image to be processed by the convolutional layer operation kernel based on the target parameter matrix of each convolutional layer in the target convolutional neural network to obtain a final feature map includes:

[0041] Through the convolution layer operation kernel, based on the target parameter matrix of each convolution layer in the target convolutional neural network, the image to be processed is convolved in parallel to obtain a final feature map.

[0042] In a third aspect, in order to achieve the above-mentioned purpose, the embodiment of the present application discloses a convolutional neural network training method, the method comprising:

[0043] For each convolutional layer in the initial convolutional neural network, based on the number of input channels and the number of output channels of the convolutional layer, and the input parallelism and output parallelism of the convolutional layer operation core of the field programmable gate array FPGA, determine the compression step size for compressing the convolutional kernels in the convolutional layer; wherein the compression step size is used to indicate: when compressing the convolutional layer, the proportion of the convolutional kernels that need to be compressed in the convolutional layer;

[0044] According to the determined compression step size, a portion of the convolution kernels specified in the convolution layer are compressed into target convolution kernels to obtain an alternative convolution neural network;

[0045] Training the candidate convolutional neural network to obtain a target convolutional neural network;

[0046] Generate a target parameter matrix corresponding to each convolutional layer in the target convolutional neural network; wherein the elements in the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network correspond one-to-one to the convolution kernels in the convolutional layer; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolutional layer, respectively.

[0047] Optionally, the elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network are divided into multiple element groups; the number of rows of the target parameter matrix is ​​the number of output channels of the convolution layer, and the number of columns of the target parameter matrix is ​​the number of input channels of the convolution layer; each element group contains M×N elements; each element group corresponds to a group of compressed convolution kernels or to a group of uncompressed convolution kernels; the convolution kernels of at least two element groups in the multiple element groups are of the same size, and the convolution kernels of at least two element groups are of different sizes; M is the input parallelism of the convolution layer operation kernel, and N is the output parallelism of the convolution layer operation kernel.

[0048] In a fourth aspect, in order to achieve the above-mentioned purpose, the embodiment of the present application discloses a field programmable gate array FPGA, wherein the FPGA includes a DDR (Double Data Rate SDRAM, double rate synchronous dynamic random access memory) controller, an AXI (Advanced eXtensible Interface, advanced extensible interface) bus, a state controller, a feature map buffer, a parameter buffer, an output buffer, an adder, a read-write controller and at least one convolutional layer operation core with an input parallelism of M and an output parallelism of N, wherein:

[0049] The DDR controller is used to obtain the image to be processed from the preset storage space and store the image to be processed in the feature map buffer area;

[0050] The state controller is used to control the read-write controller to read the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network from the parameter buffer area, read the candidate image from the feature map buffer area, and send it to the convolutional layer operation core; wherein the candidate image is a to-be-processed image or feature map stored in the feature map buffer area; the elements in the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network correspond one-to-one to the convolution kernel in the convolutional layer; the convolution kernel in the convolutional layer includes: a compressed convolution kernel and an uncompressed convolution kernel, and the compressed convolution The size of the kernel is smaller than the uncompressed convolution kernel; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolution layer, respectively; the target convolution neural network is: for each convolution layer in the initial convolution neural network, a specified part of the convolution kernel in the convolution layer is compressed to obtain a compressed convolution kernel, and the model is trained; the specified part of the convolution kernel is set based on the number of input channels and the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation kernel of the FPGA;

[0051] The convolution layer operation core is used to perform convolution processing on the received candidate graph based on the target parameter matrix corresponding to each convolution layer according to a preset clock cycle;

[0052] The state controller is further used to control the read-write controller, and for each convolution layer of the target convolutional neural network, the convolution calculation results of each clock cycle of the convolution layer operation kernel are stored in the output buffer area respectively; for each output channel of the convolution layer, the bias value of the output channel is read from the parameter buffer area, and the adder is controlled to accumulate the convolution calculation results of each clock cycle of the output channel stored in the output buffer area, and calculate the sum of the accumulated result and the bias value of the output channel to obtain the feature map corresponding to the output channel, and store it in the output buffer area;

[0053] The DDR controller is also used to store the feature map obtained based on each target convolutional layer in the output buffer to the feature map buffer through the AXI bus; obtain the final feature map obtained based on the last convolutional layer in the target convolutional neural network from the output buffer, and store it in the preset storage space; wherein the target convolutional layer includes: other convolutional layers in the target convolutional neural network except the last convolutional layer.

[0054] Optionally, the elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network are divided into multiple element groups; the number of rows of the target parameter matrix is ​​the number of output channels of the convolution layer, and the number of columns of the target parameter matrix is ​​the number of input channels of the convolution layer; each element group contains M×N elements; each element group corresponds to a group of compressed convolution kernels or to a group of uncompressed convolution kernels; the convolution kernels of at least two element groups in the multiple element groups are of the same size, and the convolution kernels of at least two element groups are of different sizes;

[0055] The convolution layer operation core is used to perform convolution processing on the received candidate images in parallel according to a preset clock cycle and based on the target parameter matrix corresponding to each convolution layer.

[0056] In a fifth aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application discloses a data processing device, the device comprising:

[0057] A module for obtaining images to be processed, used to obtain images to be processed;

[0058] A storage module is used to store the image to be processed in a preset storage space, so that the field programmable gate array FPGA obtains the image to be processed from the preset storage space, and processes the image to be processed through a convolution layer operation core based on a target parameter matrix corresponding to each convolution layer in a target convolutional neural network to obtain a final feature map, and stores it in the preset storage space;

[0059] Among them, the elements in the target parameter matrix corresponding to each convolution layer in the target convolution neural network correspond to the convolution kernels in the convolution layer one by one; the convolution kernels in the convolution layer include: compressed convolution kernels and uncompressed convolution kernels, and the size of the compressed convolution kernels is smaller than that of the uncompressed convolution kernels; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolution layer, respectively; the target convolution neural network is: for each convolution layer in the initial convolution neural network, the specified part of the convolution kernels in the convolution layer is compressed to obtain the compressed convolution kernels, and the model is trained; the specified part of the convolution kernels is set based on the number of input channels and the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation kernel of the FPGA;

[0060] A feature map acquisition module is used to acquire the final feature map from the preset storage space.

[0061] Optionally, the elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network are divided into multiple element groups; the number of rows of the target parameter matrix is ​​the number of output channels of the convolution layer, and the number of columns of the target parameter matrix is ​​the number of input channels of the convolution layer; each element group contains M×N elements; each element group corresponds to a group of compressed convolution kernels or to a group of uncompressed convolution kernels; the convolution kernels of at least two element groups in the multiple element groups are of the same size, and the convolution kernels of at least two element groups are of different sizes; M is the input parallelism of the convolution layer operation kernel, and N is the output parallelism of the convolution layer operation kernel, so that the FPGA performs convolution processing in parallel through the convolution layer operation kernel based on the target parameter matrix corresponding to the convolution layer.

[0062] Optionally, the device further comprises:

[0063] A compression step size determination module is used to determine, for each convolution layer in the initial convolutional neural network, a compression step size for compressing the convolution layer based on the number of input channels and the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation kernel; wherein the compression step size is used to indicate: when compressing the convolution layer, the proportion of the convolution kernels in the convolution layer that need to be compressed;

[0064] A convolution kernel compression module is used to compress a specified part of the convolution kernels in the convolution layer into a target convolution kernel according to a determined compression step size, so as to obtain a candidate convolution neural network;

[0065] A training module, used to train the candidate convolutional neural network to obtain a target convolutional neural network;

[0066] The target parameter matrix generation module is used to generate a target parameter matrix corresponding to each convolutional layer in the target convolutional neural network.

[0067] Optionally, the compression step length determination module includes:

[0068] A first calculation submodule is used to calculate, for each convolutional layer in the initial convolutional neural network, a ratio of the number of input channels of the convolutional layer to the input parallelism of the convolutional layer operation core as a first ratio, and calculate a ratio of the number of output channels of the convolutional layer to the output parallelism of the convolutional layer operation core as a second ratio;

[0069] A compression step determination submodule is used to determine a first compression step and a second compression step when compressing the convolution layer based on the first ratio and the second ratio; wherein the first compression step is used to indicate: when compressing the convolution layer, the proportion of convolution kernels corresponding to each output channel that need to be compressed; the second compression step is used to indicate: when compressing the convolution layer, the proportion of convolution kernels corresponding to each input channel that need to be compressed.

[0070] Optionally, the compression step size determination submodule is specifically used to determine a first compression step size and a second compression step size when compressing the convolution layer based on the smaller value of the first ratio and the second ratio if the input parallelism of the convolution layer operation kernel is equal to the output parallelism;

[0071] If the input parallelism of the convolution layer operation kernel is not equal to the output parallelism, a first compression step size when compressing the convolution layer is determined based on the first ratio, and a second compression step size when compressing the convolution layer is determined based on the second ratio.

[0072] Optionally, the convolution kernel compression module is specifically used to divide the convolution kernels corresponding to each output channel in the convolution layer into multiple first convolution kernel groups on average according to the order of the corresponding input channels; wherein the number of convolution kernels contained in each first convolution kernel group is the first compression step size; for each first convolution kernel group, determine the other convolution kernels in the first convolution kernel group except the convolution kernel at the specified position as the first convolution kernel to be processed; compress the first convolution kernel to be processed in each first convolution kernel group into a target convolution kernel to obtain an alternative convolution neural network; or,

[0073] For each convolution kernel corresponding to each input channel in the convolution layer, they are evenly divided into multiple second convolution kernel groups according to the order of the corresponding output channels; wherein the number of convolution kernels contained in each second convolution kernel group is the second compression step size; for each second convolution kernel group, the other convolution kernels in the second convolution kernel group except the convolution kernel at the specified position are determined as the second convolution kernels to be processed; the second convolution kernels to be processed in each second convolution kernel group are compressed into the target convolution kernel to obtain an alternative convolution neural network.

[0074] Optionally, the target parameter matrix generation module includes:

[0075] An initial parameter matrix acquisition submodule is used to acquire an initial parameter matrix corresponding to each convolution layer in the target convolutional neural network; wherein the elements in the initial parameter matrix correspond one-to-one to the convolution kernels in the convolution layer; and the elements in the initial parameter matrix are arranged in the order of the input channels and output channels to which the corresponding convolution kernels belong;

[0076] The target parameter matrix generation submodule is used to adjust the positions of rows and columns in the initial parameter matrix based on the first compression step size and the second compression step size to obtain a target parameter matrix corresponding to the convolution layer.

[0077] Optionally, the target parameter matrix generation submodule includes:

[0078] A first element determination unit is used to determine, for each candidate row in the initial parameter matrix, an element corresponding to the uncompressed convolution kernel in the candidate row as the first element; wherein the candidate row includes: a row where the element corresponding to the uncompressed convolution kernel in the first column of the initial parameter matrix is ​​located;

[0079] A first moving unit is used to move the columns where each adjacent element of the first compression step length is located in the first elements to adjacent positions;

[0080] A second element determination unit is used to determine, for each candidate column in the initial parameter matrix after the move, an element corresponding to the uncompressed convolution kernel in the candidate column as a second element; wherein the candidate column includes: a column where the element corresponding to the uncompressed convolution kernel in the first row of the initial parameter matrix is ​​located;

[0081] The second moving unit is used to move the rows where each adjacent second compression step element in the second element is located to adjacent positions to obtain a target parameter matrix corresponding to the convolution layer.

[0082] In a sixth aspect, in order to achieve the above-mentioned purpose, the embodiment of the present application discloses a convolutional neural network training device, the device comprising:

[0083] A compression step size determination module is used to determine, for each convolution layer in the initial convolutional neural network, a compression step size for compressing the convolution kernels in the convolution layer based on the number of input channels and the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation core of the field programmable gate array FPGA; wherein the compression step size is used to indicate: when compressing the convolution layer, the proportion of the convolution kernels that need to be compressed in the convolution layer;

[0084] A convolution kernel compression module is used to compress a specified part of the convolution kernels in the convolution layer into a target convolution kernel according to a determined compression step size, so as to obtain a candidate convolution neural network;

[0085] A training module, used to train the candidate convolutional neural network to obtain a target convolutional neural network;

[0086] A target parameter matrix generation module is used to generate a target parameter matrix corresponding to each convolutional layer in the target convolutional neural network; wherein the elements in the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network correspond one-to-one to the convolution kernel in the convolutional layer; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolutional layer, respectively.

[0087] Optionally, the elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network are divided into multiple element groups; the number of rows of the target parameter matrix is ​​the number of output channels of the convolution layer, and the number of columns of the target parameter matrix is ​​the number of input channels of the convolution layer; each element group contains M×N elements; each element group corresponds to a group of compressed convolution kernels or to a group of uncompressed convolution kernels; the convolution kernels of at least two element groups in the multiple element groups are of the same size, and the convolution kernels of at least two element groups are of different sizes; M is the input parallelism of the convolution layer operation kernel, and N is the output parallelism of the convolution layer operation kernel.

[0088] In another aspect of the implementation of the present application, in order to achieve the above-mentioned purpose, the embodiment of the present application further discloses an electronic device, the electronic device comprising a processor, a communication interface, a memory, a field programmable gate array FPGA and a communication bus, wherein the processor, the communication interface, the FPGA and the memory communicate with each other through the communication bus;

[0089] The memory is used to store computer programs;

[0090] The processor is configured to implement the data processing method described in any one of the first aspects above when executing the program stored in the memory;

[0091] The FPGA is used to execute any data processing method described in the second aspect above.

[0092] In another aspect of the implementation of the present application, in order to achieve the above-mentioned purpose, the embodiment of the present application further discloses an electronic device, the electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0093] The memory is used to store computer programs;

[0094] The processor is used to implement the convolutional neural network training method as described in any one of the third aspects above when executing the program stored in the memory.

[0095] In another aspect of the implementation of the present application, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is executed on a computer, the data processing method described in any one of the first aspects above, or any one of the convolutional neural network training methods described in the third aspect above is implemented.

[0096] In another aspect of the implementation of the present application, an embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any data processing method described in the first aspect above, or any convolutional neural network training method described in the third aspect above.

[0097] The embodiment of the present application provides a data processing method, which can obtain a picture to be processed and store the picture to be processed in a preset storage space. Accordingly, the FPGA obtains the picture to be processed from the preset storage space, and processes the picture to be processed based on the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network through the convolutional layer operation kernel to obtain the final feature map, and stores it in the preset storage space; wherein the elements in the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network correspond to the convolutional kernel in the convolutional layer one by one; the convolutional kernel in the convolutional layer includes: compressed convolution kernel and an uncompressed convolution kernel, and the size of the compressed convolution kernel is smaller than that of the uncompressed convolution kernel; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolution layer, respectively; the target convolution neural network is: for each convolution layer in the initial convolution neural network, compressing part of the convolution kernel specified in the convolution layer to obtain a compressed convolution kernel, and performing model training; the specified part of the convolution kernel is set based on the number of input channels, the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation kernel of the FPGA.

[0098] Based on the above processing, some of the convolution kernels specified in the convolution layer are compressed, which can reduce the number of network parameters of the convolution layer, thereby reducing the number of network parameters of the target convolutional neural network and reducing the amount of calculation. Furthermore, FPGA performs convolution processing on the image to be processed based on the compressed convolution layer, which can improve the efficiency of data processing based on the convolutional neural network.

[0099] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0101] Figure 1 A flow chart of a data processing method provided in an embodiment of the present application;

[0102] Figure 2 A flowchart of a method for obtaining a target parameter matrix in a data processing process provided in an embodiment of the present application;

[0103] Figure 3 A flowchart of another method for obtaining a target parameter matrix in the data processing process provided in an embodiment of the present application;

[0104] Figure 4 A schematic diagram of the structure of a convolutional layer provided in an embodiment of the present application;

[0105] Figure 5 For Figure 4 A schematic diagram of a convolution kernel arrangement obtained by compressing the convolution layer in ;

[0106] Figure 6 For Figure 4 Schematic diagram of another convolution kernel arrangement obtained by compressing the convolution layer in ;

[0107] Figure 7 Another schematic diagram of convolution kernel arrangement provided in an embodiment of the present application;

[0108] Figure 8 Another convolution kernel arrangement schematic diagram provided in an embodiment of the present application;

[0109] Fig. 9 For Figure 8 A schematic diagram of a convolution kernel arrangement obtained by compressing the corresponding convolution layer;

[0110] Fig.10 Based on Fig. 9 A schematic diagram of a convolution kernel arrangement obtained by moving the convolution kernel in the column;

[0111] Fig.11 Based on Fig.10 A schematic diagram of a convolution kernel arrangement obtained by moving the convolution kernel in rows;

[0112] Fig.12 Based on Fig. 9A schematic diagram of a convolution kernel arrangement obtained by moving the columns and rows of the convolution kernels in FIG.

[0113] Fig.13 Another flow chart of the data processing method provided in the embodiment of the present application;

[0114] Fig.14 A schematic diagram of the structure of an FPGA provided in an embodiment of the present application;

[0115] Fig.15 A schematic diagram of a principle of performing data processing based on FPGA running a convolutional neural network provided in an embodiment of the present application;

[0116] Fig.16 A structural diagram of a data processing device provided in an embodiment of the present application;

[0117] Fig.17 A structural diagram of a convolutional neural network training device provided in an embodiment of the present application;

[0118] Fig.18 A structural diagram of an electronic device provided in an embodiment of the present application;

[0119] Fig.19 Another structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0120] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0121] In the related art, the number of network parameters of the convolutional neural network is large, and the data processing based on the convolutional neural network running on the CPU is a serial processing method, which will reduce the efficiency of data processing based on the convolutional neural network.

[0122] In order to solve the above problems, the present application provides a data processing method, see Figure 1 , Figure 1 A flowchart of a data processing method provided in an embodiment of the present application, the method may include the following steps:

[0123] S101: Obtain the image to be processed.

[0124] S102: The image to be processed is stored in a preset storage space, so that the FPGA obtains the image to be processed from the preset storage space, and processes the image to be processed through the convolution layer operation kernel based on the target parameter matrix corresponding to each convolution layer in the target convolutional neural network to obtain the final feature map, and stores it in the preset storage space.

[0125] Among them, the elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network correspond one-to-one to the convolution kernels in the convolutional layer; the convolution kernels in the convolutional layer include: compressed convolution kernels and uncompressed convolution kernels, and the size of the compressed convolution kernels is smaller than that of the uncompressed convolution kernels; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolutional layer, respectively; the target convolutional neural network is: for each convolution layer in the initial convolutional neural network, the specified part of the convolution kernels in the convolution layer is compressed to obtain the compressed convolution kernels, and the model is trained; the specified part of the convolution kernels is set based on the number of input channels, the number of output channels of the convolutional layer, and the input parallelism and output parallelism of the convolutional layer operation kernel of the FPGA.

[0126] S103: Obtain a final feature map from a preset storage space.

[0127] The data processing method provided in the embodiment of the present application compresses part of the convolution kernels specified in the convolution layer, which can reduce the number of network parameters of the convolution layer, thereby reducing the number of network parameters of the target convolutional neural network and reducing the amount of calculation. Furthermore, the FPGA performs convolution processing on the image to be processed based on the compressed convolution layer, which can improve the efficiency of data processing based on the convolutional neural network.

[0128] In one embodiment, the method may be applied to an electronic device, which is used to process an acquired image to be processed. For example, the electronic device may be a monitoring device, or the electronic device may also be a server capable of communicating with the monitoring device.

[0129] For step S102, the target convolutional neural network can be pre-trained, and the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network can be obtained and written into the cache area of ​​the FPGA (for example, the parameter cache area mentioned later in the embodiment of the present application). Correspondingly, after the FPGA obtains the image to be processed from the preset storage space, the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network can be obtained from the parameter cache area, and the image to be processed can be processed.

[0130] In one embodiment, some convolution kernels in the convolution layer may be deleted, thereby reducing the number of network parameters of the convolution layer to reduce the number of network parameters of the target convolutional neural network, thereby reducing the amount of calculation. Accordingly, the number of rows and columns of the target parameter matrix may be smaller than the number of output channels and the number of input channels of the convolution layer, respectively.

[0131] An FPGA may include one or more convolutional layer cores. The input parallelism and output parallelism of a convolutional layer core are used to indicate the number of computing units included in the convolutional layer core. For example, a convolutional layer core with an input parallelism of M and an output parallelism of N may include M×N computing units.

[0132] In one embodiment, the elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network are divided into multiple element groups; the number of rows of the target parameter matrix is ​​the number of output channels of the convolution layer, and the number of columns of the target parameter matrix is ​​the number of input channels of the convolution layer. Each element group contains M×N elements; each element group corresponds to a group of compressed convolution kernels or to a group of uncompressed convolution kernels. The convolution kernels of at least two of the multiple element groups are of the same size, and the convolution kernels of at least two element groups are of different sizes.

[0133] The size of the convolution kernel indicates the number of elements contained in the convolution kernel. For example, a 1×1 convolution kernel contains one element, and a 3×3 convolution kernel contains 9 elements.

[0134] It can be understood that the uncompressed convolution kernel is the original convolution kernel in the convolutional neural network. Correspondingly, if a uncompressed convolution kernel in the convolutional neural network is compressed, the corresponding compressed convolution kernel can be obtained, that is, the compressed convolution kernel is obtained by compressing the uncompressed convolution kernel. Compressing the uncompressed convolution kernel means reducing the number of elements contained in the uncompressed convolution kernel. For example, a 3×3 uncompressed convolution kernel is compressed into a 1×1 compressed convolution kernel, and a 5×5 uncompressed convolution kernel is compressed into a 3×3 compressed convolution kernel.

[0135] That is, the size of the convolution kernels corresponding to the M×N elements in each element group in the target parameter matrix is ​​the same, that is, the data of the convolution kernels corresponding to the M×N elements in the element group can be written into the M×N computing units of the convolution layer operation kernel respectively. Accordingly, each computing unit can perform convolution processing based on the data of the written convolution kernel, and further, parallel convolution processing can be implemented within one clock cycle of the FPGA to improve the efficiency of data processing.

[0136] In addition, if the FPGA includes multiple convolutional layer operation cores, convolution operations can be performed simultaneously in parallel based on the multiple convolutional layer operation cores in one clock cycle, further improving the efficiency of data processing.

[0137] In one embodiment, see Figure 2 , the process of obtaining the target parameter matrix can include the following steps:

[0138] S201: For each convolutional layer in the initial convolutional neural network, determine a compression step for compressing the convolutional layer based on the number of input channels and output channels of the convolutional layer, and the input parallelism and output parallelism of the convolutional layer operation kernel.

[0139] The compression step size is used to indicate the proportion of convolution kernels that need to be compressed in the convolution layer when compressing the convolution layer.

[0140] S202: According to the determined compression step size, compress the specified part of the convolution kernels in the convolution layer into the target convolution kernel to obtain an alternative convolution neural network.

[0141] S203: Train the candidate convolutional neural network to obtain a target convolutional neural network.

[0142] S204: Generate a target parameter matrix corresponding to each convolutional layer in the target convolutional neural network.

[0143] In an embodiment of the present application, the convolution layer is compressed, that is, the size of a portion of the convolution kernels in the convolution layer is reduced so that the convolution layer contains convolution kernels of different sizes.

[0144] In one embodiment, the size of the convolution kernel in the convolution layer can be 3×3, or 5×5, but is not limited thereto. Accordingly, if the size of the convolution kernel in the convolution layer is 3×3, some of the convolution kernels specified in the convolution layer can be compressed into a 1×1 target convolution kernel; if the size of the convolution kernel in the convolution layer is 5×5, some of the convolution kernels specified in the convolution layer can be compressed into a 3×3 target convolution kernel, or some of the convolution kernels specified in the convolution layer can be compressed into a 1×1 target convolution kernel.

[0145] In one embodiment, some of the convolution kernels specified in the convolution layer can be compressed into target convolution kernels of the same size. For example, if the size of the convolution kernel in the convolution layer is 5×5, some of the convolution kernels specified in the convolution layer can be compressed into target convolution kernels of 3×3; or, if the size of the convolution kernel in the convolution layer is 5×5, some of the convolution kernels specified in the convolution layer can be compressed into target convolution kernels of 1×1.

[0146] In one embodiment, Figure 2 Based on Figure 3 , step S201 may include the following steps:

[0147] S2011: For each convolutional layer in the initial convolutional neural network, calculate the ratio of the number of input channels of the convolutional layer to the input parallelism of the convolutional layer operation kernel as a first ratio, and calculate the ratio of the number of output channels of the convolutional layer to the output parallelism of the convolutional layer operation kernel as a second ratio.

[0148] S2012: Based on the first ratio and the second ratio, determine a first compression step size and a second compression step size when compressing the convolution layer.

[0149] The first compression step size is used to indicate the proportion of convolution kernels that need to be compressed in the convolution kernels corresponding to each output channel when compressing the convolution layer. The second compression step size is used to indicate the proportion of convolution kernels that need to be compressed in the convolution kernels corresponding to each input channel when compressing the convolution layer.

[0150] See also Figure 4 , Figure 4 A schematic diagram of the structure of a convolutional layer provided in an embodiment of the present application. Figure 4 In , N represents the number of output channels of the convolutional layer, and C represents the number of input channels of the convolutional layer. In a convolutional neural network, the number of output channels of a convolutional layer is the same as the number of input channels of the next convolutional layer. The number of input channels of a convolutional layer is the same as the number of feature maps input to the convolutional layer.

[0151] Figure 4 In the convolution layer, the number of output channels is 4, and the number of input channels is 4. That is to say, the convolution layer contains 4×4=16 convolution kernels, which are K 00 , K 01 , K 02 , …K 31 , K 32 , K 33 .

[0152] K 00 , K 01 , K 02 and K 03 Corresponding to the same output channel; K 10 , K 11 , K 12 and K 13 Corresponding to the same output channel; K 20 , K 21 , K 22 and K 23 Corresponding to the same output channel; K30 , K 31 , K 32 and K 33 Corresponding to the same output channel.

[0153] K 00 , K 10 , K 20 and K 30 Corresponding to the same input channel; K 01 , K 11 , K 21 and K 31 Corresponding to the same input channel; K 02 , K 12 , K 22 and K 32 Corresponding to the same input channel; K 03 , K 13 , K 23 and K 33 Corresponding to the same input channel.

[0154] Since the number of input channels of the convolutional layer is 4, that is, 4 feature maps can be input to the convolutional layer, and accordingly, the convolution kernels corresponding to the same output channel (for example, K 00 , K 01 , K 02 and K 03 ), one of the four feature maps can be convolved respectively, and then, for the results of the four convolutions, the values ​​at the corresponding positions can be added, and the result of the addition can be added to the bias value of the output channel to obtain the output result corresponding to the output channel. Accordingly, the convolution layer can output four feature maps to the next convolution layer.

[0155] In order to make the structure of the compressed convolution layer match the convolution layer operation core of the FPGA, the ratio of the number of input channels of the convolution layer to the input parallelism M of the convolution layer operation core of the FPGA (i.e., the first ratio) can be calculated, and the ratio of the number of output channels of the convolution layer to the output parallelism N of the convolution layer operation core of the FPGA (i.e., the second ratio) can be calculated.

[0156] Furthermore, based on the first ratio and the second ratio, a first compression step size and a second compression step size when compressing the first convolutional layer can be determined.

[0157] For example, the first compression step size can be represented by P1, and the second compression step size can be represented by P2. Figure 4In the structure of the convolution layer, if the size of the convolution kernel in the convolution layer is 3×3, and P1 is 2, P2 is 2. That is to say, when compressing the convolution layer, the proportion of the convolution kernels that need to be compressed in each output channel is 1 / 2; the proportion of the convolution kernels that need to be compressed in each output channel is 1 / 2. Correspondingly, we can get Figure 5 The structure shown.

[0158] Figure 5 In the figure, the size of the white convolution kernel is 3×3, and the size of the gray convolution kernel is 1×1. It can be seen that among the convolution kernels corresponding to each output channel, one of every two convolution kernels is compressed into a smaller convolution kernel (i.e., a 1×1 convolution kernel), and among the convolution kernels corresponding to each input channel, one of every two convolution kernels is compressed into a smaller convolution kernel (i.e., a 1×1 convolution kernel). That is, the proportion of the reduced convolution kernels in the convolution kernels corresponding to each output channel is 1 / 2; the proportion of the reduced convolution kernels in the convolution kernels corresponding to each output channel is 1 / 2.

[0159] against Figure 4 In the structure of the convolution layer, if the size of the convolution kernel in the convolution layer is 3×3, and P1 is 4, P2 is 4. That is to say, when compressing the convolution layer, the proportion of the convolution kernels corresponding to each output channel that need to be reduced is 3 / 4; the proportion of the convolution kernels corresponding to each output channel that need to be reduced is 3 / 4. Correspondingly, we can get Figure 6 The structure shown.

[0160] Figure 6 In the figure, the size of the white convolution kernel is 3×3, and the size of the gray convolution kernel is 1×1. It can be seen that among the convolution kernels corresponding to each output channel, three out of every four convolution kernels are compressed into smaller convolution kernels (i.e., 1×1 convolution kernels), and among the convolution kernels corresponding to each input channel, three out of every four convolution kernels are compressed into smaller convolution kernels (i.e., 1×1 convolution kernels). That is, the proportion of reduced convolution kernels in the convolution kernels corresponding to each output channel is 3 / 4; the proportion of reduced convolution kernels in the convolution kernels corresponding to each output channel is 3 / 4.

[0161] In one embodiment, different approaches may be adopted to determine the first compression step size and the second compression step size in combination with the relationship between the input parallelism and the output parallelism of the convolution layer operation kernel.

[0162] Method 1: If the input parallelism of the convolutional layer operation kernel is equal to the output parallelism, then based on the smaller value of the first ratio and the second ratio, determine the first compression step size and the second compression step size when compressing the convolutional layer.

[0163] In one embodiment, if the input parallelism of the convolution layer operation kernel is equal to the output parallelism, the first compression step size and the second compression step size can be determined based on the smaller value of the first ratio and the second ratio.

[0164] For example, if the number of output channels of the convolutional layer is 96, the number of input channels is 128, the input parallelism M of the convolutional layer operation kernel is 32, and the output parallelism N is 32, then the first ratio is 128 / 32=4, and the second ratio is 96 / 32=3. Therefore, P1=P2=3 can be determined. Figure 7 .

[0165] Figure 7 In the example, a step size of P=3 is used for compression in both the input channel direction and the output channel direction. Since the number of input channels is greater than the number of output channels, the convolution kernels corresponding to the redundant input channels can no longer be compressed. Thus, the compressed convolutional neural network can be guaranteed to have a higher accuracy.

[0166] Method 2: If the input parallelism of the convolutional layer operation kernel is not equal to the output parallelism, a first compression step size when compressing the convolutional layer is determined based on the first ratio, and a second compression step size when compressing the convolutional layer is determined based on the second ratio.

[0167] In one embodiment, if the input parallelism of the convolution layer operation kernel is not equal to the output parallelism, the first compression step size can be determined based on the first ratio, and the second compression step size can be determined based on the second ratio.

[0168] For example, the number of output channels of the convolutional layer is 192, the number of input channels is 64, the input parallelism M of the convolutional layer operation kernel is 32, and the output parallelism N is 64. Then the first ratio is 64 / 32=2, and the second ratio is 192 / 64=3. Furthermore, it can be determined that P1=2 and P2=3.

[0169] In one embodiment, in order to avoid a large number of compressed convolution kernels, which in turn causes a lower accuracy of the target convolutional neural network, when the determined compression step size is larger than the preset step size, the divisor of the determined compression step size can be calculated as the final compression step size.

[0170] For example, the preset step size can be 5, the number of output channels of the convolution layer is 128, the number of input channels is 192, the input parallelism M of the convolution layer operation kernel is 32, and the output parallelism N is 64, then the first ratio can be 192 / 32=6, the second ratio can be 128 / 64=2, and then, P1=6, P2=3 can be determined. Since the calculated P1 is greater than the preset step size, the divisor of 6 (2 or 3) can be determined as the final compression step size, that is, P1=2, P2=3; or, P1=3, P2=3.

[0171] In one embodiment, if the number of input channels of the convolution layer is not an integer multiple of the input parallelism of the convolution layer operation kernel, the input channels of the convolution layer may be expanded so that the number of input channels of the convolution layer after expansion is an integer multiple of the input parallelism of the convolution layer operation kernel. The value of the convolution kernel corresponding to the expanded input channel may be 0.

[0172] Similarly, if the number of output channels of the convolutional layer is not an integer multiple of the output parallelism of the convolutional layer kernel, the output channels of the convolutional layer can be expanded so that the number of output channels of the convolutional layer after expansion is an integer multiple of the output parallelism of the convolutional layer kernel. The value of the convolutional kernel corresponding to the expanded output channel can be 0.

[0173] In one embodiment, different methods can be used for compression to obtain alternative convolutional neural networks.

[0174] Method 1: For each convolution kernel corresponding to each output channel in the convolution layer, divide them into multiple first convolution kernel groups on average according to the order of the corresponding input channels; for each first convolution kernel group, determine the other convolution kernels in the first convolution kernel group except the convolution kernel at the specified position as the first convolution kernel to be processed; compress the first convolution kernel to be processed in each first convolution kernel group into a target convolution kernel to obtain an alternative convolution neural network.

[0175] The number of convolution kernels included in each first convolution kernel group is the first compression step size.

[0176] In one embodiment, see Figure 8 , Figure 8 Schematic diagram of the arrangement of convolution kernels in a 12×12 convolution layer. Each row represents a convolution kernel corresponding to an output channel, and each column represents a convolution kernel corresponding to an input channel.

[0177] If the input parallelism of the convolutional layer kernel is 3 and the output parallelism is 3, then we can get P1=P2=4.

[0178] Therefore, for each convolution kernel corresponding to each output channel in the convolution layer, according to the order of the corresponding input channels, it can be evenly divided into three convolution kernel groups (i.e., first convolution kernel groups). Each first convolution kernel group includes four convolution kernels.

[0179] For example, for the convolution kernel in the first row, K 00 , K 01 , K 02 and K 03 Determine as a first convolution kernel group, K 04 , K 05 , K 06 and K 07 Determine as a first convolution kernel group, K 08 , K 09 , K 010 and K 011 Determine as a first convolution kernel group.

[0180] Then, for each first convolution kernel group, other convolution kernels in the first convolution kernel group except the convolution kernel at the specified position are determined as the first convolution kernels to be processed.

[0181] In one embodiment, the specified position in the first row may be the first position in each first convolution kernel group. For the above example, the convolution kernel at the specified position includes K 00 , K 04 and K 08 .

[0182] Furthermore, K 01 , K 02 , K 03 , K 05 , K 06 , K 07 , K 09 , K 010 and K 011 , compressed into a smaller target convolution kernel.

[0183] Similarly, the specified position in the second row can be the second position in each first convolution kernel group, and accordingly, K 10 , K 12 , K 13 , K 14 , K 16 , K 17 , K 18 , K 010 and K 011 , compressed into a smaller target convolution kernel. By analogy, we can get Fig. 9 Schematic diagram of the arrangement of convolution kernels shown. Fig. 9In the figure, the gray convolution kernel represents the compressed convolution kernel, and the white convolution kernel represents the uncompressed convolution kernel.

[0184] Method 2: For each convolution kernel corresponding to each input channel in the convolution layer, divide them into multiple second convolution kernel groups on average according to the order of the corresponding output channels; for each second convolution kernel group, determine the other convolution kernels in the second convolution kernel group except the convolution kernel at the specified position as the second convolution kernel to be processed; compress the second convolution kernel to be processed in each second convolution kernel group into a target convolution kernel to obtain an alternative convolution neural network.

[0185] The number of convolution kernels included in each second convolution kernel group is the second compression step size. In the embodiment of the present application, the processing is similar to the above-mentioned method 1, and reference may be made to the relevant introduction.

[0186] In actual operation, the convolution kernel can be compressed from the direction of the input channel or from the direction of the output channel, as long as the input channel and the output channel are compressed according to their respective compression step sizes.

[0187] In one embodiment, step S204 may include the following steps:

[0188] Step 1: For each convolutional layer in the target convolutional neural network, obtain the initial parameter matrix corresponding to the convolutional layer.

[0189] The elements in the initial parameter matrix correspond to the convolution kernels in the convolution layer one by one. The elements in the initial parameter matrix are arranged in the order of the input channels and output channels to which the corresponding convolution kernels belong.

[0190] In one embodiment, see Fig.10 ,according to Fig.10 The arrangement of the convolution kernels shown in Figure 1 gives the parameter matrix, which is the initial parameter matrix of the convolution layer.

[0191] Step 2: Based on the first compression step size and the second compression step size, adjust the positions of rows and columns in the initial parameter matrix to obtain the target parameter matrix corresponding to the convolution layer.

[0192] In one embodiment, the above step 2 may include the following steps:

[0193] Step 1: For each candidate row in the initial parameter matrix, determine the element corresponding to the uncompressed convolution kernel in the candidate row as the first element.

[0194] The candidate rows include: the rows where the elements corresponding to the uncompressed convolution kernels in the first column of the initial parameter matrix are located.

[0195] Step 2: Move the columns of every adjacent first compression step length elements in the first element to adjacent positions.

[0196] In the embodiment of the present application, Fig. 9 The initial parameter matrix shown in the figure has an uncompressed convolution kernel K in the first column. 00 , K 40 and K 80 Therefore, it can be determined that the candidate rows include the first row, the fifth row, and the ninth row.

[0197] The uncompressed convolution kernel in the first row is K 00 , K 04 and K 08 Therefore, we can determine that the first element, the fifth element, and the ninth element in the first row are the first elements.

[0198] Then, the first, fifth, and ninth columns can be moved to adjacent positions to obtain Fig.10 The parameter matrix corresponding to the structure diagram shown.

[0199] Step 3: For each candidate column in the initial parameter matrix after the move, determine the element corresponding to the uncompressed convolution kernel in the candidate column as the second element.

[0200] The candidate columns include: the columns where the elements corresponding to the uncompressed convolution kernels in the first row of the initial parameter matrix are located.

[0201] Step 4: Move the rows of each adjacent second compression step element in the second element to adjacent positions to obtain the target parameter matrix corresponding to the convolution layer.

[0202] In the embodiments of the present application, Fig.10 In the first row, the uncompressed convolution kernel is K 00 , K 04 and K 08 Therefore, it can be determined that the candidate columns include the first column, the fifth column, and the ninth column.

[0203] The uncompressed convolution kernel in the first column is K 00 , K 40 and K 80 Therefore, we can determine that the first element, the fifth element, and the ninth element in the first column are the second element. Then, we can move the first row, the fifth row, and the ninth row to adjacent positions, and get Fig.11 The parameter matrix corresponding to the structure diagram shown.

[0204] Similarly, we can determine the first element in the fifth row and move the corresponding column. Determine the second element in the fifth column and move the corresponding row. Determine the first element in the ninth row and move the corresponding column; determine the second element in the ninth column and move the corresponding row, and finally we can get Fig.12 Schematic diagram of the convolution kernel arrangement shown. Fig.12 The parameter matrix corresponding to the schematic diagram shown is also the target parameter matrix.

[0205] In one embodiment, after the target parameter matrix is ​​determined, the target parameter matrix may be converted from a floating point format to a fixed point format to adapt to the computing environment of the FPGA and improve the efficiency of data processing.

[0206] For example, the format of the data may be converted based on formula (1).

[0207] W int =(-1) s ×2 -fl ×round(2 fl ×W f ) (1)

[0208] Among them, W f Indicates the floating point number to be converted, fl indicates the preset number, round indicates rounding operation, S indicates the sign bit, W int Represents the converted fixed-point number.

[0209] Based on the same inventive concept, the embodiment of the present application also provides a convolutional neural network training method, which may include the following steps:

[0210] For each convolutional layer in the initial convolutional neural network, based on the number of input channels and output channels of the convolutional layer, and the input parallelism and output parallelism of the convolutional layer operation core of the FPGA, determine the compression step size for compressing the convolutional kernels in the convolutional layer; wherein the compression step size is used to indicate: when compressing the convolutional layer, the proportion of the convolutional kernels in the convolutional layer that need to be compressed;

[0211] According to the determined compression step size, the specified part of the convolution kernels in the convolution layer is compressed into the target convolution kernel to obtain an alternative convolution neural network; the target convolution kernel is smaller than the specified part of the convolution kernel;

[0212] Train the candidate convolutional neural network to obtain the target convolutional neural network;

[0213] Generate a target parameter matrix corresponding to each convolutional layer in the target convolutional neural network; wherein the elements in the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network correspond one-to-one to the convolution kernels in the convolutional layer; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolutional layer, respectively.

[0214] The convolutional neural network training method provided in the embodiment of the present application compresses part of the convolution kernels specified in the convolution layer, which can reduce the number of network parameters of the convolution layer, thereby reducing the number of network parameters of the target convolutional neural network and reducing the amount of calculation. Furthermore, the FPGA performs convolution processing on the image to be processed based on the compressed convolution layer, which can improve the efficiency of data processing based on the convolutional neural network.

[0215] In one embodiment, the elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network are divided into multiple element groups; the number of rows of the target parameter matrix is ​​the number of output channels of the convolution layer, and the number of columns of the target parameter matrix is ​​the number of input channels of the convolution layer; each element group contains M×N elements; each element group corresponds to a group of compressed convolution kernels or to a group of uncompressed convolution kernels; the convolution kernels of at least two element groups in the multiple element groups are of the same size, and the convolution kernels of at least two element groups are of different sizes.

[0216] Based on the above processing, the convolution kernels corresponding to the M×N elements in each element group in the target parameter matrix have the same size, that is, the data of the convolution kernels corresponding to the M×N elements in the element group can be written into the M×N computing units of the convolution layer operation kernel respectively. Accordingly, each computing unit can perform convolution processing based on the written convolution kernel data, and further, parallel convolution processing can be implemented within one clock cycle of the FPGA to improve the efficiency of data processing.

[0217] Based on the same inventive concept, the present application embodiment also provides a data processing method that can be applied to FPGA, see Fig.13 , the method may include the following steps:

[0218] S1301: Obtain a picture to be processed from a preset storage space.

[0219] S1302: Process the image to be processed through the convolution layer operation kernel based on the target parameter matrix of each convolution layer in the target convolutional neural network to obtain a final feature map.

[0220] Among them, the elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network correspond one-to-one to the convolution kernels in the convolutional layer; the convolution kernels in the convolutional layer include: compressed convolution kernels and uncompressed convolution kernels, and the size of the compressed convolution kernels is smaller than that of the uncompressed convolution kernels; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolutional layer, respectively; the target convolutional neural network is: for each convolution layer in the initial convolutional neural network, the specified part of the convolution kernels in the convolution layer is compressed to obtain the compressed convolution kernels, and the model is trained; the specified part of the convolution kernels is set based on the number of input channels, the number of output channels of the convolutional layer, and the input parallelism and output parallelism of the convolutional layer operation kernel of the FPGA.

[0221] S1303: Store the final feature map into a preset storage space.

[0222] The data processing method provided in the embodiment of the present application compresses part of the convolution kernels specified in the convolution layer, which can reduce the number of network parameters of the convolution layer, thereby reducing the number of network parameters of the target convolutional neural network and reducing the amount of calculation. Furthermore, the FPGA performs convolution processing on the image to be processed based on the compressed convolution layer, which can improve the efficiency of data processing based on the convolutional neural network.

[0223] In one embodiment, the elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network are divided into multiple element groups; the number of rows of the target parameter matrix is ​​the number of output channels of the convolution layer, and the number of columns of the target parameter matrix is ​​the number of input channels of the convolution layer; each element group contains M×N elements; each element group corresponds to a group of compressed convolution kernels or to a group of uncompressed convolution kernels; the convolution kernels of at least two element groups in the multiple element groups are of the same size, and the convolution kernels of at least two element groups are of different sizes.

[0224] Accordingly, S1302 may include:

[0225] Through the convolution layer operation kernel, based on the target parameter matrix of each convolution layer in the target convolutional neural network, the processed image is convolved in parallel to obtain the final feature map.

[0226] In the embodiment of the present application, the FPGA can obtain the image to be processed from the preset storage space, and perform convolution processing on the image to be processed based on the target parameter matrix corresponding to the first convolution layer through convolution calculation and according to the preset clock cycle. In addition, for each output channel of the first convolution layer, the convolution calculation results of each clock cycle of the output channel can be accumulated, and the sum of the accumulated result and the bias value of the output channel is calculated to obtain the feature map corresponding to the output channel.

[0227] Then, the FPGA can perform convolution processing on the feature map obtained by the first convolution layer based on the target parameter matrix corresponding to the second convolution layer through the convolution layer operation core according to the preset clock cycle. In addition, for each output channel of the second convolution layer, the convolution calculation results of each clock cycle of the output channel can be accumulated, and the sum of the accumulated result and the bias value of the output channel can be calculated to obtain the feature map corresponding to the output channel.

[0228] And so on, until the final feature map determined by the last convolutional layer is obtained.

[0229] Based on the above processing, the convolution kernels corresponding to the M×N elements in each element group in the target parameter matrix have the same size, that is, the data of the convolution kernels corresponding to the M×N elements in the element group can be written into the M×N computing units of the convolution layer operation kernel respectively. Accordingly, each computing unit can perform convolution processing based on the written convolution kernel data, and further, parallel convolution processing can be implemented within one clock cycle of the FPGA to improve the efficiency of data processing.

[0230] Based on the same inventive concept, an embodiment of the present application further provides an FPGA, which includes a DDR controller, an AXI bus, a state controller, a feature map buffer, a parameter buffer, an output buffer, an adder, a read / write controller, and at least one convolutional layer operation core with an input parallelism of M and an output parallelism of N, wherein:

[0231] The DDR controller is used to obtain the image to be processed from the preset storage space and store the image to be processed in the feature map buffer area.

[0232] The state controller is used to control the read-write controller, read the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network from the parameter buffer, read the alternative map from the feature map buffer, and send it to the convolutional layer operation core.

[0233] Among them, the alternative image is a to-be-processed image or feature image stored in the feature image buffer; the elements in the target parameter matrix corresponding to each convolution layer in the target convolution neural network correspond one-to-one to the convolution kernel in the convolution layer; the convolution kernel in the convolution layer includes: a compressed convolution kernel and an uncompressed convolution kernel, and the size of the compressed convolution kernel is smaller than the uncompressed convolution kernel; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolution layer, respectively; the target convolution neural network is: for each convolution layer in the initial convolution neural network, compressing part of the convolution kernel specified in the convolution layer to obtain a compressed convolution kernel, and performing model training; the specified part of the convolution kernel is set based on the number of input channels, the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation kernel of the FPGA.

[0234] The convolution layer operation core is used to perform convolution processing on the received candidate graph based on the target parameter matrix corresponding to each convolution layer according to a preset clock cycle.

[0235] The state controller is also used to control the read-write controller, and for each convolution layer of the target convolutional neural network, the convolution calculation results of each clock cycle of the convolution layer operation kernel are stored in the output buffer respectively; for each output channel of the convolution layer, the bias value of the output channel is read from the parameter buffer, and the adder is controlled to accumulate the convolution calculation results of each clock cycle of the output channel stored in the output buffer, and the sum of the accumulated result and the bias value of the output channel is calculated to obtain the feature map corresponding to the output channel, and store it in the output buffer.

[0236] The DDR controller is also used to store the feature map obtained based on each target convolution layer in the output buffer into the feature map cache through the AXI bus; obtain the final feature map obtained based on the last convolution layer in the target convolutional neural network from the output buffer, and store it in a preset storage space.

[0237] The target convolutional layer includes: other convolutional layers except the last convolutional layer in the target convolutional neural network.

[0238] Based on the above processing, the specified part of the convolution kernel in the convolution layer is compressed, which can reduce the number of network parameters of the convolution layer, thereby reducing the number of network parameters of the target convolutional neural network and reducing the amount of calculation. Furthermore, the FPGA performs convolution processing on the image to be processed based on the compressed convolution layer, which can improve the efficiency of data processing based on the convolutional neural network.

[0239] In one embodiment, the elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network are divided into multiple element groups; the number of rows of the target parameter matrix is ​​the number of output channels of the convolution layer, and the number of columns of the target parameter matrix is ​​the number of input channels of the convolution layer; each element group contains M×N elements; each element group corresponds to a group of compressed convolution kernels or to a group of uncompressed convolution kernels; the convolution kernels of at least two element groups in the multiple element groups are of the same size, and the convolution kernels of at least two element groups are of different sizes.

[0240] Correspondingly, the convolution layer operation core can perform convolution processing on the received candidate images in parallel according to the preset clock cycle and based on the target parameter matrix corresponding to each convolution layer.

[0241] Based on the above processing, the convolution kernels corresponding to the M×N elements in each element group in the target parameter matrix have the same size, that is, the data of the convolution kernels corresponding to the M×N elements in the element group can be written into the M×N computing units of the convolution layer operation kernel respectively. Accordingly, each computing unit can perform convolution processing based on the written convolution kernel data, and further, parallel convolution processing can be implemented within one clock cycle of the FPGA to improve the efficiency of data processing.

[0242] In one embodiment, see Fig.14 , Fig.14 A schematic diagram of the structure of the FPGA provided in an embodiment of the present application.

[0243] The target parameter matrix corresponding to each convolutional layer in the target convolutional neural network and the bias value of each output channel of the convolutional layer can be pre-written in the parameter buffer 1404. The above data written can be in the format of fixed-point numbers. The format of the image to be processed stored in the preset storage space can also be in the format of fixed-point numbers. Fig.14 The convolution layer operation core 1408 in the embodiment may include: convolution layer operation core 1, convolution layer operation core 2, convolution layer operation core 3, convolution layer operation core 4, convolution layer operation core 5 and convolution layer operation core 6. However, the present invention is not limited thereto.

[0244] The DDR controller 1401 can obtain the image to be processed from the preset storage space and store the image to be processed in the feature map buffer area 1405 .

[0245] When the image to be processed is processed based on the first convolution layer, the state controller 1409 can control the read-write controller 1407 to read the target parameter matrix corresponding to the first convolution layer from the parameter buffer 1404, read the image to be processed from the feature map buffer 1405, and send it to the convolution layer operation core 1408.

[0246] The convolution layer operation core 1408 can perform convolution processing on the image to be processed according to a preset clock cycle based on the target parameter matrix corresponding to the first convolution layer.

[0247] The state controller 1409 can control the read / write controller 1407 to store the convolution calculation results of each clock cycle of the convolution layer operation core 1408 in the output buffer 1403. In addition, for each output channel of the first convolution layer, the bias value of the output channel is read from the parameter buffer 1403, and the adder 1406 is controlled to accumulate the convolution calculation results of each clock cycle of the output channel stored in the output buffer 1403, and calculate the sum of the accumulated result and the bias value of the output channel to obtain the feature map corresponding to the output channel, and store it in the output buffer 1403.

[0248] The DDR controller 1401 can also store the feature map obtained by the first convolution layer in the feature map buffer 1405 through the AXI bus 1402.

[0249] Then, the state controller 1409 can control the read-write controller 1407 to read the target parameter matrix corresponding to the second convolutional layer from the parameter buffer 1404, read the feature map obtained by the first convolutional layer from the feature map buffer 1405, and send it to the convolutional layer operation core 1408.

[0250] The convolution layer operation core 1408 can perform convolution processing on the feature map obtained by the first convolution layer based on the target parameter matrix corresponding to the second convolution layer according to a preset clock cycle.

[0251] The state controller 1409 can control the read / write controller 1407 to store the convolution calculation results of each clock cycle of the convolution layer operation core 1408 in the output buffer 1403. In addition, for each output channel of the second convolution layer, the bias value of the output channel is read from the parameter buffer 1403, and the adder 1406 is controlled to accumulate the convolution calculation results of each clock cycle of the output channel stored in the output buffer 1403, and calculate the sum of the accumulated result and the bias value of the output channel to obtain the feature map corresponding to the output channel, and store it in the output buffer 1403.

[0252] The DDR controller 1401 can also store the feature map obtained by the second convolution layer in the feature map buffer 1405 through the AXI bus 1402.

[0253] And so on, until the final feature map determined based on the last convolution layer is obtained. The DDR controller 1401 can obtain the final feature map from the output buffer area and store it in a preset storage space.

[0254] See also Fig.15 , Fig.15 A schematic diagram of the principle of data processing based on running a convolutional neural network on an FPGA provided in an embodiment of the present application.

[0255] The parameter buffer can store the target parameter matrix of each convolutional layer of the target convolutional neural network and the bias values ​​of each output channel of each convolutional layer.

[0256] The feature map buffer is used to store the image to be processed, as well as the feature maps obtained based on each target convolutional layer and all convolutional layers except the last convolutional layer in the target convolutional neural network.

[0257] The heterogeneous convolution controller is equivalent to the state controller in the above embodiment. Under the control of the heterogeneous convolution controller, the convolution layer operation core of the FPGA can obtain the image to be processed and the target parameter matrix of each convolution layer, as well as the feature map obtained based on each target convolution layer, and perform convolution processing.

[0258] In addition, for each convolution layer of the target convolutional neural network, the convolution calculation results of each clock cycle of the convolution layer operation kernel can be stored in the output buffer respectively, and for each output channel of the convolution layer, the bias value of the output channel is read from the parameter buffer, and the convolution calculation results of each clock cycle of the output channel stored in the output buffer are accumulated, and the sum of the accumulated result and the bias value of the output channel is calculated to obtain the feature map corresponding to the output channel and store it in the output buffer.

[0259] Fig.15 In the example, the FPGA contains n convolutional layer operation cores, that is, in one clock cycle, the feature map can be convolved based on the n convolutional layer operation cores at the same time.

[0260] Based on the same inventive concept, the present application also provides a data processing device, see Fig.16 , Fig.16 A structural diagram of a data processing device provided in an embodiment of the present application, the device may include:

[0261] The to-be-processed picture acquisition module 1601 is used to acquire the to-be-processed picture;

[0262] The storage module 1602 is used to store the image to be processed in a preset storage space, so that the field programmable logic gate array FPGA obtains the image to be processed from the preset storage space, and processes the image to be processed through the convolution layer operation core based on the target parameter matrix corresponding to each convolution layer in the target convolution neural network to obtain a final feature map, and stores it in the preset storage space;

[0263] Among them, the elements in the target parameter matrix corresponding to each convolution layer in the target convolution neural network correspond to the convolution kernels in the convolution layer one by one; the convolution kernels in the convolution layer include: compressed convolution kernels and uncompressed convolution kernels, and the size of the compressed convolution kernels is smaller than that of the uncompressed convolution kernels; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolution layer, respectively; the target convolution neural network is: for each convolution layer in the initial convolution neural network, the specified part of the convolution kernels in the convolution layer is compressed to obtain the compressed convolution kernels, and the model is trained; the specified part of the convolution kernels is set based on the number of input channels and the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation kernel of the FPGA;

[0264] The feature map acquisition module 1603 is used to acquire the final feature map from the preset storage space.

[0265] Optionally, the elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network are divided into multiple element groups; the number of rows of the target parameter matrix is ​​the number of output channels of the convolution layer, and the number of columns of the target parameter matrix is ​​the number of input channels of the convolution layer; each element group contains M×N elements; each element group corresponds to a group of compressed convolution kernels or to a group of uncompressed convolution kernels; the convolution kernels of at least two element groups in the multiple element groups are of the same size, and the convolution kernels of at least two element groups are of different sizes; M is the input parallelism of the convolution layer operation kernel, and N is the output parallelism of the convolution layer operation kernel, so that the FPGA performs convolution processing in parallel through the convolution layer operation kernel based on the target parameter matrix corresponding to the convolution layer.

[0266] Optionally, the device further comprises:

[0267] A compression step size determination module is used to determine, for each convolution layer in the initial convolutional neural network, a compression step size for compressing the convolution layer based on the number of input channels and the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation kernel; wherein the compression step size is used to indicate: when compressing the convolution layer, the proportion of the convolution kernels in the convolution layer that need to be compressed;

[0268] A convolution kernel compression module is used to compress a specified part of the convolution kernels in the convolution layer into a target convolution kernel according to a determined compression step size, so as to obtain a candidate convolution neural network;

[0269] A training module, used to train the candidate convolutional neural network to obtain a target convolutional neural network;

[0270] The target parameter matrix generation module is used to generate a target parameter matrix corresponding to each convolutional layer in the target convolutional neural network.

[0271] Optionally, the compression step length determination module includes:

[0272] A first calculation submodule is used to calculate, for each convolutional layer in the initial convolutional neural network, a ratio of the number of input channels of the convolutional layer to the input parallelism of the convolutional layer operation core as a first ratio, and calculate a ratio of the number of output channels of the convolutional layer to the output parallelism of the convolutional layer operation core as a second ratio;

[0273] A compression step determination submodule is used to determine a first compression step and a second compression step when compressing the convolution layer based on the first ratio and the second ratio; wherein the first compression step is used to indicate: when compressing the convolution layer, the proportion of convolution kernels corresponding to each output channel that need to be compressed; the second compression step is used to indicate: when compressing the convolution layer, the proportion of convolution kernels corresponding to each input channel that need to be compressed.

[0274] Optionally, the compression step size determination submodule is specifically used to determine a first compression step size and a second compression step size when compressing the convolution layer based on the smaller value of the first ratio and the second ratio if the input parallelism of the convolution layer operation kernel is equal to the output parallelism;

[0275] If the input parallelism of the convolution layer operation kernel is not equal to the output parallelism, a first compression step size when compressing the convolution layer is determined based on the first ratio, and a second compression step size when compressing the convolution layer is determined based on the second ratio.

[0276] Optionally, the convolution kernel compression module is specifically used to divide the convolution kernels corresponding to each output channel in the convolution layer into multiple first convolution kernel groups on average according to the order of the corresponding input channels; wherein the number of convolution kernels contained in each first convolution kernel group is the first compression step size; for each first convolution kernel group, determine the other convolution kernels in the first convolution kernel group except the convolution kernel at the specified position as the first convolution kernel to be processed; compress the first convolution kernel to be processed in each first convolution kernel group into a target convolution kernel to obtain an alternative convolution neural network; or,

[0277] For each convolution kernel corresponding to each input channel in the convolution layer, they are evenly divided into multiple second convolution kernel groups according to the order of the corresponding output channels; wherein the number of convolution kernels contained in each second convolution kernel group is the second compression step size; for each second convolution kernel group, the other convolution kernels in the second convolution kernel group except the convolution kernel at the specified position are determined as the second convolution kernels to be processed; the second convolution kernels to be processed in each second convolution kernel group are compressed into the target convolution kernel to obtain an alternative convolution neural network.

[0278] Optionally, the target parameter matrix generation module includes:

[0279] An initial parameter matrix acquisition submodule is used to acquire an initial parameter matrix corresponding to each convolution layer in the target convolutional neural network; wherein the elements in the initial parameter matrix correspond one-to-one to the convolution kernels in the convolution layer; and the elements in the initial parameter matrix are arranged in the order of the input channels and output channels to which the corresponding convolution kernels belong;

[0280] The target parameter matrix generation submodule is used to adjust the positions of rows and columns in the initial parameter matrix based on the first compression step size and the second compression step size to obtain a target parameter matrix corresponding to the convolution layer.

[0281] Optionally, the target parameter matrix generation submodule includes:

[0282] A first element determination unit is used to determine, for each candidate row in the initial parameter matrix, an element corresponding to the uncompressed convolution kernel in the candidate row as the first element; wherein the candidate row includes: a row where the element corresponding to the uncompressed convolution kernel in the first column of the initial parameter matrix is ​​located;

[0283] A first moving unit is used to move the columns where each adjacent element of the first compression step length is located in the first elements to adjacent positions;

[0284] A second element determination unit is used to determine, for each candidate column in the initial parameter matrix after the move, an element corresponding to the uncompressed convolution kernel in the candidate column as a second element; wherein the candidate column includes: a column where the element corresponding to the uncompressed convolution kernel in the first row of the initial parameter matrix is ​​located;

[0285] The second moving unit is used to move the rows where each adjacent second compression step element in the second element is located to adjacent positions to obtain a target parameter matrix corresponding to the convolution layer.

[0286] Based on the same inventive concept, the present application also provides a convolutional neural network training device, see Fig.17 , Fig.17A structural diagram of a convolutional neural network training device provided in an embodiment of the present application, the device may include:

[0287] The compression step size determination module 1701 is used to determine, for each convolution layer in the initial convolutional neural network, a compression step size for compressing the convolution kernels in the convolution layer based on the number of input channels and the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation core of the field programmable logic gate array FPGA; wherein the compression step size is used to indicate: when compressing the convolution layer, the proportion of the convolution kernels that need to be compressed in the convolution layer;

[0288] A convolution kernel compression module 1702 is used to compress a specified part of the convolution kernels in the convolution layer into a target convolution kernel according to a determined compression step size, so as to obtain a candidate convolution neural network;

[0289] A training module 1703 is used to train the candidate convolutional neural network to obtain a target convolutional neural network;

[0290] The target parameter matrix generation module 1704 is used to generate a target parameter matrix corresponding to each convolutional layer in the target convolutional neural network; wherein the elements in the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network correspond one-to-one to the convolution kernel in the convolutional layer; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolutional layer, respectively.

[0291] Optionally, the elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network are divided into multiple element groups; the number of rows of the target parameter matrix is ​​the number of output channels of the convolution layer, and the number of columns of the target parameter matrix is ​​the number of input channels of the convolution layer; each element group contains M×N elements; each element group corresponds to a group of compressed convolution kernels or to a group of uncompressed convolution kernels; the convolution kernels of at least two element groups in the multiple element groups are of the same size, and the convolution kernels of at least two element groups are of different sizes; M is the input parallelism of the convolution layer operation kernel, and N is the output parallelism of the convolution layer operation kernel.

[0292] The present application also provides an electronic device, such as Fig.18 As shown, it includes a processor 1801, a communication interface 1802, a memory 1803, an FPGA 1805 and a communication bus 1804, wherein the processor 1801, the communication interface 1802, the memory 1803, and the FPGA 1805 communicate with each other through the communication bus 1804.

[0293] Memory 1803, used for storing computer programs;

[0294] The processor 1801 is used to implement the following steps when executing the program stored in the memory 1803:

[0295] Get the image to be processed;

[0296] The image to be processed is stored in a preset storage space, so that FPGA1805 obtains the image to be processed from the preset storage space, and processes the image to be processed through the convolution layer operation kernel based on the target parameter matrix corresponding to each convolution layer in the target convolutional neural network to obtain a final feature map, and stores it in the preset storage space;

[0297] Among them, the elements in the target parameter matrix corresponding to each convolution layer in the target convolution neural network correspond to the convolution kernels in the convolution layer one by one; the convolution kernels in the convolution layer include: compressed convolution kernels and uncompressed convolution kernels, and the size of the compressed convolution kernels is smaller than that of the uncompressed convolution kernels; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolution layer, respectively; the target convolution neural network is: for each convolution layer in the initial convolution neural network, the specified part of the convolution kernels in the convolution layer is compressed to obtain the compressed convolution kernels, and the model is trained; the specified part of the convolution kernels is set based on the number of input channels and the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation kernel of the FPGA1805;

[0298] The final feature map is obtained from the preset storage space.

[0299] FPGA1805 is used to perform the following steps:

[0300] Get the image to be processed from the preset storage space;

[0301] Through the convolution layer operation kernel, based on the target parameter matrix of each convolution layer in the target convolution neural network, the image to be processed is processed to obtain a final feature map;

[0302] The final feature map is stored in the preset storage space.

[0303] The present application also provides an electronic device, such as Fig.19 As shown, it includes a processor 1901, a communication interface 1902, a memory 1903 and a communication bus 1904, wherein the processor 1901, the communication interface 1902, and the memory 1903 communicate with each other through the communication bus 1904.

[0304] Memory 1903, used for storing computer programs;

[0305] The processor 1901 is used to implement the following steps when executing the program stored in the memory 1903:

[0306] For each convolutional layer in the initial convolutional neural network, based on the number of input channels and the number of output channels of the convolutional layer, and the input parallelism and output parallelism of the convolutional layer operation core of the field programmable gate array FPGA, determine the compression step size for compressing the convolutional kernels in the convolutional layer; wherein the compression step size is used to indicate: when compressing the convolutional layer, the proportion of the convolutional kernels that need to be compressed in the convolutional layer;

[0307] According to the determined compression step size, a portion of the convolution kernels specified in the convolution layer are compressed into target convolution kernels to obtain an alternative convolution neural network;

[0308] Training the candidate convolutional neural network to obtain a target convolutional neural network;

[0309] Generate a target parameter matrix corresponding to each convolutional layer in the target convolutional neural network; wherein the elements in the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network correspond one-to-one to the convolution kernels in the convolutional layer; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolutional layer, respectively.

[0310] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0311] The communication interface is used for communication between the above electronic device and other devices.

[0312] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0313] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0314] The embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the data processing method provided in the embodiment of the present application, or the convolutional neural network training method.

[0315] It should be noted that the above data processing method and other implementation methods of the convolutional neural network training method are the same as those of the aforementioned method embodiments and will not be repeated here.

[0316] The embodiment of the present application also provides another computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the data processing method or the convolutional neural network training method provided in the embodiment of the present application.

[0317] It should be noted that the above data processing method and other implementation methods of the convolutional neural network training method are the same as those of the aforementioned method embodiments and will not be repeated here.

[0318] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0319] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0320] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0321] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Get the image to be processed; The image to be processed is stored in a preset storage space, so that the field programmable gate array FPGA obtains the image to be processed from the preset storage space, and processes the image to be processed through a convolutional layer operation core based on a target parameter matrix corresponding to each convolutional layer in a target convolutional neural network to obtain a final feature map, and stores it in the preset storage space; Among them, the elements in the target parameter matrix corresponding to each convolution layer in the target convolution neural network correspond one-to-one to the convolution kernels in the convolution layer; the convolution kernels in the convolution layer include: compressed convolution kernels and uncompressed convolution kernels, and the size of the compressed convolution kernels is smaller than that of the uncompressed convolution kernels; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolution layer, respectively; the target convolution neural network is: for each convolution layer in the initial convolution neural network, the specified part of the convolution kernels in the convolution layer is compressed to obtain the compressed convolution kernels, and the model is trained; the specified part of the convolution kernels is set based on the number of input channels and the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation kernel of the FPGA; The final feature map is obtained from the preset storage space.

2. The method according to claim 1, characterized in that The elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network are divided into multiple element groups; the number of rows of the target parameter matrix is ​​the number of output channels of the convolution layer, and the number of columns of the target parameter matrix is ​​the number of input channels of the convolution layer; each element group contains M×N elements; each element group corresponds to a group of compressed convolution kernels or a group of uncompressed convolution kernels; the convolution kernels of at least two element groups in the multiple element groups are of the same size, and the convolution kernels of at least two element groups are of different sizes; M is the input parallelism of the convolution layer operation kernel, and N is the output parallelism of the convolution layer operation kernel, so that the FPGA performs convolution processing in parallel through the convolution layer operation kernel based on the target parameter matrix corresponding to the convolution layer.

3. The method according to claim 2, characterized in that The process of obtaining the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network includes: For each convolutional layer in the initial convolutional neural network, based on the number of input channels and the number of output channels of the convolutional layer, and the input parallelism and output parallelism of the convolutional layer operation kernel, determine the compression step size for compressing the convolutional layer; wherein the compression step size is used to indicate: when compressing the convolutional layer, the proportion of the convolutional kernels that need to be compressed in the convolutional layer; According to the determined compression step size, a portion of the convolution kernels specified in the convolution layer are compressed into target convolution kernels to obtain an alternative convolution neural network; Training the candidate convolutional neural network to obtain a target convolutional neural network; Generate a target parameter matrix corresponding to each convolutional layer in the target convolutional neural network.

4. The method according to claim 3, characterized in that For each convolutional layer in the initial convolutional neural network, based on the number of input channels and the number of output channels of the convolutional layer, and the input parallelism and output parallelism of the convolutional layer operation kernel, determining the compression step size for compressing the convolutional layer, including: For each convolutional layer in the initial convolutional neural network, calculating the ratio of the number of input channels of the convolutional layer to the input parallelism of the convolutional layer operation core as a first ratio, and calculating the ratio of the number of output channels of the convolutional layer to the output parallelism of the convolutional layer operation core as a second ratio; Based on the first ratio and the second ratio, determine a first compression step and a second compression step when compressing the convolution layer; wherein the first compression step is used to indicate: when compressing the convolution layer, the proportion of the convolution kernels corresponding to each output channel that need to be compressed; the second compression step is used to indicate: when compressing the convolution layer, the proportion of the convolution kernels corresponding to each input channel that need to be compressed.

5. The method according to claim 4, characterized in that The determining, based on the first ratio and the second ratio, a first compression step size and a second compression step size when compressing the convolution layer includes: If the input parallelism of the convolutional layer operation kernel is equal to the output parallelism, determining a first compression step size and a second compression step size when compressing the convolutional layer based on the smaller value of the first ratio and the second ratio; If the input parallelism of the convolution layer operation kernel is not equal to the output parallelism, a first compression step size when compressing the convolution layer is determined based on the first ratio, and a second compression step size when compressing the convolution layer is determined based on the second ratio.

6. The method according to claim 4, characterized in that The method of compressing a portion of the convolution kernels specified in the convolution layer into target convolution kernels according to the determined compression step length to obtain an alternative convolution neural network includes: For each convolution kernel corresponding to each output channel in the convolution layer, the convolution kernels are evenly divided into a plurality of first convolution kernel groups according to the order of the corresponding input channels; wherein the number of convolution kernels contained in each first convolution kernel group is the first compression step size; for each first convolution kernel group, the convolution kernels other than the convolution kernel at the specified position in the first convolution kernel group are determined as the first convolution kernels to be processed; the first convolution kernels to be processed in each first convolution kernel group are compressed into the target convolution kernel to obtain the candidate convolution neural network; or, For each convolution kernel corresponding to each input channel in the convolution layer, they are evenly divided into multiple second convolution kernel groups according to the order of the corresponding output channels; wherein the number of convolution kernels contained in each second convolution kernel group is the second compression step size; for each second convolution kernel group, the other convolution kernels in the second convolution kernel group except the convolution kernel at the specified position are determined as the second convolution kernels to be processed; the second convolution kernels to be processed in each second convolution kernel group are compressed into the target convolution kernel to obtain an alternative convolution neural network.

7. The method according to claim 6, characterized in that The generating a target parameter matrix corresponding to each convolutional layer in the target convolutional neural network includes: For each convolution layer in the target convolutional neural network, an initial parameter matrix corresponding to the convolution layer is obtained; wherein the elements in the initial parameter matrix correspond one-to-one to the convolution kernels in the convolution layer; and the elements in the initial parameter matrix are arranged in the order of the input channels and output channels to which the corresponding convolution kernels belong; Based on the first compression step size and the second compression step size, the positions of the rows and columns in the initial parameter matrix are adjusted to obtain a target parameter matrix corresponding to the convolution layer.

8. The method according to claim 7, characterized in that The adjusting the positions of rows and columns in the initial parameter matrix based on the first compression step size and the second compression step size to obtain a target parameter matrix corresponding to the convolution layer includes: For each candidate row in the initial parameter matrix, determine the element corresponding to the unreplaced convolution kernel in the candidate row as the first element; wherein the candidate row includes: the row where the element corresponding to the unreplaced convolution kernel in the first column of the initial parameter matrix is ​​located; Move the columns of each adjacent element of the first compression step length in the first element to adjacent positions; For each candidate column in the initial parameter matrix after the move, determine the element corresponding to the unreplaced convolution kernel in the candidate column as the second element; wherein the candidate column includes: the column where the element corresponding to the unreplaced convolution kernel in the first row of the initial parameter matrix is ​​located; Move each adjacent row of the second compression step length elements in the second element to an adjacent position to obtain a target parameter matrix corresponding to the convolution layer.

9. A data processing method, characterized in that: The method is applied to a field programmable gate array (FPGA), and the method comprises: Get the image to be processed from the preset storage space; Through the convolution layer operation kernel, based on the target parameter matrix of each convolution layer in the target convolution neural network, the image to be processed is processed to obtain a final feature map; Among them, the elements in the target parameter matrix corresponding to each convolution layer in the target convolution neural network correspond to the convolution kernels in the convolution layer one by one; the convolution kernels in the convolution layer include: compressed convolution kernels and uncompressed convolution kernels, and the size of the compressed convolution kernels is smaller than that of the uncompressed convolution kernels; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolution layer, respectively; the target convolution neural network is: for each convolution layer in the initial convolution neural network, the specified part of the convolution kernels in the convolution layer is compressed to obtain the compressed convolution kernels, and the model is trained; the specified part of the convolution kernels is set based on the number of input channels and the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation kernel of the FPGA; The final feature map is stored in the preset storage space.

10. The method according to claim 9, characterized in that The elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network are divided into multiple element groups; the number of rows of the target parameter matrix is ​​the number of output channels of the convolution layer, and the number of columns of the target parameter matrix is ​​the number of input channels of the convolution layer; each element group contains M×N elements; each element group corresponds to a group of compressed convolution kernels or a group of uncompressed convolution kernels; the convolution kernels of at least two element groups in the multiple element groups are of the same size, and the convolution kernels of at least two element groups are of different sizes; M is the input parallelism of the convolution layer operation kernel, and N is the output parallelism of the convolution layer operation kernel; The process of processing the image to be processed by the convolutional layer operation kernel based on the target parameter matrix of each convolutional layer in the target convolutional neural network to obtain a final feature map includes: Through the convolution layer operation kernel, based on the target parameter matrix of each convolution layer in the target convolutional neural network, the image to be processed is convolved in parallel to obtain a final feature map.

11. A convolutional neural network training method, characterized in that: The method comprises: For each convolutional layer in the initial convolutional neural network, based on the number of input channels and the number of output channels of the convolutional layer, and the input parallelism and output parallelism of the convolutional layer operation core of the field programmable gate array FPGA, determine the compression step size for compressing the convolutional kernels in the convolutional layer; wherein the compression step size is used to indicate: when compressing the convolutional layer, the proportion of the convolutional kernels that need to be compressed in the convolutional layer; According to the determined compression step size, a portion of the convolution kernels specified in the convolution layer are compressed into target convolution kernels to obtain an alternative convolution neural network; Training the candidate convolutional neural network to obtain a target convolutional neural network; Generate a target parameter matrix corresponding to each convolutional layer in the target convolutional neural network; wherein the elements in the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network correspond one-to-one to the convolution kernels in the convolutional layer; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolutional layer, respectively.

12. A field programmable gate array FPGA, characterized in that: The FPGA includes a double rate synchronous dynamic random access memory DDR controller, an advanced extensible interface AXI bus, a state controller, a feature map buffer, a parameter buffer, an output buffer, an adder, a read / write controller, and at least one convolutional layer operation core with an input parallelism of M and an output parallelism of N, wherein: The DDR controller is used to obtain the image to be processed from the preset storage space and store the image to be processed in the feature map buffer area; The state controller is used to control the read-write controller to read the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network from the parameter buffer area, read the candidate image from the feature map buffer area, and send it to the convolutional layer operation core; wherein the candidate image is a to-be-processed image or feature map stored in the feature map buffer area; the elements in the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network correspond one-to-one to the convolution kernel in the convolutional layer; the convolution kernel in the convolutional layer includes: a compressed convolution kernel and an uncompressed convolution kernel, and the compressed convolution The size of the kernel is smaller than the uncompressed convolution kernel; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolution layer, respectively; the target convolution neural network is: for each convolution layer in the initial convolution neural network, a specified part of the convolution kernel in the convolution layer is compressed to obtain a compressed convolution kernel, and the model is trained; the specified part of the convolution kernel is set based on the number of input channels and the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation kernel of the FPGA; The convolution layer operation core is used to perform convolution processing on the received candidate graph based on the target parameter matrix corresponding to each convolution layer according to a preset clock cycle; The state controller is further used to control the read-write controller, and for each convolution layer of the target convolutional neural network, the convolution calculation results of each clock cycle of the convolution layer operation kernel are stored in the output buffer area respectively; for each output channel of the convolution layer, the bias value of the output channel is read from the parameter buffer area, and the adder is controlled to accumulate the convolution calculation results of each clock cycle of the output channel stored in the output buffer area, and calculate the sum of the accumulated result and the bias value of the output channel to obtain the feature map corresponding to the output channel, and store it in the output buffer area; The DDR controller is also used to store the feature map obtained based on each target convolutional layer in the output buffer to the feature map buffer through the AXI bus; obtain the final feature map obtained based on the last convolutional layer in the target convolutional neural network from the output buffer, and store it in the preset storage space; wherein the target convolutional layer includes: other convolutional layers in the target convolutional neural network except the last convolutional layer.

13. The FPGA according to claim 12, characterized in that: The elements in the target parameter matrix corresponding to each convolution layer in the target convolutional neural network are divided into multiple element groups; the number of rows of the target parameter matrix is ​​the number of output channels of the convolution layer, and the number of columns of the target parameter matrix is ​​the number of input channels of the convolution layer; each element group contains M×N elements; each element group corresponds to a group of compressed convolution kernels or to a group of uncompressed convolution kernels; the convolution kernels of at least two element groups in the multiple element groups are of the same size, and the convolution kernels of at least two element groups are of different sizes; The convolution layer operation core is used to perform convolution processing on the received candidate images in parallel according to a preset clock cycle and based on the target parameter matrix corresponding to each convolution layer.

14. A data processing device, characterized in that: The device comprises: A module for obtaining images to be processed, used to obtain images to be processed; A storage module is used to store the image to be processed in a preset storage space, so that the field programmable gate array FPGA obtains the image to be processed from the preset storage space, and processes the image to be processed through a convolution layer operation core based on a target parameter matrix corresponding to each convolution layer in a target convolutional neural network to obtain a final feature map, and stores it in the preset storage space; Among them, the elements in the target parameter matrix corresponding to each convolution layer in the target convolution neural network correspond one-to-one to the convolution kernels in the convolution layer; the convolution kernels in the convolution layer include: compressed convolution kernels and uncompressed convolution kernels, and the size of the compressed convolution kernels is smaller than that of the uncompressed convolution kernels; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolution layer, respectively; the target convolution neural network is: for each convolution layer in the initial convolution neural network, the specified part of the convolution kernels in the convolution layer is compressed to obtain the compressed convolution kernels, and the model is trained; the specified part of the convolution kernels is set based on the number of input channels and the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation kernel of the FPGA; A feature map acquisition module is used to acquire the final feature map from the preset storage space.

15. A convolutional neural network training device, characterized in that: The device comprises: A compression step size determination module is used to determine, for each convolution layer in the initial convolutional neural network, a compression step size for compressing the convolution kernels in the convolution layer based on the number of input channels and the number of output channels of the convolution layer, and the input parallelism and output parallelism of the convolution layer operation core of the field programmable gate array FPGA; wherein the compression step size is used to indicate: when compressing the convolution layer, the proportion of the convolution kernels that need to be compressed in the convolution layer; A convolution kernel compression module is used to compress a specified part of the convolution kernels in the convolution layer into a target convolution kernel according to a determined compression step size, so as to obtain a candidate convolution neural network; A training module, used to train the candidate convolutional neural network to obtain a target convolutional neural network; A target parameter matrix generation module is used to generate a target parameter matrix corresponding to each convolutional layer in the target convolutional neural network; wherein the elements in the target parameter matrix corresponding to each convolutional layer in the target convolutional neural network correspond one-to-one to the convolution kernel in the convolutional layer; the number of rows and columns of the target parameter matrix are pre-set based on the number of output channels and the number of input channels of the convolutional layer, respectively.

16. An electronic device, characterized in that: It includes a processor, a communication interface, a memory, a field programmable gate array FPGA and a communication bus, wherein the processor, the communication interface, the FPGA and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method steps of any one of claims 1 to 8 when executing the program stored in the memory; The FPGA is used to execute the method steps described in any one of claims 9-10.

17. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method steps described in claim 11 when executing the program stored in the memory.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1-8 or 11 are implemented.

Images