A method for implementing CNN by FPGA with flexible resource configuration
By implementing a CNN neural network using FPGA with flexible resource configuration, optimizing the design of convolutional and hidden layers, the problem of long training time was solved, and the accuracy of pest quantity prediction and the running speed of the neural network were improved.
Patent Information
- Application Number
- CN202310748451.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-25
- Publication Date
- 2026-01-27
- Estimated Expiration
- 2043-06-25
AI Technical Summary
Existing neural network models require a large number of labeled training samples to evaluate the severity of plant pests, and the statistical analysis of pest numbers is time-consuming, leading to distorted statistical results and affecting the accuracy of the output.
A CNN neural network is implemented using FPGA with flexible resource configuration. By designing convolutional layers, hidden layers, and output layers, and using random walks to generate vertex network graphs, combined with convolution operation optimization, the training sample labeling time is reduced and the accuracy of pest quantity prediction is improved.
It reduces the time required to label training samples, improves the accuracy of pest quantity prediction, and enhances the running speed of neural network models by enabling efficient convolution processing through FPGA with flexible resource configuration.
Smart Images
Figure CN116543291B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of neural network model technology, and more specifically, to a method for implementing CNNs on an FPGA with flexible resource configuration. Background Technology
[0002] For some plants with relatively simple three-dimensional structures and leaves as the main body, neural network models can be used to evaluate the degree of pest infestation. The obstacles are that training a general neural network requires a large number of training samples with labeled pest numbers. On the other hand, manually counting the number of pests in the samples is time-consuming. Furthermore, pests are dynamic and active, and the time span of the statistics will lead to the distortion of the statistical results. Therefore, it is impossible to obtain sufficient training samples to train the neural network, resulting in poor accuracy of the pest count output by the neural network. Summary of the Invention
[0003] This invention provides a method for implementing CNN on an FPGA with flexible resource configuration, which solves the technical problem in related technologies that the training of neural networks for calculating the number of pests generally requires a large number of training samples labeled with the number of pests.
[0004] This invention provides a CNN neural network implemented on an FPGA with flexible resource configuration, comprising:
[0005] H concatenated convolutional layers, where the first convolutional layer takes a cell image as input, and multiple cell images are input sequentially; the feature map output by the last convolutional layer is linearly transformed to generate the first feature vector;
[0006] Generate an original vertex graph, where each vertex corresponds to a unit image; generate a vertex network graph for each vertex, and vectorize the vertices of the vertex network graph to obtain vertex vectors;
[0007] There are A first hidden layers, each receiving the first feature vector and vertex vector of a vertex in the vertex network graph as input. The h-th first hidden layer includes C channels. A random subset of the same size is generated by randomly sampling the vertices of the h-th layer of the vertex network graph. The first feature vector and vertex vector of the i-th random subset and the vertex at the center of the vertex network graph are input into the i-th channel. The calculation includes:
[0008] , This represents the i-th propagation vector of the h-th hidden layer. and These represent the weight parameters and bias parameters of the h-th hidden layer, respectively. This represents the activation function. This represents the i-th propagated information of vertex v at the h-th layer of the vertex network graph, where vertex v represents the vertex at the center of the vertex network graph.
[0009] ,in express The i-th random subset in This represents the set of vertices connected to vertex v at the h-th level of the vertex network graph, where vertex e belongs to... , The attention parameters represent the first feature vector corresponding to vertex v and the first feature vector corresponding to vertex e; This represents the vertex vector of vertex e.
[0010] The second hidden layer is formed by preprocessing the output of the first hidden layer before inputting it into the second hidden layer.
[0011] The calculation of the second hidden layer includes:
[0012] in, The graph encoding vector representing vertex v. The vector representing the y-th cluster center. Obtained by preprocessing the output of the first hidden layer. Indicates the number of cluster centers. and These represent the weight parameters and bias parameters of the second hidden layer, respectively. This represents the activation function. This represents the vertex vector corresponding to vertex v;
[0013] The output layer takes the graph encoding vector of the input vertex and searches for local maxima in the graph encoding vector. The number of local maxima is the number of pests in the region corresponding to the cell image of that vertex.
[0014] During training, the second hidden layer is connected to the fully connected layer. The fully connected layer is used to input the graph encoding vector of the vertex and the output is mapped to the classification space. The classification label in the classification space represents the degree of pest infestation.
[0015] Furthermore, in the plant image, vertices corresponding to adjacent unit images have connecting edges.
[0016] Furthermore, the formulas for calculating the attention parameters of the first feature vector corresponding to vertex v and the first feature vector corresponding to vertex e are as follows: , , Represents the original attention parameters. This represents the random subset of the vertex network graph to which vertex e belongs. This represents an exponential function with the natural constant as its base. and These represent the first eigenvectors corresponding to vertices v and e, respectively. This represents the scaling factor.
[0017] Furthermore, the preprocessing of the output of the first hidden layer includes: using the C propagation vectors output by the first first hidden layer as cluster centers, clustering the propagation vectors output by the other first hidden layers to generate C clusters, and calculating the vector of the cluster center for each cluster.
[0018] Furthermore, a random walk is performed on the original vertex graph to generate a vertex network graph for each vertex, with the number of layers in the vertex network graph being the same as the number of layers in the random walk.
[0019] Furthermore, methods for generating a vertex network graph for each vertex using random walks include:
[0020] Step 201: Select a vertex and start a random walk centered on the selected vertex until the number of layers reached is A; A is the number of the first hidden layers.
[0021] Step 202: Add the sequence of traversed vertices to the vertex network graph. If the number of traversals is less than B, increment the number of traversals by one and return to step 201; otherwise, end the step.
[0022] Furthermore, the classification labels in the classification space represent no pests, general pests, and severe pests, respectively.
[0023] Furthermore, the unit image is generated by uniformly segmenting the plant image.
[0024] This invention provides a method for implementing a CNN using a resource-flexible FPGA, which is used to perform convolution operations on the convolutional layers of a CNN neural network implemented using a resource-flexible FPGA, as described above, and includes the following steps:
[0025] Step 101: Before inputting the convolutional layer, first select the serial-parallel combination configuration, which includes the K value;
[0026] Step 102: Generate the serial step count, which is equal to "N / K", where N is the number of feature maps in the previous layer, N*M is the number of convolutional layers in the current layer, and M is the number of feature maps in the next layer.
[0027] Step 103: The K-1 convolutional computation structure computes the K feature maps of the previous layer in parallel. The K-1 convolutional computation structure is then merged serially through a serial merging structure in "N / K" steps to obtain the final result of the feature map in the next layer. This completes the convolutional computation of a total of N input feature maps in the previous layer.
[0028] Furthermore, the K-1 convolutional computation structure performs convolutional computations between K convolutional kernels and corresponding feature maps, and sums the values at corresponding points to obtain the values in the feature map. The convolutional operations between these K feature maps and K convolutional kernels are performed in parallel, and the output K convolutional values are summed at each clock cycle.
[0029] The serial merging structure accumulates the output of each K-1 convolution calculation. After each K-1 convolution calculation, K input feature maps are changed and the calculation continues. After N / K times, the convolution calculation of a total of N input feature maps in the previous layer is completed. The accumulated value is biased and then processed by the activation function f to obtain the final result of the feature map.
[0030] The beneficial effects of this invention are as follows: the labeling time of the training samples of the neural network model of this invention is short, and the information propagation of multiple unit images is integrated to generate the encoding vector of the corresponding unit image. The number of pests is indirectly output by utilizing the focus of the neural network on the features corresponding to pests during training and the vector. This ensures accuracy while reducing workload. At the same time, the FPGA with flexible resource configuration performs efficient convolution processing operations on the convolutional layer, ensuring the speed of the neural network model. Attached Figure Description
[0031] Figure 1 This is a flowchart of a method for implementing CNN on an FPGA with flexible resource configuration according to the present invention;
[0032] Figure 2 This is a flowchart of the method for generating a vertex network graph for each vertex using random walks according to the present invention. Detailed Implementation
[0033] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.
[0034] A CNN neural network implemented on an FPGA with flexible resource configuration, comprising:
[0035] There are H convolutional layers, where the first convolutional layer takes a cell image as input, and multiple cell images are input sequentially.
[0036] H convolutional layers are connected in series, and the feature map output by the last convolutional layer is defined as the final feature map. Each unit image corresponds to a final feature map.
[0037] The first linear layer takes the final feature map as input and performs a linear transformation on it to generate the first feature vector.
[0038] Generate the original vertex graph. Each vertex in the original vertex graph corresponds to a unit image. Vertices in adjacent unit images in the plant image are connected by edges.
[0039] A random walk is performed on the original vertex graph to generate a vertex network graph for each vertex. The starting point of the random walk is the center vertex of the vertex network graph. The number of layers in the vertex network graph is the same as the number of layers in the random walk. The vertices of the vertex network graph are vectorized to obtain vertex vectors.
[0040] A parallel first hidden layers, the first hidden layers are input to the first feature vector and vertex vector of the vertex network graph;
[0041] The h-th hidden layer includes C channels. A random subset of the same size is generated by randomly sampling the vertices of the h-th layer of the vertex network graph. The first feature vector and vertex vector of the i-th random subset and the center vertex of the vertex network graph are input into the i-th channel (each channel shares weight and bias parameters). The calculation includes:
[0042]
[0043] This represents the i-th propagation vector of the h-th hidden layer. and These represent the weight parameters and bias parameters of the h-th hidden layer, respectively. This represents the activation function. This represents the i-th propagated information of vertex v (the vertex at the center of the vertex network graph) at the h-th layer of the vertex network graph;
[0044]
[0045] in express The i-th random subset in This represents the set of vertices connected to vertex v at the h-th layer of the vertex network graph. The attention parameters represent the first feature vector corresponding to vertex v and the first feature vector corresponding to vertex e; This represents the vertex vector of vertex e.
[0046]
[0047]
[0048] Represents the original attention parameters. This represents the random subset of the vertex network graph to which vertex e belongs. This represents an exponential function with the natural constant as its base. and These represent the first eigenvectors corresponding to vertices v and e, respectively. This represents the scaling factor, an adjustable parameter with a default value of 0.2.
[0049] The second hidden layer is formed by preprocessing the output of the first hidden layer. This preprocessing includes:
[0050] The C propagation vectors output by the first hidden layer are used as cluster centers. The propagation vectors output by the other first hidden layers are clustered to generate C clusters. The vector of the cluster center is calculated for each cluster.
[0051] The calculation of the second hidden layer includes:
[0052]
[0053] in, The graph encoding vector representing vertex v. The vector representing the y-th cluster center. Indicates the number of cluster centers. and These represent the weight parameters and bias parameters of the second hidden layer, respectively. This represents the activation function. This represents the vertex vector corresponding to vertex v;
[0054] The output layer takes the graph encoding vector of the input vertex and searches for local maxima in the graph encoding vector. The number of local maxima is the number of pests in the region corresponding to the cell image of that vertex.
[0055] In one embodiment of the present invention, the CNN neural network implemented by the FPGA with flexible resource configuration described above connects the second hidden layer to a fully connected layer during training. The fully connected layer is used to input the graph encoding vectors of the vertices and maps the output to a classification space. The classification labels in the classification space represent no pests, moderate pests, and severe pests, respectively. Specifically, vertex tracing corresponds to a unit image, and the classification label actually represents the degree of pest infestation in the region corresponding to the unit image. For the training set, this category label can be manually labeled by examining the images based on experience, without needing to obtain specific information such as the number of pests in the unit image.
[0056] The algorithm used to find local maxima in the output layer is a conventional method, and the output layer can be connected after the other parts have been trained.
[0057] In one embodiment of the present invention, each vector component of the graph encoding vector is mapped to a two-dimensional coordinate system. For a vector component, its component value corresponds to the Y-axis coordinate, and its order needs to correspond to the X-axis coordinate. After fitting the curve, the peak value of the curve is found as a local maximum.
[0058] In one embodiment of the present invention, the dimension of the graph encoding vector of the vertex is the same as the number of elements of the matrix of the final feature map. The graph encoding vectors are cut into equal lengths and concatenated to form an intermediate matrix. The intermediate matrix has the same size as the matrix of the final feature map. The local maxima of the intermediate matrix are found as the local maxima of the graph encoding vector.
[0059] like Figure 2 As shown, in one embodiment of the present invention, the method for generating a vertex network graph for each vertex using a random walk includes:
[0060] Step 201: Select a vertex and start a random walk centered on the selected vertex until the number of layers reached is A; A is the number of the first hidden layers.
[0061] Step 202: Add the sequence of traversed vertices to the vertex network graph. If the number of traversals is less than B, increment the number of traversals by one and return to step 201; otherwise, end the step.
[0062] The vertex sequence is generated into a one-hot encoding, which is input into a continuous bag-of-words model and outputs a vectorized representation of the vertices, denoted as the vertex vector.
[0063] In one embodiment of the present invention, the unit image is generated by uniformly segmenting a plant image.
[0064] In one embodiment of the present invention, the plant image is a top-down view of the plant planting site, the plant is tobacco, etc., and the plant image is an RGB image. Therefore, each unit image is also an RGB image, and three channels of image feature input are generated respectively.
[0065] In one embodiment of the present invention, as with general convolution, pooling layers are provided between convolutional layers.
[0066] In one embodiment of the present invention, the activation function Choose the sigmod activation function.
[0067] In one embodiment of the present invention, the generation of the original vertex graph, the generation of a vertex network graph for each vertex, and the vectorization of the vertices of the vertex network graph to obtain vertex vectors are performed outside of a CNN neural network implemented on a resource-flexible FPGA.
[0068] like Figure 1 As shown, a method for implementing CNN on an FPGA with flexible resource configuration includes the following steps:
[0069] Step 101: Before inputting the convolutional layer, first select the serial-parallel combination configuration, which includes the K value;
[0070] 1≤K≤N, the value of K determines the resource overhead of parallel computing. When the FPGA has limited resources, the value of K can be set to a smaller value.
[0071] Step 102: Generate the serial step count, which is equal to "N / K", where N is the number of feature maps in the previous layer, N*M is the number of convolutional layers in the current layer, and M is the number of feature maps in the next layer.
[0072] Step 103: The K-1 convolutional computation structure computes the K feature maps of the previous layer in parallel. Then, through a serial merging structure, the K-1 convolutional computation structures are merged serially in "N / K" steps to obtain the final result for the feature map in the next layer. This completes the convolutional computation of a total of N input feature maps from the previous layer. Here, "" represents rounding up.
[0073] The K-1 convolutional computation structure is a convolutional computation architecture that computes K feature maps in the previous layer in parallel to obtain a single feature map in the next layer. It performs convolutional computations between K convolutional kernels and their corresponding feature maps, and sums the values at corresponding points to obtain the values in the single feature map. The convolutional operations of these K feature maps and K convolutional kernels are performed in parallel, and the output K convolutional values are summed at each clock cycle.
[0074] The serial merging structure accumulates the output of each K-1 convolution calculation. After each K-1 convolution calculation, K input feature maps are changed and the calculation continues. After N / K times, the convolution calculation of a total of N input feature maps in the previous layer is completed. The accumulated value is biased and then processed by an activation function f (such as Sigmoid) to obtain the final result of the feature map.
[0075] The feature map input to the first convolutional layer is the cell image.
[0076] The feature map information for each calculation can be either cached off-chip or stored on-chip. When there are many feature maps and the feature dimensions are large, the storage resources are consumed in a large way. It is necessary to cache off-chip after completing the calculation of one feature map. Otherwise, it can be stored on-chip directly to reduce transmission time and final computation latency.
[0077] The activation function, pooling, and fully connected layer calculations are performed in the normal manner. The method of this invention focuses on optimizing the most critical, resource-intensive, and performance-impacting inter-layer convolution operation.
[0078] The (L-1)th layer has N feature maps, the Lth layer has M feature maps, and the number of convolutional kernels in this layer is N*M. Each output feature map of the Lth layer is obtained by convolving all the (L-1)th feature maps with their corresponding convolutional kernels, summing the results, adding a bias, and then applying the activation function. The calculation formula is as follows:
[0079]
[0080] in This represents the j-th feature map information of the L-th layer. This represents the information of the i-th feature map in the (L-1)-th layer. This represents the convolution kernel with input number i and output number j. Represents convolution. Let f represent the bias of the j-th output feature map of layer L, and f represent the activation function. For ease of explanation later, assume N=4, M=3, and that this layer has 12 convolutional kernels, each with a size of 3x3.
[0081] According to the serial-parallel structure of the present invention, the K-1 convolution calculation structure is first executed, K input features of the L-1th layer are taken, convolved with K convolution kernels and added together to obtain a temporary value of the Lth output feature map, where 1≤K≤N, and K is assumed to be 2.
[0082] For the convolution operation between layers L-1 and L, the first feature map information of layer L-1 is taken first. and the second feature map information Perform a K-1 convolutional computation structure to obtain the first temporary feature map of the Lth layer. Perform on-chip caching; then retrieve the third feature map information from layer L-1. and the third feature map information Perform a K-1 convolutional computation structure to obtain the second temporary feature map of the Lth layer. ,Will and After accumulation, a bias is added, and the activation function f is applied to obtain the result of the first output feature map of layer L.
[0083] The value of K determines the resource overhead of parallel computation. When the FPGA has limited resources, the value of K can be set to a smaller value, up to K=1, to perform completely serial computation. In this case, the N feature maps of layer L-1 and the M feature maps of layer L would require N*M operations to complete all the convolution operations of that layer. When the FPGA has more resources, the value of K can be set to a larger value, up to N. In this case, the N feature maps of layer L-1 and the M feature maps of layer L would only require M operations to complete, improving the timeliness of computation.
[0084] For the CNN neural network implemented by FPGA with flexible resource configuration according to the present invention, since its input is a single unit image, the number of convolution operations will increase exponentially with the increase of the number of unit images. Therefore, the above method can improve the operating efficiency of the entire neural network by performing convolution.
[0085] The embodiments of this example have been described above. However, this example is not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms based on the guidance of this example, and all of them are within the protection scope of this example.
Claims
1. A CNN neural network device implemented on an FPGA with flexible resource configuration, characterized in that, include: H concatenated convolutional layers, where the first convolutional layer takes a cell image as input, and multiple cell images are input sequentially; the feature map output by the last convolutional layer is linearly transformed to generate the first feature vector; Generate an original vertex graph, where each vertex corresponds to a unit image; generate a vertex network graph for each vertex, and vectorize the vertices of the vertex network graph to obtain vertex vectors; There are A first hidden layers, each receiving the first feature vector and vertex vector of a vertex in the vertex network graph as input. The h-th first hidden layer includes C channels. A random subset of the same size is generated by randomly sampling the vertices of the h-th layer of the vertex network graph. The first feature vector and vertex vector of the i-th random subset and the vertex at the center of the vertex network graph are input into the i-th channel. The calculation includes: , This represents the i-th propagation vector of the h-th hidden layer. and These represent the weight parameters and bias parameters of the h-th hidden layer, respectively. This represents the activation function. This represents the i-th propagated information of vertex v at the h-th layer of the vertex network graph, where vertex v represents the vertex at the center of the vertex network graph. ,in express The i-th random subset in This represents the set of vertices connected to vertex v at the h-th level of the vertex network graph, where vertex e belongs to... , The attention parameters represent the first feature vector corresponding to vertex v and the first feature vector corresponding to vertex e; The vertex vector representing vertex e; The second hidden layer is formed by preprocessing the output of the first hidden layer before inputting it into the second hidden layer. The calculation of the second hidden layer includes: ,in, The graph encoding vector representing vertex v. The vector representing the y-th cluster center. Obtained by preprocessing the output of the first hidden layer. Indicates the number of cluster centers. and These represent the weight parameters and bias parameters of the second hidden layer, respectively. This represents the activation function. This represents the vertex vector corresponding to vertex v; The output layer takes the graph encoding vector of the input vertex and searches for local maxima in the graph encoding vector. The number of local maxima is the number of pests in the region corresponding to the cell image of that vertex. During training, the second hidden layer is connected to the fully connected layer. The fully connected layer is used to input the graph encoding vector of the vertex and the output is mapped to the classification space. The classification label in the classification space represents the degree of pest infestation.
2. The CNN neural network device implemented in an FPGA with flexible resource configuration according to claim 1, characterized in that, In the plant image, vertices corresponding to adjacent unit images are connected by edges.
3. The CNN neural network device implemented in an FPGA with flexible resource configuration according to claim 1, characterized in that, The formulas for calculating the attention parameters of the first eigenvector corresponding to vertex v and the first eigenvector corresponding to vertex e are as follows: , , Represents the original attention parameters. This represents the random subset of the vertex network graph to which vertex e belongs. This represents an exponential function with the natural constant as its base. and These represent the first eigenvectors corresponding to vertices v and e, respectively. This represents the scaling factor.
4. The CNN neural network device implemented in an FPGA with flexible resource configuration according to claim 1, characterized in that, The preprocessing of the output of the first hidden layer includes: using the C propagation vectors output by the first hidden layer as cluster centers, clustering the propagation vectors output by the other first hidden layers to generate C clusters, and calculating the cluster center vector for each cluster.
5. A CNN neural network device implemented in an FPGA with flexible resource configuration according to claim 1, characterized in that, A random walk is performed on the original vertex graph to generate a vertex network graph for each vertex. The number of layers in the vertex network graph is the same as the number of layers in the random walk.
6. A CNN neural network device implemented in an FPGA with flexible resource configuration according to claim 5, characterized in that, Methods for generating a vertex network graph for each vertex using random walks include: Step 201: Select a vertex and start a random walk centered on the selected vertex until the number of layers reached is A; A is the number of the first hidden layers. Step 202: Add the sequence of traversed vertices to the vertex network graph. If the number of traversals is less than B, increment the number of traversals by one and return to step 201; otherwise, end the step.
7. A CNN neural network device implemented in an FPGA with flexible resource configuration according to claim 1, characterized in that, The category labels in the category space represent no pests, moderate pests, and severe pests, respectively.
8. A CNN neural network device implemented in an FPGA with flexible resource configuration according to claim 1, characterized in that, The unit image is generated by uniformly segmenting the plant image.
9. A method for implementing CNN on an FPGA with flexible resource configuration, characterized in that, It is used to perform convolution operations on the convolutional layers of a CNN neural network device implemented in an FPGA with flexible resource configuration as described in any one of claims 1-8, and includes the following steps: Step 101: Before inputting the convolutional layer, first select the serial-parallel combination configuration, which includes the K value; Step 102: Generate the serial step count, which is equal to "N / K", where N is the number of feature maps in the previous layer, N*M is the number of convolutional layers in the current layer, and M is the number of feature maps in the next layer. Step 103: The K-1 convolutional computation structure computes the K feature maps of the previous layer in parallel. The K-1 convolutional computation structure is then merged serially through a serial merging structure in "N / K" steps to obtain the final result of the feature map in the next layer. This completes the convolutional computation of a total of N input feature maps in the previous layer.
10. A method for implementing CNN on an FPGA with flexible resource configuration according to claim 9, characterized in that, The K-1 convolutional computation structure performs convolutional computations on K convolutional kernels and their corresponding feature maps, and sums the values at corresponding points to obtain the values in the feature map. The convolutional operations of these K feature maps and K convolutional kernels are performed in parallel, and the output K convolutional values are summed at each clock cycle. The serial merging structure accumulates the output of each K-1 convolution calculation. After each K-1 convolution calculation, K input feature maps are changed and the calculation continues. After N / K times, the convolution calculation of a total of N input feature maps in the previous layer is completed. The accumulated value is biased and then processed by the activation function f to obtain the final result of the feature map.
Citation Information
Patent Citations
Plant leaf pest identification method based on improved convolutional neural network
CN111563431A
Lightweight apple leaf disease identification method based on knowledge distillation
CN113610163A