An FPGA-based image classification method and system
By transforming convolution operations into matrix operations for parallel computation on FPGAs, the problems of deployment difficulty and high computational load of deep neural networks on FPGAs are solved, thereby improving operating efficiency and reducing power consumption.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING INST OF INTELLIGENT TECH INST OF MICROELECTRONICS OF THE CHINESE ACAD OF
- Filing Date
- 2022-12-29
- Publication Date
- 2026-05-26
AI Technical Summary
Deep neural networks are difficult to deploy on FPGAs, have a high computational load, and low operating efficiency.
By transforming convolution operations into matrix operations through the unfolding rule, a sign bit matrix and its bit operations are introduced, and multiple vector operation units are used to compute vector operation groups in parallel. The system is designed as a structured system with data layer, operation layer and network layer.
It improves the operating efficiency of the FPGA, reduces the difficulty of deploying deep neural networks, and reduces power consumption.
Smart Images
Figure CN115797704B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an image classification method and system based on FPGA, belonging to the field of image classification technology. Background Technology
[0002] Image classification is one of the fundamental tasks of machine vision, and deep neural network models are widely used due to their fast classification speed and high accuracy. The high accuracy of deep neural network models often relies on a large amount of computation. Commonly used operations include convolution, pooling, and activation. The parameters of the units that implement these operations are generally stored in four-dimensional tensors.
[0003] The massive computational demands and complex data structures of deep neural networks necessitate the support of highly available memory and high computing power from the target platform, which increases the difficulty of deployment on FPGAs and significantly reduces operating efficiency. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide an image classification method and system based on FPGA, which can make full use of hardware resources, linearly reduce the computational load of the basic convolution module, and achieve the goal of improving operating efficiency.
[0005] To achieve the above objectives, the present invention is implemented using the following technical solution:
[0006] On one hand, the present invention provides an FPGA-based image classification method, comprising the following steps:
[0007] S1. Initialize the system: Read the required parameters from the configuration file, set the system parameters, and establish an unweighted deep neural network model through the model definition configuration file.
[0008] S2. Load Model Weights: Read the model weights from the configuration file, generate the weight matrix for each convolutional unit according to the unfolding rule, collect the sign bit of each element in the weight matrix to form the weight sign bit matrix for each convolutional unit, and arrange the weight matrix and weight sign bit matrix for each convolutional unit according to the deep neural network model structure.
[0009] S3. Acquire image data: Read the image data, preprocess it to obtain a data matrix, collect the sign bits of each element in the data matrix, form a data sign bit matrix, and input the data matrix and the data sign bit matrix into the deep neural network model.
[0010] S4. Generate the feature matrix: Perform an XOR operation on the obtained data sign bit matrix and weight sign bit matrix to generate the output sign bit matrix, and then determine the vector operation group of the data matrix and weight matrix; use multiple vector operation units to calculate the vector operation group in parallel, and fill the results into the corresponding positions of the output sign bit matrix to obtain the feature matrix;
[0011] The feature matrix is used as the input to the next level of convolutional unit, and the operation process is repeated until the output layer of the deep neural network model is reached.
[0012] S5. Display Results: Reshape the feature matrix to obtain a one-dimensional feature vector, generate a vector operation group according to the matrix multiplication rules, use multiple vector operation units to calculate the elements in the vector operation group in parallel, collect the calculation results according to the index order, and obtain the output vector; obtain the position marker of the maximum value in the output vector, look up the table to obtain the classification result, and display the result.
[0013] Furthermore, the deep neural network model consists of a backbone network and an output layer, wherein the output layer is composed of fully connected layers.
[0014] Furthermore, the weights of the output layer are directly read from the configuration file, while the weights of each convolutional unit in the backbone network need to be expanded using an expansion rule after being read from the configuration file. The expansion rule is as follows:
[0015]
[0016] Among them, C ou C in These are the number of output channels and the number of input channels of the convolutional unit, respectively, while K... h K w These are the height and width of the convolution kernel, respectively.
[0017] Furthermore, the image data in step S3 is read from the acquisition device or the settings file, and the preprocessing includes gamma transformation, normalization operation and image unfolding to obtain a data matrix;
[0018] The dimensions of the data matrix are:
[0019]
[0020] Where C is the number of channels in the image, H and W are the height and width of the image, respectively, and N is calculated using the following formula:
[0021]
[0022] Where S is the stride of the convolutional unit, and Kh and Kw are the height and width of the convolutional kernel.
[0023] Furthermore, the calculation process of the vector operation group in step S4 is as follows: traverse the output sign bit matrix. If the value of the element is all binary 0, convert it to the numerical value "0". If the value of the element contains binary 1, determine multiple vector operations according to the matrix operation rules to obtain the vector operation group.
[0024] On the other hand, the present invention also provides an FPGA-based image classification system, including an image acquisition device, an image classification device, and a storage medium, wherein the storage medium is used to store instructions, the image acquisition device is used to acquire images, and the image classification device is used to operate according to the instructions to execute the steps of any of the methods described above.
[0025] Optionally, the image classification device includes a data layer, a computation layer, and a network layer from bottom to top. The data layer is used to provide the basic data structure and data storage model. The computation layer includes a data unrolling module, a matrix XOR operation module, and a matrix multiplication operation module. The network layer includes a convolutional layer, a fully connected layer, and a basic convolutional module.
[0026] Optionally, the data structure includes one-dimensional vectors, two-dimensional matrices, and four-dimensional tensors.
[0027] Optionally, the data expansion module is used to expand the input four-dimensional tensor into a one-dimensional vector, and then reshape it into a two-dimensional matrix according to the expansion rule;
[0028] The matrix XOR operation module implements the XOR operation of two matrix elements through a logic gate array.
[0029] Optionally, the matrix multiplication operation module includes a matrix XOR operation module, a vector group generator, and an arithmetic unit containing multiple vector operation units;
[0030] The data sign bit matrix and the weight sign bit matrix are used as inputs to the matrix XOR operation module to obtain the data matrix and the weight matrix. The data matrix and the weight matrix are used together as inputs to the vector operation group generator to obtain a vector operation group containing multiple vector operations. Based on the number of elements in the vector operation group, multiple vector operation units are called to perform parallel calculations.
[0031] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:
[0032] The image classification method provided by this invention transforms convolution operations into matrix operations through an unfolding rule; by introducing a sign bit matrix and its bit operations, the path of matrix operations is determined, generating vector operation groups. Multiple vector operation units are used to compute the vector operation groups in parallel to fully utilize hardware resources and improve operational efficiency.
[0033] The image classification system provided by this invention adopts a structured design approach, dividing the system from bottom to top into a data layer, a computation layer, and a network layer. By reusing modules and providing a unified interface in the network layer, the difficulty of deploying deep neural networks on FPGAs is effectively reduced. The proposed matrix multiplication module and its method can linearly reduce the computational load of the basic convolution module, thereby achieving the goal of reducing power consumption. Attached Figure Description
[0034] Figure 1 This is a flowchart illustrating an FPGA-based image classification method in one embodiment of the present invention.
[0035] Figure 2 This is a schematic diagram of a data storage model for an FPGA-based image classification system in one embodiment of the present invention;
[0036] Figure 3 This is a schematic diagram of the convolutional unit calculation process of an FPGA-based image classification method in one embodiment of the present invention;
[0037] Figure 4 This is a schematic diagram of the matrix multiplication operation module of an FPGA-based image classification system in one embodiment of the present invention. Detailed Implementation
[0038] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.
[0039] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are used only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.
[0040] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0041] Example 1:
[0042] like Figure 1 As shown, this embodiment of the invention provides an image classification method based on FPGA, including the following steps:
[0043] Step S1: Initialize the system: Read the required parameters from the configuration file, set the system parameters, and establish an unweighted deep neural network model through the model definition configuration file. The deep neural network model consists of a backbone network and an output layer, and the output layer consists of fully connected layers.
[0044] Step S2: Load model weights: The model weights are obtained from training on the server. The model is preprocessed by merging the weights of the convolutional layer and the batch normalization layer on the server side to improve the speed of forward inference of the model.
[0045] The weights are extracted to a configuration file for the system to read. The model weights are read from the configuration file. The weights of the output layer can be directly read from the configuration file, while the weights of each convolutional unit in the backbone network need to be expanded using an expansion rule after being read from the configuration file.
[0046] The expansion rule is:
[0047]
[0048] Among them, C ou C in These are the number of output channels and the number of input channels of the convolutional unit, respectively, while K... h K w These are the height and width of the convolution kernel, respectively.
[0049] The weight matrix of each convolutional unit is generated according to the unfolding rule. The weight matrix and weight sign matrix of each convolutional unit are arranged according to the deep neural network model structure in step S1. The elements of the weight matrix are selected as follows: Figure 2 The data model shown is stored, and the sign bit of each element in the weight matrix is collected to form the weight sign bit matrix of each convolution unit.
[0050] Step S3: Acquire image data: Read image data from the acquisition device or settings file, preprocess it to obtain a data matrix. Preprocessing methods can include gamma transformation, normalization, and image unfolding to obtain the data matrix. The dimensions of the data matrix are:
[0051]
[0052] Where C is the number of channels in the image, H and W are the height and width of the image, respectively, and N is calculated using the following formula:
[0053]
[0054] Where S is the stride of the convolutional unit, and K... h K w It represents the height and width of the convolution kernel.
[0055] Collect the sign bits of each element in the data matrix to form a data sign bit matrix, and input the data matrix and the data sign bit matrix into the deep neural network model.
[0056] Step S4, Generating the Feature Matrix: The specific calculation process of the convolutional unit is as follows... Figure 3 As shown,
[0057] Step 1: Obtain the convolution input by inputting the data matrix and the data sign bit matrix into the deep neural network model;
[0058] Step 2: Calculate the output sign bit matrix, and perform an XOR operation on the obtained data matrix and the data sign bit matrix to generate the output sign bit matrix;
[0059] Specifically, the sign bit matrix W of the weights in the input layer of the backbone network s With data sign bit matrix I s Perform an XOR operation to generate the output sign bit matrix O. s :
[0060]
[0061] For example:
[0062]
[0063] Step 3: Generate vector operation groups, traverse and output the sign bit matrix, and determine the vector operation groups according to the matrix multiplication formula;
[0064] The matrix multiplication formula is as follows:
[0065]
[0066] Specifically, if the value of an element is all binary 0, it is converted to the numerical value "0" and the calculation ends; if the value of an element contains binary 1 (or is equal to binary 1), then according to the matrix operation rules, multiple vector operations are determined to obtain a vector operation group.
[0067] For example:
[0068]
[0069] in:
[0070]
[0071] Step 4: Calculate the feature matrix, use multiple vector operation units to calculate the elements in the vector operation group in parallel, fill the results into the corresponding positions of the output sign bit matrix (the subscripts of each element in the vector operation group), and obtain the feature matrix;
[0072] The obtained feature matrix is used as the input to the next level of convolutional unit, and the above process is repeated until the next layer is the output layer of the deep neural network model.
[0073] Step S5: Display Results: Reshape the input feature matrix to obtain a one-dimensional feature vector. Generate a vector operation group according to the matrix multiplication rules. Use multiple vector operation units to calculate the elements in the vector operation group in parallel. Collect the calculation results according to the index order to form the output vector.
[0074] Obtain the position marker of the maximum value in the output vector, look up the classification result in the table, and display the result.
[0075] This is a method for image classification using a deep neural network model on an FPGA. The method transforms convolution operations into matrix operations through an unfolding rule. By introducing a sign bit matrix and its bitwise operations (XOR operation), the path of matrix operations is determined, a vector operation group is generated, and multiple vector operation units are used to compute the vector operation group in parallel to make full use of hardware resources and improve operating efficiency.
[0076] Example 2:
[0077] This embodiment provides an FPGA-based image classification system, including an image acquisition device, an image classification device, and a storage medium.
[0078] Image acquisition devices refer to ordinary color cameras. Image classification devices employ a structured design approach, which can be divided into a data layer, a computation layer, and a network layer from bottom to top.
[0079] The data layer provides basic data structures and data storage models. Data structures include one-dimensional vectors, two-dimensional matrices, and four-dimensional tensors. Vector structures are implemented using one-dimensional arrays, two-dimensional matrix structures are implemented through nested vectors, and four-dimensional tensor structures are implemented through nested two-dimensional matrices. The data storage model is used to store data in the system, such as... Figure 2 As shown.
[0080] The computation layer includes a data expansion module, a matrix XOR operation module, and a matrix multiplication operation module. The data expansion module expands the input four-dimensional tensor into a one-dimensional vector, and then reshapes it into a two-dimensional matrix according to the expansion rule. The matrix XOR operation module uses a logic gate array to implement the XOR operation between the elements of two matrices, as shown in the following formula:
[0081]
[0082] The matrix multiplication module consists of a matrix XOR operation module, a vector group generator, and an arithmetic unit containing multiple vector operation units. The input to the matrix XOR operation module is a data matrix and a weight matrix; its output is an output sign bit matrix. This output sign bit matrix, along with the data matrix and weight matrix, serves as the input to the vector operation group generator, which outputs a vector operation group containing multiple vector operations. Then, based on the number of elements in the vector operation group, multiple vector operation units are invoked for parallel computation, achieving acceleration.
[0083] The matrix multiplication module can linearly reduce the computational cost of the basic convolution module, as follows:
[0084] Assumption:
[0085] The dimension of matrix A is m×n, and the sign bit matrix is As;
[0086] The dimension of matrix B is n×km×n, and the sign bit matrix is Bs;
[0087] Therefore, the dimension of the output matrix C is m×k.
[0088] The computational workload is as follows: m×n×k multiplications and m×(n-1)×k additions.
[0089] Input As and Bs into the XOR operation module to obtain the output sign bit matrix Cs (dimension is m×k); in Cs, the number of elements with the value of binary 0 is N.
[0090] So, when performing matrix operations,
[0091] Computational complexity: m×n×(kN) multiplications, m×(n-1)×(kN) additions.
[0092] thereby:
[0093]
[0094] That is, by introducing a sign bit matrix, the amount of computation can be reduced linearly.
[0095] The network layer includes convolutional layers, fully connected layers, and basic convolutional modules. Based on the structural characteristics of each unit in the deep neural network model, the network layer module is encapsulated by combining different components in the computational layer and data layer, and a unified interface is provided.
[0096] The storage medium can be a removable memory card, and the configuration file is stored on the storage medium. During network deployment, the system reads the weights of the deep neural network model from the storage medium, expands the weights to obtain a weight matrix and a weight sign matrix, and sorts them according to the connection order of each module.
[0097] An image classification system employs a structured design approach. The system is divided into a bottom-up structure: a data layer, a computation layer, and a network layer. By reusing modules and providing a unified interface in the network layer, the difficulty of deploying deep neural networks on FPGAs is effectively reduced. The proposed matrix multiplication module and its method can linearly reduce the computational load of the basic convolution module, achieving the goal of reducing power consumption.
[0098] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. An FPGA-based image classification method, characterized in that: Includes the following steps, S1. Initialize the system: Read the required parameters from the configuration file, set the system parameters, and establish an unweighted deep neural network model through the model definition configuration file. S2. Load Model Weights: Read the model weights from the configuration file, generate the weight matrix for each convolutional unit according to the unfolding rule, collect the sign bit of each element in the weight matrix to form the weight sign bit matrix for each convolutional unit, and arrange the weight matrix and weight sign bit matrix for each convolutional unit according to the deep neural network model structure. S3. Acquire image data: Read the image data, preprocess it to obtain a data matrix, collect the sign bits of each element in the data matrix, form a data sign bit matrix, and input the data matrix and the data sign bit matrix into the deep neural network model. S4. Generate feature matrix: Perform an XOR operation on the obtained data sign bit matrix and weight sign bit matrix to generate an output sign bit matrix, and then determine the vector operation group of the data matrix and weight matrix. The vector operation group is computed in parallel using multiple vector operation units, and the results are filled into the corresponding positions of the output sign bit matrix to obtain the feature matrix; The feature matrix is used as the input to the next level of convolutional unit, and the operation process is repeated until the output layer of the deep neural network model is reached. S5. Display Results: Reshape the feature matrix to obtain a one-dimensional feature vector, generate a vector operation group according to the matrix multiplication rules, use multiple vector operation units to calculate the elements in the vector operation group in parallel, collect the calculation results according to the index order, and obtain the output vector; obtain the position marker of the maximum value in the output vector, look up the table to obtain the classification result, and display the result.
2. The FPGA-based image classification method of claim 1, wherein: The deep neural network model consists of a backbone network and an output layer, and the output layer is composed of fully connected layers.
3. The image classification method based on FPGA according to claim 2, characterized in that: The weights of the output layer are directly read from the configuration file. The weights of each convolutional unit in the backbone network need to be expanded using an expansion rule after being read from the configuration file. The expansion rule is as follows: ； in, C ou ，C in These are the number of output channels and the number of input channels of the convolutional unit, respectively. K h ，K w These are the height and width of the convolution kernel, respectively.
4. The image classification method based on FPGA according to claim 1, characterized in that: The image data in step S3 is read from the acquisition device or the settings file. The preprocessing includes gamma transformation, normalization operation and image unfolding to obtain a data matrix. The dimensions of the data matrix are: ； in, C It is the number of channels in the image. H,W These are the height and width of the image, respectively. N The following formula is used to calculate: ； in, S It is the stride of the convolutional unit. K h ，K w It represents the height and width of the convolution kernel.
5. The image classification method based on FPGA according to claim 1, characterized in that: The calculation process of the vector operation group in step S4 is as follows: traverse the output sign bit matrix. If the value of the element is all binary 0, convert it to the numerical value "0". If the value of the element contains binary 1, determine multiple vector operations according to the matrix operation rules to obtain the vector operation group.
6. An FPGA-based image classification system, characterized in that: The method includes an image acquisition device, an image classification device, and a storage medium. The storage medium is used to store instructions. The image acquisition device is used to acquire images. The image classification device is used to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 5.
7. The FPGA-based image classification system according to claim 6, characterized in that: The image classification device comprises, from bottom to top, a data layer, a computation layer, and a network layer. The data layer provides the basic data structure and data storage model. The computation layer includes a data unrolling module, a matrix XOR operation module, and a matrix multiplication operation module. The network layer includes a convolutional layer, a fully connected layer, and a basic convolutional module.
8. The FPGA-based image classification system according to claim 7, characterized in that: The basic data structure includes one-dimensional vectors, two-dimensional matrices, and four-dimensional tensors.
9. The FPGA-based image classification system according to claim 7, characterized in that: The data expansion module is used to expand the input four-dimensional tensor into a one-dimensional vector, and then reshape it into a two-dimensional matrix according to the expansion rule; The matrix XOR operation module implements the XOR operation of two matrix elements through a logic gate array.
10. An FPGA-based image classification system according to claim 7, characterized in that: The matrix multiplication operation module includes a matrix XOR operation module, a vector group generator, and an arithmetic unit containing multiple vector operation units. The data sign bit matrix and the weight sign bit matrix are used as inputs to the matrix XOR operation module to obtain the data matrix and the weight matrix. The data matrix and the weight matrix are used together as inputs to the vector operation group generator to obtain a vector operation group containing multiple vector operations. Based on the number of elements in the vector operation group, multiple vector operation units are called to perform parallel calculations.