A method and device for image classification based on FPGA embedded system

Abstract

The present invention relates to an FPGA-based embedded image classification method and device, comprising: an image acquisition module for acquiring images via a camera; an image recognition and classification module for identifying and classifying images and transmitting the results to an image display module; and an image display module for triggering the image acquisition module to control its start and stop, and receiving and displaying the results of the image recognition and classification module. The FPGA-based embedded image classification method and device described in the present invention utilizes the image acquisition module, the image recognition and classification module, and the image display module to construct an FPGA-based embedded image classification device capable of acquiring image data in real time and accurately identifying and classifying the acquired image data. Furthermore, the image acquisition effect can be displayed in real time on a display, and a button can be triggered to control the opening and closing of the image acquisition module and image capture.

Description

Technical Field

[0001] The present invention belongs to the field of civil aviation transportation industry, specifically to a method and device for classifying images based on FPGA, which is applied to the real-time classification and counting of passengers in the aviation transportation industry. Background Art

[0002] my country's civil aviation industry has experienced rapid growth in recent years. With the rapid advancement of science and technology, significant progress has been made in communications, navigation, meteorology, and air traffic control systems. The technical foundation for ensuring flight safety has become increasingly solid, and human error is now playing an increasingly important role in flight accidents. Verifying the presence of passengers, crew, and attendants in the cabin before closing the aircraft's doors is a crucial pre-takeoff safety check.

[0003] Therefore, how to accurately identify and count passengers is a very important issue.

[0004] The confirmation of the number of passengers involved in the present invention currently mainly uses manual counting to verify passengers. However, this traditional method has major safety hazards and shortcomings, mainly manifested in that manual counting involves certain human errors. The counter-based method of counting passengers is still a manual counting method, which is subject to human errors. Once an error occurs in the passenger inspection, it will cause a serious error, and the crew and security personnel will be severely punished; there may even be criminals who take advantage of loopholes to threaten flight safety, causing significant losses to people's lives and property safety. The airport boarding security check reconfirmation system based on facial recognition is used in the security check process. Due to its non-portability, it cannot be used in the cabin to make the final confirmation of the identity and number of passengers. Summary of the Invention

[0005] Based on the technical problems raised above, an FPGA-based embedded image classification method and device are proposed to solve the errors existing in traditional manual statistics, assist crew members in verifying passenger identities, effectively avoid errors in manual statistics, and achieve accurate identification and counting of cabin crew members; at the same time, it overcomes the non-portability problem of other identification devices.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: an FPGA-based embedded image classification device, comprising:

[0007] An image acquisition module, used for acquiring images through a camera;

[0008] Image recognition and classification module, used to recognize and classify images and send the results to the image display module;

[0009] The image display module is used to trigger the image acquisition module to control the start and stop of the image acquisition module, and receive the results of the image recognition and classification module for display.

[0010] The image captured by the image acquisition module is saved in the DDR through three-frame buffer, and then the image data is read from the DDR, and then the image display module is driven by the HDMI IP core to display the read image.

[0011] The image recognition and classification module is used to identify and classify the input image through the neural network AlexNet model to obtain the probability of the image being classified.

[0012] The image recognition and classification module is used to optimize the matrix calculations involved in the neural network AlexNet model using BLAS to improve image recognition and classification performance.

[0013] The image recognition and classification module is implemented through FPGA.

[0014] An FPGA-based embedded image classification method includes the following steps:

[0015] The image acquisition module collects multiple frames of images through a monocular camera;

[0016] The image display module displays the acquired image in real time and triggers the image acquisition module to start working; the image display module sends a control signal to the image acquisition module, captures a frame of image in the image acquisition module and saves it in the SD card; the image display module sends a stop signal to stop the image acquisition module; after the acquisition is completed, the image format is converted into the set format image;

[0017] The image recognition and classification module performs image recognition and classification based on the image data output by the image acquisition module, and outputs the current recognition and classification results.

[0018] The image recognition and classification module uses the neural network AlexNet model to identify and classify the input image to obtain the probability of the image classification, including the following steps:

[0019] After the image is input into the AlexNet model, the first convolution layer calculates the feature map, and then performs local response normalization and maximum pooling on the feature map; the second convolution layer performs convolution processing on the normalized feature map and the feature map after maximum pooling respectively, and outputs two sets of feature maps; the third convolution layer combines the two sets of feature maps into one group, and uses the convolution kernel to perform convolution calculation on the combined feature map; then the fourth and fifth convolution layers use different numbers of convolution kernels to perform convolution operations on the feature map, and then perform maximum pooling; the feature map after maximum pooling passes through the sixth, seventh, and eighth fully connected layers in turn to output features and send them to softmax, and finally softmax outputs the probability of classification.

[0020] Optimizing the matrix calculation in the convolutional layer includes the following steps:

[0021] Matrix A represents the input image, matrix B represents the convolution kernel, and matrix C represents the output feature map;

[0022] First, loop through the rows of matrix C, then loop through the columns of matrix C, and update matrix C with the inner product of the i-th row of A and the j-th column of B in each loop;

[0023] Then, expand the columns of matrix C by 4 and encapsulate the expanded content into an independent function. The order of looping matrix C is reversed, that is, first looping over the columns of matrix C by increasing number 4, and then looping over the rows of matrix C by increasing number 1. In each loop, matrix C is updated with the inner product of the i-th row of A and the j-th column of B.

[0024] Finally, register variables are applied to perform multiplication and addition operations on matrices A, B, and C inside the independent function. Pointer indexes are used for continuous movement operations to avoid calculating the coordinates of matrix B each time. Memory access is reused, vectorized instructions are used, etc., and matrices A and B are packaged and copied to a continuous space.

[0025] In the image recognition and classification module, the image to be recognized and classified is obtained through the image acquisition module, and the image to be recognized and classified is input into the pre-trained neural network AlexNet model to obtain an image recognition and classification module for real-time image recognition and classification.

[0026] The image acquisition module obtains images of a known data set, and the neural network AlexNet model is trained in advance to obtain the neural network AlexNet model for real-time image recognition and classification.

[0027] The present invention has the following beneficial effects and advantages:

[0028] 1. This paper optimizes the matrix calculations involved in the convolutional layer, reduces the time consumption of image recognition and classification while maintaining the same accuracy, uses a lightweight framework and model for image classification, and greatly saves embedded device storage resources.

[0029] 2. The FPGA-based embedded image classification method and device described in this invention utilizes an image acquisition module, an image recognition and classification module, and an image display module to construct an FPGA-based embedded image classification device capable of real-time image data acquisition, accurate recognition, and classification. Simultaneously, the image acquisition results can be displayed in real time on a display, and a trigger button can be used to control the opening and closing of the image acquisition module and image capture. This FPGA-based embedded image classification method and device can assist flight crew members in final confirmation of the identity and number of passengers, with classification performance exceeding that of manual methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of the structural principle of the image classification device of the present invention;

[0031] Figure 2 Schematic diagram of the structure of the image acquisition module of the present invention;

[0032] Figure 3 Schematic diagram of the image display module of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0034] A method and device for image classification based on FPGA embedded system, comprising:

[0035] Image acquisition module, image recognition and classification module, and image display module. The image acquisition module includes a monocular camera configured on the ZYNQ-7000 series development board according to the SCCB protocol to acquire images at a rate of 30 frames per second. A single frame of image is captured by pressing a button and saved in BMP format to the SD card, while converting the image into JPG format. The image classification module recognizes and classifies images stored on the SD card. By optimizing the matrix calculations in the convolutional neural network, the time consumption for image recognition and classification is reduced while maintaining the same accuracy. The image display module displays the images captured by the camera in real time, and the three buttons "Open", "Close", and "Capture" are used to control the opening and closing of the camera acquisition module and image capture. By adjusting the camera, high-quality and clear images can be obtained.

[0036] The image acquisition module includes a monocular camera, which is installed in the ZYNQ-7000 series development board interface.

[0037] The image recognition and classification module includes Linux system porting, compilation of the DarkNet lightweight framework, use of the AlexNet model for image recognition and classification, and optimization of matrix calculations in the convolutional layer of the model.

[0038] The image display module includes an image display unit, opening, closing and capturing function buttons, and sends control signals to the image acquisition module via the function buttons to control the opening, closing and capturing of the image acquisition module.

[0039] The classification mainly includes three processes: Linux porting, DarkNet lightweight framework compilation, and matrix calculation optimization.

[0040] The FPGA embedded image classification method is characterized by comprising the following steps:

[0041] The image acquisition module is installed on the corresponding interface of the ZYNQ-7000 series development board. The camera acquisition mode is configured to capture 30 frames per second through the SCCB protocol, and the camera is adjusted to the optimal acquisition state.

[0042] The image display unit in the image display module displays the captured image in real time. Click the "Open" button in the image display module, and the image display module sends a control signal to the image acquisition module. At this time, the image acquisition module starts to work normally. Click the "Capture" button in the image display module, and the image display module sends a control signal to the image acquisition module. At this time, a frame of image in the image acquisition module is captured and saved in BMP format in the SD card. Click the "Close" button in the image display module, and the image display module sends a control signal to the image acquisition module. At this time, the image acquisition module stops working.

[0043] After the acquisition is completed, the saved BMP format images need to be converted into JPG format images to ensure that the image recognition and classification modules are not affected by the BMP image format;

[0044] The image recognition and classification module combines the image data output by the image acquisition module to perform image recognition and classification, and outputs the current recognition and classification results;

[0045] Repeat the above steps until all collected images are identified and classified.

[0046] The image recognition step includes:

[0047] The image is processed through random inactivation, local response normalization, maximum pooling, and ReLU activation function, and multiple convolutional layers are combined to effectively extract features in the image and match them with the dataset to identify the image category.

[0048] The image classification is divided into the following three types:

[0049] Linux system porting includes: kernel configuration and compilation, uboot configuration and compilation, creating a Linux image, and DarkNet lightweight framework installation and compilation.

[0050] Matrix calculation optimization in the convolution layer includes: matrix multiplication GEMM optimization.

[0051] The classification results include: calculating the Top-5 error rate of the image data output by the image acquisition module of the image recognition and the time consumed by the classification.

[0052] A method and device for image classification based on FPGA embedded system, such as Figure 1 As shown, it includes: an image acquisition module, an image recognition and classification module, and an image display module. The image acquisition module includes: configuring a monocular camera in the ZYNQ-7000 series development board according to the SCCB protocol to acquire images at a rate of 30 frames per second. By pressing a button, a frame of image is captured and saved in BMP format to the SD card, and the image is converted to JPG format. The image classification module recognizes and classifies the images stored in the SD card. By optimizing the matrix calculations in the convolutional neural network, the time consumption of image recognition and classification is reduced while maintaining the same accuracy. The image display module displays the images captured by the camera in real time, and the three buttons "On", "Off", and "Capture" are used to control the opening and closing of the camera acquisition module and the capture of images. By adjusting the camera, high-quality and clear images can be obtained. The device can acquire images in real time and display them on the display end in real time. It also performs real-time format conversion of the acquired image data and performs image recognition and classification.

[0053] The image acquisition module includes the ZYNQ-7000 series development board and a monocular camera. Its structure is as follows: Figure 2 As shown in the figure, install the camera into the port connected to the ZYNQ-7000 series development board and the SD card into the corresponding port. Connect the power cord and start the development board. The camera operates by reading and writing to the custom camera IP core using the SCCB protocol and configuring the camera parameters to capture images. When the image display unit in the image display module displays image data, adjust the camera to capture a clear image. The clarity of the captured image can be observed on the image display unit.

[0054] The image display module includes an image display unit and three control buttons: image acquisition on, off, and capture. The image display function is implemented on an HDMI display, which is connected to the corresponding HDMI interface of the ZYNQ-7000 series development board via an HDMI data cable. The structure is as follows Figure 3As shown in the figure, the image acquisition module captures image data through three frame buffers and saves it to DDR. The image data is then read from DDR and driven by a custom HDMI IP core to display the read image on the monitor. Image display and saving are controlled by the Open and Capture control buttons. The specific operation process can be achieved by following the steps below.

[0055] Step 1: Test in the cabin environment. First, click the "Open" button. The image display module sends a control signal to the image acquisition module. The image display unit now displays the cabin environment image captured by the camera. Rotate the camera to adjust the focus until the image display unit clearly displays the captured cabin environment image.

[0056] Step 2: Click the "Capture" button. The Image Display Module sends a control signal to the Image Acquisition Module. A frame of the cabin environment is captured from the Image Display Unit and saved to the SD card in BMP format. A format conversion command converts the BMP image to a JPG format and saves it to the dataset path used for image classification.

[0057] After the image is saved, the cabin environment image is recognized and classified in the image recognition and classification module, serving as the input image for the AlexNet model. In the first convolution layer, 48 convolution kernels of 11*11*3 are applied to the input image, resulting in 48 feature maps of 55*55. Following the convolution, the feature maps are subjected to local response normalization (LRN) and max pooling with a stride of 2 and a pooling window of 3*3, outputting feature maps of 27*27. The second convolution layer uses 128 convolution kernels of 5*5*48 across two channels to convolve the two input feature maps, outputting two sets of 128 feature maps of 27*27. Similarly, the output feature maps are subjected to local response normalization (LRN) and max pooling with a stride of 2 and a pooling window of 3*3, outputting feature maps of 13*13. In the third convolution layer, the two feature maps are combined into one group and convolved with 192 3*3*256 kernels. Another convolution layer with the same number of kernels is then used to convolve the input feature maps, outputting two sets of 192 13*13 feature maps. In the fourth convolution layer, the two input feature maps are convolved with 192 3*3*192 kernels. In the fifth convolution layer, the two input feature maps are convolved with 128 3*3*192 kernels. After the fifth convolution, max pooling with a stride of 2 and a pooling window of 3*3 is performed, outputting a 6*6 feature map. The sixth and seventh fully connected layers each have two groups of 2048 neurons. The eighth fully connected layer outputs 1000 features, which are fed into a softmax filter. The softmax filter outputs the probability of cabin occupants being present in the cabin environment image.

[0058] The steps of ordinary matrix operation in the convolutional layer are as follows:

[0059] Step 1: Circulate the rows of matrix C, starting from the first row of matrix C where i is equal to 0 and incrementing by 1 to m;

[0060] Step 2: Circulate the columns of matrix C, starting from the first column j of matrix C and incrementing from 1 to n;

[0061] Step 3: Update the matrix C(i,j) with the inner product of the i-th row of matrix A and the j-th column of matrix B, C(i,j) = C(i,j) + A(i,p)*B(p,j); that is, loop with p equal to 0, increasing by 1 to k, starting from the first element of the i-th row of the matrix and the first element of the j-th column of the matrix, and ending with the multiplication of the last element of the i-th row of the matrix and the last element of the j-th column of the matrix.

[0062] The open-source BLAS (Basic Linear Algebra Subprograms) is called to optimize matrix calculations. The idea is to swap the order of the i-th row and j-th column to be traversed in matrix multiplication, expand the columns of matrix C by 4, and loop with a step length of i+4. The third step is encapsulated into an independent function, and the calculation order is swapped. The function is optimized using register variables rather than memory operations to reduce cache overhead. Then, pointer indexing is used to perform continuous move operations on the memory access of matrix B to avoid calculating the coordinates of matrix B each time. Memory access is reused, vectorized instructions are used, etc., and matrices A and B are packaged and copied to a continuous space.

[0063] Experiments have shown that this FPGA-based embedded image classification device, utilizing an image acquisition module, an image recognition and classification module, and an image display module, can perform real-time image acquisition and display, and can also control image display and storage through signals. By converting the image format and saving it in the image classification dataset path, using the image recognition and classification module to identify and classify the image, and optimizing the matrix calculations in the convolution operation using an acceleration library, classification time is reduced by approximately five times while maintaining the same classification accuracy. This increases the potential for real-time performance in embedded environments.

[0064] The FPGA-based embedded image classification method and device described in this invention utilizes an image acquisition module, an image recognition and classification module, and an image display module to construct a complete FPGA-based embedded image classification device and method. Crew members can carry this device to collect real-time information about the cabin environment. The image display module can also display the cabin environment in real time. Control buttons are used to capture images of the cabin environment, perform image classification, and count passengers, assisting crew members in verifying passengers.

[0065] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should be regarded as within the scope of protection of the present invention.

Claims

1. An FPGA-based embedded image classification device, characterized in that: include: An image acquisition module, used for acquiring images through a camera; Image recognition and classification module, used to recognize and classify images and send the results to the image display module; The image display module is used to trigger the image acquisition module to control the start and stop of the image acquisition module, and receive the results of the image recognition and classification module for display; The image captured by the image acquisition module is saved in the DDR through the three-frame buffer, and then the image data is read from the DDR, and then the image display module is driven by the HDMI IP core to display the read image; The image recognition and classification module is used to optimize the matrix calculations involved in the neural network AlexNet model using BLAS to improve image recognition and classification performance; The image recognition and classification module uses the neural network AlexNet model to identify and classify the input image to obtain the probability of the image classification, including the following steps: After the image is input into the AlexNet model, the first convolution layer calculates the feature map, and then performs local response normalization and maximum pooling on the feature map; the second convolution layer performs convolution processing on the normalized feature map and the feature map after maximum pooling respectively, and outputs two sets of feature maps; the third convolution layer combines the two sets of feature maps into one group, and uses the convolution kernel to perform convolution calculation on the combined feature map; then the fourth and fifth convolution layers sequentially perform convolution operations on the feature map using different numbers of convolution kernels, and then perform maximum pooling; the feature map after maximum pooling is sequentially output by the sixth, seventh, and eighth fully connected layers and sent to the softmax, and finally the softmax outputs the probability of classification; Optimizing the matrix calculation in the convolutional layer includes the following steps: Matrix A represents the input image, matrix B represents the convolution kernel, and matrix C represents the output feature map; First, loop through the rows of matrix C, then loop through the columns of matrix C, and update matrix C with the inner product of the i-th row of A and the j-th column of B in each loop; Then, expand the columns of matrix C by 4 and encapsulate the expanded content into an independent function. The order of looping matrix C is reversed, that is, first looping over the columns of matrix C by increasing number 4, and then looping over the rows of matrix C by increasing number 1. In each loop, matrix C is updated with the inner product of the i-th row of A and the j-th column of B. Finally, register variables are allocated to perform multiplication and addition operations on matrices A, B, and C inside the independent function. Pointer indexes are used for continuous movement operations to avoid calculating the coordinates of matrix B each time. Memory access is reused, vectorized instructions are used, and matrices A and B are packaged and copied to a continuous space.

2. The FPGA-based embedded image classification device according to claim 1, characterized in that: The image recognition and classification module is used to identify and classify the input image through the neural network AlexNet model to obtain the probability of the image being classified.

3. The FPGA-based embedded image classification device according to claim 1, characterized in that: The image recognition and classification module is implemented through FPGA.

4. A FPGA-based embedded image classification method, characterized in that The following steps are involved: The image acquisition module collects multiple frames of images through a monocular camera; The image display module displays the collected image in real time and triggers the image acquisition module to start working; The image display module sends a control signal to the image acquisition module to capture a frame of image in the image acquisition module and save it in the SD card; the image display module sends a stop signal to stop the image acquisition module; After the acquisition is completed, the image format is converted into the set format image; The image recognition and classification module performs image recognition and classification based on the image data output by the image acquisition module, and outputs the current recognition and classification results; The image recognition and classification module uses the neural network AlexNet model to identify and classify the input image to obtain the probability of the image classification, including the following steps: After the image is input into the AlexNet model, the first convolution layer calculates the feature map, and then performs local response normalization and maximum pooling on the feature map; the second convolution layer performs convolution processing on the normalized feature map and the feature map after maximum pooling respectively, and outputs two sets of feature maps; the third convolution layer combines the two sets of feature maps into one group, and uses the convolution kernel to perform convolution calculation on the combined feature map; then the fourth and fifth convolution layers sequentially perform convolution operations on the feature map using different numbers of convolution kernels, and then perform maximum pooling; the feature map after maximum pooling is sequentially output by the sixth, seventh, and eighth fully connected layers and sent to the softmax, and finally the softmax outputs the probability of classification; Optimizing the matrix calculation in the convolutional layer includes the following steps: Matrix A represents the input image, matrix B represents the convolution kernel, and matrix C represents the output feature map; First, loop through the rows of matrix C, then loop through the columns of matrix C, and update matrix C with the inner product of the i-th row of A and the j-th column of B in each loop; Then, expand the columns of matrix C by 4 and encapsulate the expanded content into an independent function. The order of looping matrix C is reversed, that is, first looping over the columns of matrix C by increasing number 4, and then looping over the rows of matrix C by increasing number 1. In each loop, matrix C is updated with the inner product of the i-th row of A and the j-th column of B. Finally, register variables are allocated to perform multiplication and addition operations on matrices A, B, and C inside the independent function. Pointer indexes are used for continuous movement operations to avoid calculating the coordinates of matrix B each time. Memory access is reused, vectorized instructions are used, and matrices A and B are packaged and copied to a continuous space. In the image recognition and classification module, the image to be recognized and classified is obtained through the image acquisition module, and the image to be recognized and classified is input into the pre-trained neural network AlexNet model to obtain an image recognition and classification module for real-time image recognition and classification; The image acquisition module obtains images of a known data set, and the neural network AlexNet model is trained in advance to obtain the neural network AlexNet model for real-time image recognition and classification.

Images