Image classification system based on GY-Net network

Abstract

The invention discloses an image classification system based on a GY-Net network. The system comprises a CMOS camera, a ZYNQ processing system, a DDR4 memory and an HDMI display. Classification of different types of animals is realized through combination of a hardware circuit and a deep learning algorithm; firstly, a ZYNQ processing system undertakes an initialization task, activates a CMOS camera, prompts the CMOS camera to continuously generate video data, and accurately and efficiently collects real-time data output by the CMOS camera; then, rapidly transmitting the collected video data to a GY-Net network, and carrying out precise classification on the video data by utilizing the strong real-time processing capability of the GY-Net network; and finally, a result obtained after GY-Net network processing and original video data are displayed on an HDMI display in a real-time, visual and clear picture. According to the invention, the classification results of different animals can be accurately obtained in different environments, so that the original image and the classification results can be efficiently displayed in real time.

Description

Technical Field

[0001] The invention belongs to the field of image classification, and in particular relates to an image classification system based on a GY-Net network. Background Art

[0002] In today's digital age, image data is growing explosively, and image classification technology has crucial applications in many fields. From medical imaging diagnosis, helping doctors quickly identify lesions and determine disease types, to security monitoring, real-time identification of abnormal personnel and behaviors, and even the classification and management of product images on e-commerce platforms, all rely on efficient and accurate image classification systems.

[0003] Through patent search, it was found that the authorization announcement number: CN106778775B discloses an image classification method based on SIFT feature soft matching, which mainly includes the following steps: using the SIFT feature extraction algorithm to extract features for each image to be classified; soft matching the extracted SIFT features between every two images; defining a similarity metric based on the results of matching between images; and using the similarity between the above images as the input of a subsequent classification algorithm for classification.

[0004] Authorization announcement number: CN115099294B discloses a flower image classification algorithm based on feature enhancement and decision fusion, which belongs to the field of image classification technology. It includes data preprocessing stage, feature extraction and enhancement stage, training stage and decision fusion classification stage. In the data preprocessing stage, the data set is subjected to corresponding cropping, scaling, normalization and other operations; in the feature extraction and enhancement stage, the preprocessed data is sent to the VGG16 model pre-trained by ImageNet to extract multi-layer deep image features, and a feature enhancement strategy is introduced to adaptively allocate feature weights; in the training stage, multiple sets of features from the previous stage are used to train multiple custom softmax classifiers; in the decision fusion classification stage, information entropy is introduced to represent the degree of certainty of each classifier, and the fusion weight is determined based on the information entropy, and the fusion decision is used to achieve classification.

[0005] Traditional image classification methods, such as those based on manual feature extraction, such as SIFT (Scale Invariant Feature Transform) and HOG (Histogram of Oriented Gradients), have played a certain role in the early days, but their limitations are becoming increasingly apparent in the face of complex and ever-changing real-life image scenes. These manually designed features are difficult to capture the rich and deep semantic information in the image, and have poor adaptability to lighting changes, scale scaling, object deformation, etc., which often leads to a significant reduction in classification accuracy.

[0006] With the rise of deep learning technology, convolutional neural networks (CNN) have become the mainstream means of image classification. Classic CNN architectures such as AlexNet and VGGNet have promoted the development of technology. They automatically learn image features through multi-layer convolution and pooling. However, as the requirements for classification accuracy continue to rise, problems are gradually exposed. On the one hand, deep networks are prone to gradient vanishing or gradient explosion, making it difficult for model training to converge to an ideal state; on the other hand, a large number of parameters lead to a sharp increase in computing costs and extremely high requirements for hardware resources, which limits its deployment and application in FPGAs. Moreover, when processing small samples and imbalanced data, traditional CNNs have insufficient generalization capabilities and are prone to overfitting, which cannot meet the actual complex and changing business needs. Summary of the invention

[0007] The present invention aims to solve the deficiencies of the above-mentioned prior art and proposes an image classification system based on the GY-Net network, so as to accurately obtain the classification results of different animals in different environments, so as to display the original image and classification results in real time and efficiently.

[0008] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0009] The image classification system based on the GY-Net network of the present invention is characterized in that it includes: a CMOS camera, a ZYNQ processing system, a DDR4 memory, and an HDMI display;

[0010] The ZYNQ processing system module includes: a PL logic design module and a PS control design module;

[0011] The PL logic design module includes: a video acquisition module, a video to video stream module, a VDMA module, a GY-Net network, a video stream to video module, a timing control module, and a display driver module;

[0012] The PS control design module includes: an IIC control unit, an ARM control unit, and a DDR4 storage control unit;

[0013] The IIC control unit, under the initialization and driving of the ARM control unit, establishes a communication connection with the CMOS camera through the IIC interface and configures the parameters of the CMOS camera, including: image resolution, refresh rate, and output timing parameters;

[0014] The video acquisition module acquires image data segments in RGB565 format output by the configured CMOS camera to obtain complete image data in RGB888 format;

[0015] The video to video stream module converts the image data in RGB888 format into image data in AXI-Stream format;

[0016] The VDMA module obtains the image data in the AXI-Stream format, and under the drive of the ARM control unit, converts it into intermediate data in the RGB888 format, and then stores it in the DDR4 memory through the DDR4 storage control unit;

[0017] The GY-Net network, under the control of the ARM control unit, reads and processes the intermediate data in RGB888 format in the DDR4 memory through the DDR4 storage control unit, and outputs the predicted label data corresponding to the image data;

[0018] The ARM control unit splices the Chinese font corresponding to the predicted label data and the intermediate data in RGB888 format into an image in RGB888 format and its classification result data, and stores them in the DDR4 memory through the DDR4 storage control unit;

[0019] The VDMA module, driven by the ARM control unit, reads the image and its classification result data in RGB888 format from the DDR4 memory, and converts them into the image and its classification result data in AXI-Stream format;

[0020] The video stream to video module converts the image in the AXI-Stream format and the classification result data thereof back to the image in the RGB888 format and the classification result data thereof;

[0021] The timing control module matches the image in RGB888 format and its classification result data with the corresponding output resolution, refresh rate and timing parameters, and transmits them to the display driving module;

[0022] The display driving module encodes the image in RGB888 format with output resolution, refresh rate and timing parameters and the classification result data thereof, and converts them into image data in HDMI format, and then displays them on the HDMI display.

[0023] The image classification system based on the GY-Net network described in the present invention is characterized in that the video acquisition module acquires image data segments in the RGB565 format according to the following steps:

[0024] Step 1, define a 1-bit counter cnt and initialize it to 0, the image data fragment in RGB565 format includes: an 8-bit image data signal pdata and a control signal for controlling pdata; the control signal for controlling pdata includes: a clock signal pclk, a field signal pvsync, and a line signal phsync;

[0025] Step 2, in the clock domain of pclk, with pclk as the reference unit, after pvsync experiences a high level of several clock cycles of the clock signal pclk, pvsync becomes a low level, and when phsync is still at a high level, pdata, pvsync and phsync are simultaneously delayed by a clock cycle of the clock signal pclk, thereby obtaining a new 8-bit image data signal r_pdata, a new field signal r_pvsync and a new line signal r_phsync;

[0026] Step 3, when pvsync is low and phsync is high, cnt starts counting. After cnt counts to two data, that is, when cnt=1, the new 8-bit image data signal r_pdata and the 8-bit image data signal pdata are bit-joined to obtain a 16-bit image data signal r_fdata.

[0027] Step 4, the 16-bit image data signal r_fdata is delayed by two clock cycles of pclk under the action of counter cnt, and at the same time, r_pvsync and r_phsync are also delayed by two clock cycles of pclk respectively to obtain an updated field signal rr_pvsync and an updated line signal rr_phsync;

[0028] Step 5, after dividing the clock signal pclk by two, an output clock signal fclk is obtained;

[0029] Step 6, in the clock domain of fclk, with fclk as the reference unit, delay rr_pvsync, rr_phsync, and r_fdata by one clock cycle of fclk to obtain the output field signal fvsync, the output line signal fhsync, and the updated 16-bit image data signal rr_fdata;

[0030] Step 7: Use equation (1) to convert the updated 16-bit image data signal rr_fdata into a 24-bit image data signal fdata:

[0031] (1)

[0032] In formula (1), express 19-23bit data, express 16-18bit data, express 10-15bit data, express 8-9bit data, express 3-7bit data, express 0-2bit data;

[0033] Step 8: Output the 24-bit image data signal fdata, the output field signal fvsync, the output line signal fhsync and the output clock signal fclk as complete image data in RGB888 format.

[0034] Furthermore, the GY-Net network includes: N convolutional layers and fully connected layers;

[0035] Any n+1th convolutional layer in the GY-Net module converts the feature map A output by the nth convolutional layer into n Process and output the feature map A of the n+1th convolutional layer n+1 :

[0036] (2)

[0037] In formula (2), W n represents the convolution kernel weight of the nth convolution layer, B n represents the bias of the nth convolutional layer, represents the scale factor alignment parameter of the nth convolutional layer, ⊙ represents the convolution operation, and clip represents the truncation operation. Indicates a shift operation;

[0038] The fully connected layer of the GY-Net module uses formula (3) to obtain the category probability vector , and output the category corresponding to the maximum probability as the predicted label data:

[0039] (3)

[0040] In formula (3), Represents the feature map of the Nth convolutional layer, represents the weight of the fully connected layer, represents the bias of the fully connected layer, Represents the scale factor alignment parameter of the fully connected layer.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. The present invention is implemented based on the processing system of the mobile device ZYNQ, that is, the ARM+FPGA architecture, which adopts the technology of ARM controlling data flow and FPGA parallel processing of data, thereby reducing system power consumption and improving system work efficiency.

[0043] 2. The present invention adopts a method for quantizing convolutional neural network parameters, realizes the fusion of convolutional layers and batch normalization layers, quantizes the floating-point numbers of traditional convolutional neural networks into fixed-point integers, overcomes the problem of large-bit-width floating-point numbers appearing in the ZYNQ system, and reduces the space size of parameter storage in the ZYNQ system.

[0044] 3. The present invention adopts 8-bit fixed-point integer multiplication operation in the convolution multiplication process, which avoids the floating-point multiplication in the hardware, reduces the amount of calculation, improves the data processing speed, and improves the working efficiency of the system.

[0045] 4. The present invention adopts 32-bit fixed-point integer addition operation in the convolution bias addition process, which avoids the floating-point addition in the hardware, reduces the amount of calculation, improves the data processing speed, and improves the working efficiency of the system.

[0046] 5. The present invention adopts shift operation in the convolution process, which avoids the floating point division in the hardware, reduces the amount of calculation, increases the data processing speed, and improves the working efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a framework diagram of the overall system structure of the present invention;

[0048] Figure 2 This is a diagram of the ZYNQ processing system architecture in an implementation example of the present invention;

[0049] Figure 3 It is a GY-Net network processing flow chart before fixed-point quantization in the present invention;

[0050] Figure 4 This is a GY-Net network processing flow chart used by the ZYNQ processing system of the present invention; DETAILED DESCRIPTION

[0051] In this embodiment, Figure 1 As shown, an image classification system based on a GY-Net network includes: a CMOS camera, a ZYNQ processing system, a DDR4 memory, and an HDMI display;

[0052] like Figure 2 As shown, the ZYNQ processing system module includes: PL logic design module and PS control design module;

[0053] The PL logic design module includes: video acquisition module, video to video stream module, VDMA module, GY-Net network, video stream to video module, timing control module, and display driver module;

[0054] The PS control design module includes: IIC control unit, ARM control unit, and DDR4 storage control unit.

[0055] Under the initialization and drive of the ARM control unit, the IIC control unit establishes a communication connection with the CMOS camera through the IIC interface and configures the parameters of the CMOS camera, including: image resolution, refresh rate, and output timing parameters;

[0056] The video acquisition module acquires the image data fragments in RGB565 format output by the configured CMOS camera to obtain complete image data in RGB888 format; the video acquisition module acquires the image data fragments in RGB565 format according to the following steps:

[0057] Step 1, define a 1-bit counter cnt and initialize it to 0, the image data fragment in RGB565 format includes: 8-bit image data signal pdata and control signal for controlling pdata; the control signal for controlling pdata includes: clock signal pclk, field signal pvsync, and line signal phsync;

[0058] Step 2, in the clock domain of pclk, with pclk as the reference unit, after pvsync experiences a high level of several clock cycles of the clock signal pclk, pvsync becomes a low level, and when phsync is still at a high level, pdata, pvsync and phsync are simultaneously delayed by a clock cycle of the clock signal pclk, thereby obtaining a new 8-bit image data signal r_pdata, a new field signal r_pvsync and a new line signal r_phsync;

[0059] Step 3, when pvsync is low and phsync is high, cnt starts counting. After cnt counts to two data, that is, when cnt=1, the new 8-bit image data signal r_pdata and the 8-bit image data signal pdata are bit-joined to obtain a 16-bit image data signal r_fdata.

[0060] Step 4: The 16-bit image data signal r_fdata is delayed by two clock cycles of pclk under the action of the counter cnt. Meanwhile, r_pvsync and r_phsync are also delayed by two clock cycles of pclk respectively to obtain an updated field signal rr_pvsync and an updated row signal rr_phsync.

[0061] Step 5, after dividing the clock signal pclk by two, an output clock signal fclk is obtained;

[0062] Step 6, in the clock domain of fclk, with fclk as the reference unit, delay rr_pvsync, rr_phsync, and r_fdata by one clock cycle of fclk to obtain the output field signal fvsync, the output line signal fhsync, and the updated 16-bit image data signal rr_fdata;

[0063] Step 7: Use equation (1) to convert the updated 16-bit image data signal rr_fdata into a 24-bit image data signal fdata:

[0064] (1)

[0065] In formula (1), express 19-23bit data, express 16-18bit data, express 10-15bit data, express 8-9bit data, express 3-7bit data, express 0-2bit data.

[0066] Step 8: Output the 24-bit image data signal fdata, the output field signal fvsync, the output line signal fhsync and the output clock signal fclk as complete image data in RGB888 format.

[0067] The video to video stream module converts the image data in RGB888 format into image data in AXI-Stream format;

[0068] The VDMA module obtains the image data in AXI-Stream format, and under the drive of the ARM control unit, converts it into intermediate data in RGB888 format, and then stores it in the DDR4 memory through the DDR4 storage control unit;

[0069] Under the control of the ARM control unit, the GY-Net network reads the intermediate data in RGB888 format and the weights and biases of GY-Net from the DDR4 memory through the DDR4 storage control unit and processes them, and outputs the predicted label data corresponding to the image data; Figure 3 As shown in the figure, when the PC side is not quantized by fixed-point number, the GY-Net network includes: 19 layers of batch normalization, N layers of convolutional layers, 6 layers of maximum pooling layers and 1 layer of fully connected layer; among them, the N layers of convolutional layers include: 1 layer of ordinary convolutional layer, 9 layers of point-by-point convolutional layers, and 9 layers of depth convolutional layers.

[0070] In the GY-Net module, batch normalization is quantized by the convolutional neural network and then fused with the convolutional layer to form a new convolutional layer, such as Figure 4 As shown in the figure, after fixed-point quantization in the ZYNQ processing system, the GY-Net network includes: N convolutional layers, 6 maximum pooling layers and 1 fully connected layer;

[0071] In the GY-Net module, any n+1th convolutional layer outputs the feature map A of the nth convolutional layer according to formula (2): n Process and output the feature map A of the n+1th convolutional layer n+1 :

[0072] (2)

[0073] In formula (2), W n represents the convolution kernel weight of the nth convolution layer, B n represents the bias of the nth convolutional layer, represents the scale factor alignment parameter of the nth convolutional layer, ⊙ represents the convolution operation, and clip represents the truncation operation. Indicates a shift operation;

[0074] The maximum pooling layer of the GY-Net module performs a sliding operation with a window size of 2×2 on the input feature map. In the area covered by each sliding window, the maximum value of all pixel values ​​in the area is selected as the value of the corresponding position in the output feature map.

[0075] The fully connected layer of the GY-Net module uses formula (3) to obtain the category probability vector , and output the category corresponding to the maximum probability as the predicted label data:

[0076] (3)

[0077] In formula (3), Represents the feature map of the Nth convolutional layer, represents the weight of the fully connected layer, represents the bias of the fully connected layer, Represents the scale factor alignment parameter of the fully connected layer.

[0078] The ARM control unit splices the Chinese fonts corresponding to the predicted label data and the intermediate data in RGB888 format into an image in RGB888 format and its classification result data, and stores them in the DDR4 memory through the DDR4 storage control unit;

[0079] Driven by the ARM control unit, the VDMA module reads the image and its classification result data in RGB888 format from the DDR4 memory and converts it into the image and its classification result data in AXI-Stream format.

[0080] The video stream to video module converts the image in AXI-Stream format and its classification result data back to the image in RGB888 format and its classification result data;

[0081] The timing control module matches the image in RGB888 format and its classification result data with the corresponding output resolution, refresh rate and timing parameters, and transmits them to the display driver module;

[0082] The display driver module encodes the image in RGB888 format with output resolution, refresh rate and timing parameters and its classification result data, and converts them into image data in HDMI format, and then displays them on the HDMI display.

[0083] The image data in HDMI format includes: a pair of differential clock pairs and three pairs of differential data; the display driver module includes: three TMDS algorithm encoding modules, four ioserdes modules, and four single-ended signal conversion differential signal modules; wherein the TMDS algorithm module performs TMDS algorithm processing on the RGB components of the image data in RGB888 format with output resolution, refresh rate and timing parameters, respectively, in order to maintain the DC balance of the output signal and avoid data confusion during transmission; the ioserdes module converts parallel signals to serial signals for the three RGB components and one clock component after TMDS algorithm processing; the single-ended signal conversion differential signal module converts the data processed by the ioserdes module into differential signals for output, and finally displays them on the HDMI display.

Claims

1. An image classification system based on GY-Net network, characterized in that: include: CMOS camera, ZYNQ processing system, DDR4 memory, HDMI display; The ZYNQ processing system module includes: a PL logic design module and a PS control design module; The PL logic design module includes: a video acquisition module, a video to video stream module, a VDMA module, a GY-Net network, a video stream to video module, a timing control module, and a display driver module; The PS control design module includes: an IIC control unit, an ARM control unit, and a DDR4 storage control unit; The IIC control unit, under the initialization and driving of the ARM control unit, establishes a communication connection with the CMOS camera through the IIC interface and configures the parameters of the CMOS camera, including: image resolution, refresh rate, and output timing parameters; The video acquisition module acquires image data segments in RGB565 format output by the configured CMOS camera to obtain complete image data in RGB888 format; The video to video stream module converts the image data in RGB888 format into image data in AXI-Stream format; The VDMA module obtains the image data in the AXI-Stream format, and under the drive of the ARM control unit, converts it into intermediate data in the RGB888 format, and then stores it in the DDR4 memory through the DDR4 storage control unit; The GY-Net network, under the control of the ARM control unit, reads and processes the intermediate data in RGB888 format in the DDR4 memory through the DDR4 storage control unit, and outputs the predicted label data corresponding to the image data; The ARM control unit splices the Chinese font corresponding to the predicted label data and the intermediate data in RGB888 format into an image in RGB888 format and its classification result data, and stores them in the DDR4 memory through the DDR4 storage control unit; The VDMA module, driven by the ARM control unit, reads the image and its classification result data in RGB888 format from the DDR4 memory, and converts them into the image and its classification result data in AXI-Stream format; The video stream to video module converts the image in the AXI-Stream format and the classification result data thereof back to the image in the RGB888 format and the classification result data thereof; The timing control module matches the image in RGB888 format and its classification result data with the corresponding output resolution, refresh rate and timing parameters, and transmits them to the display driving module; The display driving module encodes the image in RGB888 format with output resolution, refresh rate and timing parameters and the classification result data thereof, and converts them into image data in HDMI format, and then displays them on the HDMI display.

2. According to claim 1, an image classification system based on a GY-Net network is characterized in that: The video acquisition module acquires image data segments in RGB565 format according to the following steps: Step 1, define a 1-bit counter cnt and initialize it to 0, the image data fragment in RGB565 format includes: an 8-bit image data signal pdata and a control signal for controlling pdata; the control signal for controlling pdata includes: a clock signal pclk, a field signal pvsync, and a line signal phsync; Step 2, in the clock domain of pclk, with pclk as the reference unit, after pvsync experiences a high level of several clock cycles of the clock signal pclk, pvsync becomes a low level, and when phsync is still at a high level, pdata, pvsync and phsync are simultaneously delayed by a clock cycle of the clock signal pclk, thereby obtaining a new 8-bit image data signal r_pdata, a new field signal r_pvsync and a new line signal r_phsync; Step 3, when pvsync is low and phsync is high, cnt starts counting. After cnt counts to two data, that is, when cnt=1, the new 8-bit image data signal r_pdata and the 8-bit image data signal pdata are bit-joined to obtain a 16-bit image data signal r_fdata. Step 4, the 16-bit image data signal r_fdata is delayed by two clock cycles of pclk under the action of counter cnt, and at the same time, r_pvsync and r_phsync are also delayed by two clock cycles of pclk respectively to obtain an updated field signal rr_pvsync and an updated line signal rr_phsync; Step 5, after dividing the clock signal pclk by two, an output clock signal fclk is obtained; Step 6, in the clock domain of fclk, with fclk as the reference unit, delay rr_pvsync, rr_phsync, and r_fdata by one clock cycle of fclk to obtain the output field signal fvsync, the output line signal fhsync, and the updated 16-bit image data signal rr_fdata; Step 7: Use equation (1) to convert the updated 16-bit image data signal rr_fdata into a 24-bit image data signal fdata: (1) In formula (1), express 19-23bit data, express 16-18bit data, express 10-15bit data, express 8-9bit data, express 3-7bit data, express 0-2bit data; Step 8: Output the 24-bit image data signal fdata, the output field signal fvsync, the output line signal fhsync and the output clock signal fclk as complete image data in RGB888 format.

3. The image classification system based on the GY-Net network according to claim 1, characterized in that: The GY-Net network includes: N convolutional layers and fully connected layers; Any n+1th convolutional layer in the GY-Net module converts the feature map A output by the nth convolutional layer into n Process and output the feature map A of the n+1th convolutional layer n+1 : (2) In formula (2), W n represents the convolution kernel weight of the nth convolution layer, B n represents the bias of the nth convolutional layer, represents the scale factor alignment parameter of the nth convolutional layer, ⊙ represents the convolution operation, and clip represents the truncation operation. Indicates a shift operation; The fully connected layer of the GY-Net module uses formula (3) to obtain the category probability vector , and output the category corresponding to the maximum probability as the predicted label data: (3) In formula (3), represents the feature map of the Nth convolutional layer, represents the weight of the fully connected layer, represents the bias of the fully connected layer, Represents the scale factor alignment parameter of the fully connected layer.

Images