An analysis type incremental learning method and system based on an FPGA development board

Abstract

The application discloses a kind of based on FPGA development board's analytic class incremental learning method and system.The method uses model back propagation training method to train base class data set, obtains basic model;Extract weight parameter, construct initial model as target model;If there are multiple new class data sets to be incrementally learned, set training order for each new class data set;Feature extraction and feature fusion are carried out to each new class data set, and the feature fusion vector corresponding to each new class data set is obtained;Class incremental learning is carried out to initial model in multiple rounds according to training order, and the class incremental learning model of last round is output as target model;The classification label of image is obtained by inputting the image to be detected into target model.The class incremental learning process of the application does not need to rely on old class data sample, while effectively protecting data privacy, by optimizing input mode, can be quickly deployed on FPGA development board.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image detection, in particular to a parsing incremental learning method and system based on an FPGA development board. BACKGROUND

[0002] In recent years, artificial intelligence technologies represented by deep learning have made rapid progress, driving innovation and transformation in multiple fields. Under this background, FPGA (Field-Programmable Gate Array) has gradually become an ideal choice for deploying deep learning models due to its flexible and reconfigurable hardware characteristics. FPGA can efficiently perform complex computing tasks, meet real-time and low-power consumption requirements, and provide effective solutions for edge computing and applications that require efficient inference. Especially in scenarios that require fast response and continuous computation, the parallel processing capability and low latency characteristics of FPGA are particularly prominent. However, most current deep learning models are still statically trained, lacking real-time adaptability to new data. When new data appears, it is usually necessary to retrain and redeploy the model, which not only increases development costs but also brings challenges in maintenance. As a solution, class incremental learning can gradually adapt to new data without global retraining. However, it faces the problem of "catastrophic forgetting", that is, when learning new knowledge, the model may forget the old knowledge it has mastered, leading to a decline in overall performance. To address this challenge, researchers have proposed various class incremental learning strategies, such as methods based on regularization, replay mechanisms, and bias correction. However, these methods often have certain limitations in practical applications, some perform poorly, some infringe on data privacy, and some are difficult to directly deploy to resource-constrained hardware platforms such as FPGA due to high computational complexity.

[0003] In existing technologies, most FPGA accelerators use static model deployment, where the model is first trained on a CPU / GPU and then transplanted to FPGA for inference. These accelerators cannot dynamically adapt to new data or unseen data and can only detect trained data, lacking the ability to continuously learn. If you want to adapt to new data, you need to retrain the model on a PC and redeploy the updated model to FPGA, which will cause time delay and complicate the deployment process (Nechi, Anouar, et al. "Fpga-based deep learning inference accelerators: Where are we standing?." 2018 IEEE Symposium on Computers and Communications (ISCC). ACM Transactions on Reconfigurable Technology and Systems16.4(2023): 1-32.). Some existing technologies, such as the FPGA-based deep learning training and inference system disclosed in Chinese patent document CN114781623A, mention training models on edge FPGAs, but only place part of the operations in the training process on FPGAs.

[0004] Most current deep learning models are still statically trained, lacking real-time adaptability to new data. When new data appears, the model usually needs to be retrained and redeployed, which not only increases development costs but also brings challenges in maintenance.

[0005] Class-incremental learning has emerged as an effective approach to address this issue. Class-incremental learning allows models to gradually adapt to new data without the need for global retraining, enabling efficient updating and optimization of models in dynamic environments. This approach effectively avoids frequent training and deployment, improves the adaptability of models in handling dynamic data, and reduces computational resource consumption. However, class-incremental learning faces the problem of "catastrophic forgetting", i.e., when learning new knowledge, the model may forget the old knowledge it has mastered, leading to a decline in overall performance. To address this challenge, researchers have proposed various class-incremental learning strategies, such as regularization-based, replay mechanism-based, and bias correction-based methods. These methods alleviate the forgetting problem to some extent, but still have some limitations in practical applications. For example, replay mechanism-based methods require storing old data or generating old class samples, which may result in high storage overhead or data privacy risks; regularization-based methods may not perform well when facing large-scale classes and complex tasks; and bias correction methods may have unstable performance when dealing with class imbalance problems. In addition, these methods often have high hardware resource requirements, especially for some computationally complex methods, making it difficult to directly deploy them on resource-constrained hardware platforms such as FPGAs, limiting the application of class-incremental learning in real-time and low-power scenarios. SUMMARY

[0006] The present application provides a class-incremental learning method and system based on an FPGA development board to solve one or more technical problems existing in the prior art.

[0007] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0008] The purpose of the present application is achieved at least by one of the following technical solutions.

[0009] A class-incremental learning method based on an FPGA development board, comprising the following steps:

[0010] S1, select a neural network model, use a pre-set base class data set, use a model back propagation training method to train the neural network model, obtain a basic model including an input layer, a feature extraction layer, a classifier layer and an output layer;

[0011] S2, freeze the weight parameters of the feature extraction layer of the basic model, rearrange the basic model through the base class data set, obtain an initial model including an initial classifier layer as a target model;

[0012] S3, determine whether there are multiple new class data sets to be incrementally learned, if yes, set a training order for each new class data set, execute step S4, otherwise execute step S6;

[0013] S4, feature extraction and feature fusion are performed on each new class data set to obtain a feature fusion vector corresponding to each new class data set;

[0014] S5, according to the training order, the initial model is subjected to multiple rounds of class incremental learning according to the new class data set and the feature fusion vector corresponding to each new class data set, and the class incremental learning model of the last round is output as the target model;

[0015] S6, input the image to be detected into the target model to obtain the classification label of the image; return to step S3.

[0016] Further, the base class data set includes images of multiple categories for training the feature extraction layer and category labels corresponding to the images;

[0017] The new class data set includes images of multiple categories for class incremental learning and category labels corresponding to the images;

[0018] There is no image of the same category between the new class data set and the base class data set;

[0019] There is no image of the same category between the multiple new class data sets.

[0020] Further, in step S1, the basic model is specifically as follows:

[0021]

[0022] wherein, represents the image to be detected; represents the prediction result; represents the weight parameters of the feature extraction layer; represents the weight parameters of the classifier layer; represents the feature extraction operation; represents the flattening operation; represents the softmax classification operation;

[0023] The feature extraction layer formula is specifically as follows:

[0024]

[0025] wherein, represents the output of the feature extraction layer;

[0026] The classifier layer formula is specifically as follows:

[0027]

[0028] wherein, represents the output of the classifier layer.

[0029] Further, in step S2, the initial model comprises an input layer, a feature extraction layer, an initial classifier layer and an output layer; the structure of the initial model is the same as that of the basic model except for the classifier layer and the initial classifier layer; the initial classifier layer of the initial model comprises a rearrangement layer and a full connection layer connected in sequence;

[0030] The input layer receives image data in the base class data set and inputs the image data in the base class data set to the feature extraction layer;

[0031] The feature extraction layer extracts features from the input image data to obtain a feature vector;

[0032] In the initial classifier layer, the feature vector passes through the rearrangement layer, outputs a feature expansion vector to the full connection layer, and the full connection layer outputs a prediction result vector to the output layer according to the feature expansion vector;

[0033] The output layer calculates the prediction result vector according to a preset softmax function and outputs each prediction result corresponding to the base class data set;

[0034] The base class data set comprises image data of multiple categories and category labels corresponding to the image data;

[0035] Step S2 specifically comprises the following steps:

[0036] S2.1, input the image data in the base class data set to the feature extraction layer, and the feature extraction layer extracts features from the input image data to obtain a feature vector; ;

[0037] S2.2, input the feature vector to the rearrangement layer to perform feature expansion and rearrangement and obtain a feature expansion vector; ;

[0038] S2.3, according to the feature expansion vector, output a prediction result vector to the output layer; and ​​and the class label in the base class dataset determines the weight parameters of the full connection layer , specifically as follows:

[0039]

[0040]

[0041] wherein, is the transpose matrix of denotes the unit matrix; denotes the inverse operation of the matrix; is the regularization term coefficient; is the autocorrelation matrix; denotes the corresponding class label;

[0042] S2.4, determining the initial model according to the weight parameters of the full connection layer .

[0043] Further, step S2.2 specifically as follows:

[0044] S2.2.1, dimensioning the feature vector to obtain the feature matrix ;

[0045] S2.2.2, transposing the feature matrix to obtain the transposed feature matrix ;

[0046] S2.2.3, performing feature expansion to obtain the expansion matrix , specifically as follows:

[0047]

[0048] S2.2.4, rearranging the expansion matrix to obtain the feature expansion vector .

[0049] Further, step S4 includes the following steps:

[0050] S4.1, according to the neural network model selected in step S1, using each new class dataset, respectively using the model back propagation training method to train the neural network model, obtaining a plurality of target base models corresponding to each new class dataset;

[0051] S4.2, let ;

[0052] S4.3, the first a first feature extraction dataset is obtained by randomly sampling the first new class dataset a first feature extraction dataset is obtained by randomly sampling the first new class dataset the class distribution of the first feature extraction dataset is consistent with the class distribution of the first new class dataset the class distribution of the first feature extraction dataset is consistent with the class distribution of the first new class dataset the data amount of the first feature extraction dataset is one-tenth of the first new class dataset the data amount of the first feature extraction dataset is one-tenth of the first new class dataset

[0053] S4.4, inputting the first feature extraction dataset into a target base model corresponding to the first new class dataset to obtain a first target vector S4.4, inputting the first feature extraction dataset into a target base model corresponding to the first new class dataset to obtain a first target vector S4.4, inputting the first feature extraction dataset into a target base model corresponding to the first new class dataset to obtain a first target vector S4.4, inputting the first feature extraction dataset into a target base model corresponding to the first new class dataset to obtain a first target vector

[0054] S4.5, obtaining a feature fusion vector corresponding to the first new class dataset according to the first target vector, specifically as follows: S4.5, obtaining a feature fusion vector corresponding to the first new class dataset according to the first target vector, specifically as follows: S4.5, obtaining a feature fusion vector corresponding to the first new class dataset according to the first target vector, specifically as follows:

[0055] ;

[0056] wherein, f is the feature fusion vector corresponding to the first new class dataset, a is a preset fusion coefficient, b is the first target vector, and c is the feature fusion vector corresponding to the first new class dataset; when c wherein, f is the feature fusion vector corresponding to the first new class dataset, a is a preset fusion coefficient, b is the first target vector, and c is the feature fusion vector corresponding to the first new class dataset; when c wherein, f is the feature fusion vector corresponding to the first new class dataset, a is a preset fusion coefficient, b is the first target vector, and c is the feature fusion vector corresponding to the first new class dataset; when c wherein, f is the feature fusion vector corresponding to the first new class dataset, a is a preset fusion coefficient, b is the first target vector, and c is the feature fusion vector corresponding to the first new class dataset; when c wherein, f is the feature fusion vector corresponding to the first new class dataset, a is a preset fusion coefficient, b is the first target vector, and c is the feature fusion vector corresponding to the first new class dataset; when c wherein, f is the feature fusion vector corresponding to the first new class dataset, a is a preset fusion coefficient, b is the first target vector, and c is the feature fusion vector corresponding to the first new class dataset; when c wherein, f is the feature fusion vector corresponding to the first new class dataset, a is a preset fusion coefficient, b is the first target vector, and c is the feature fusion vector corresponding to the first new class dataset; when c wherein, f is the feature fusion vector corresponding to the first new class dataset, a is a preset fusion coefficient, b is the first target vector, and c is the feature fusion vector corresponding to the first new class dataset; when c wherein, f is the feature fusion vector corresponding to the first new class dataset, a is a preset fusion coefficient, b is the first target vector, and c is the feature fusion vector corresponding to the first new class dataset; when c wherein, f is the feature fusion vector corresponding to the first new class dataset, a is a preset fusion coefficient, b is the first target vector, and c is the feature fusion vector corresponding to the first new class dataset; when c wherein, f is the feature fusion vector corresponding to the first new class dataset, a is a preset fusion coefficient, b is the first target vector, and c is the feature fusion vector corresponding to the first new class dataset; when c wherein, f is the feature fusion vector corresponding to the first new class dataset, a is a preset fusion coefficient, b is the first target vector, and c is the feature fusion vector corresponding to the first new class dataset; when c wherein, f is the feature fusion vector corresponding to the first new class dataset, a is a preset fusion coefficient, b is the first target vector, and c is the feature fusion vector corresponding to the first new class dataset; when c

[0057] S4.6, if , is the number of new class datasets (any positive integer except 1), the calculation is ended to obtain the feature fusion vectors corresponding to each new class dataset, otherwise, let , and returning to step S4.3.

[0058] Further, the class incremental learning model comprises an input layer, a feature extraction layer, a class incremental classifier layer and an output layer; the structure of the initial model and the structure of the class incremental learning model are the same except for the initial classifier layer and the class incremental classifier layer; the class incremental classifier layer of the class incremental learning model comprises a class incremental rearrangement layer and a full connection layer connected in sequence;

[0059] The input layer receives new class image data in a new class data set and inputs the new class image data in the new class data set to the feature extraction layer;

[0060] The feature extraction layer extracts features of the input new class image data to obtain a new class feature vector;

[0061] In the class increment classifier layer, the new class feature vector passes through a class increment rearrangement layer, and a new class feature expansion vector is output to a full connection layer. The full connection layer outputs a class increment prediction result vector to the output layer according to the new class feature expansion vector;

[0062] The output layer calculates the class increment prediction result vector according to a preset softmax function and outputs each prediction result corresponding to the new class data set;

[0063] The new class data set includes new class image data and a class label corresponding to the new class image data;

[0064] Step S5 includes the following steps:

[0065] S5.1, in the first round, new class image data in the first new class data set in the training sequence is input to the feature extraction layer of the initial model for feature extraction to obtain a first new class feature vector;

[0066] S5.2, the first new class feature vector is input to the class increment rearrangement layer for feature expansion and feature fusion, and a first new class feature expansion vector is obtained through rearrangement;

[0067] S5.3, the weight parameters of the full connection layer are expanded in dimension to obtain expanded weight parameters of the full connection layer; ;

[0068] S5.4, the full connection weight of the first class increment learning model is calculated according to the expanded weight parameters of the full connection layer , and a first class increment learning model is obtained;

[0069] S5.5, let ;

[0070] S5.6, in the th round, new class image data in the th new class data set in the training sequence is input to the feature extraction layer of the initial model for feature extraction to obtain a th new class feature vector;

[0071] S5.7, the ​​The new class feature vector is input into a class increment rearrangement layer, feature expansion and feature fusion are performed, and then a new class feature expansion vector is obtained through rearrangement.

[0072] S5.8, the full connection weight of the first class increment learning model is expanded in dimension to obtain the full connection weight of the expanded first class increment learning model.

[0073] S5.9, the full connection weight of the first class increment learning model is expanded in dimension to obtain the full connection weight of the expanded first class increment learning model.

[0074] S.10, if , the number of new class data sets is obtained, that is, the class increment learning model of the last round, otherwise let , and returning to step S5.6.

[0075] Further, step S5.2 is as follows:

[0076] S5.2.1, the first new class feature vector is expanded in dimension to obtain the first new class feature matrix ;

[0077] S5.2.2, the first new class feature matrix is transposed to obtain ;

[0078] S5.2.3, the feature fusion vector of the first new class data set is expanded in dimension to obtain the first new class feature fusion matrix ;

[0079] S5.2.4, feature expansion and feature fusion are performed to obtain the first new class expansion matrix , which is specifically as follows:

[0080]

[0081] wherein, is a fusion coefficient.

[0082] S5.2.5, the first new class expansion matrix is transposed to obtain ​​​​​​​​​​​​​​​Perform rearrangement (see rearrangement process) Figure 4 ), thus obtaining the first new class feature expansion vector ;

[0083] In step S5.4, based on the weight parameters of the expanded fully connected layer... Find the fully connected weights of the incremental learning model for the first class. The details are as follows:

[0084]

[0085]

[0086] in, This is the expansion vector of the first new class feature. This is the transpose of the first new class feature expansion vector. This refers to the category label of the new class image data in the first new class dataset; This is the first new type of autocorrelation matrix.

[0087] Furthermore, step S5.7 is detailed as follows:

[0088] S5.7.1, regarding the first New class feature vectors By increasing the dimensionality, we obtain the first... New class feature matrix ;

[0089] S5.7.2, regarding the first New class feature matrix Transpose to get ;

[0090] S5.7.3, regarding the first Feature fusion vector of a new class dataset By increasing the dimensionality, we obtain the first... A new type of feature fusion matrix ;

[0091] S5.7.4 Perform feature expansion and feature fusion to obtain the first... A new type of expansion matrix The details are as follows:

[0092]

[0093] in, The fusion coefficient;

[0094] In step S5.9, based on the expanded dimension of the first... Fully connected weights in an incremental learning model for each class , find the first Fully connected weights in an incremental learning model for each class The details are as follows:

[0095]

[0096]

[0097] in, For the first A new class feature expansion vector, For the first The transpose of the new class feature expansion vector For the corresponding number Category labels for new class image data in a new class dataset; For the first A new class of autocorrelation matrices.

[0098] An analytical incremental learning system based on an FPGA development board includes an FPGA development board and a target model deployed on the FPGA development board, which is obtained according to an analytical incremental learning method based on an FPGA development board.

[0099] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0100] This application constructs an initial model using the weight parameters of the feature extraction layer and preset classifier layer parameters from the extracted base model. Based on this initial model, it performs the first round of incremental class learning on the first-ranked new class dataset from multiple new class datasets. In incremental class learning, the target base model, trained using backpropagation on the new class dataset input in the current round, yields the feature fusion vector. Incremental class learning can be performed without relying on the base class dataset. By continuously updating the parameters of the incremental classifier layer in the next round, the incremental class learning model is continuously output, ultimately producing the final incremental class learning model. This iterative update of the incremental class learning model is completed. This application's incremental class learning model, through feature fusion, solves the problem that the model cannot learn new class image features after the feature extraction layer weights are frozen. In subsequent incremental class learning processes, it does not rely on data samples from old classes, thus effectively protecting data privacy. Meanwhile, the incremental learning model of this application can be quickly deployed on FPGA development boards by optimizing the input method, thereby solving the problem that method-type models with high computational complexity are difficult to deploy on FPGAs in related technologies, and providing an efficient solution for the application of incremental learning in real-time and low-power scenarios.

[0101] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. BRIEF DESCRIPTION OF DRAWINGS

[0102] Figure 1 A flow chart of a parsing-based incremental learning method using an FPGA development board in an embodiment of the present application.

[0103] Figure 2 A model architecture diagram of a base model in an embodiment of the present application.

[0104] Figure 3 A model architecture diagram of an initial model or a class-incremental learning model in an embodiment of the present application.

[0105] Figure 4 A schematic diagram of expansion and rearrangement in an embodiment of the present application.

[0106] Figure 5 A logic schematic diagram of a class-incremental learning process on an FPGA in an embodiment of the present application. DETAILED DESCRIPTION

[0107] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art.

[0108] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the

[0109] The block diagrams in the drawings show only the functionality of the features and can not imply a particular order of execution. For example, a block diagram can show a flow of information between blocks without specifying the order in which the blocks are executed. The functionality of the blocks can be combined or divided into other blocks. The functionality of the blocks can be implemented in hardware, software or a combination of hardware and software.

[0110] The flow diagrams shown in the drawings are examples only and are not necessarily to be construed as having any special order of execution. For example, some operations / steps can be broken down into further operations / steps, and some operations / steps can be combined or partially combined, so the actual order of execution can be changed according to the actual situation.

[0111] It should be noted that the "multiple" mentioned in the present text refers to two or more than two. The association relationship of "and / or" describes the association object, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the associated objects before and after are in an "or" relationship.

[0112] The embodiment provides a parsing incremental learning method based on an FPGA development board, as shown in the following formula (1) : Figure 1 The embodiment provides a parsing incremental learning method based on an FPGA development board, as shown in the following formula (1) :

[0113] S1, selecting a neural network model, using a pre-set base class data set, using a model back propagation training method to train the neural network model, obtaining a basic model including an input layer, a feature extraction layer, a classifier layer and an output layer, as shown in the following formula (1) : Figure 2 The base class data set includes image data of multiple categories and category labels corresponding to the image data;

[0114] In one embodiment, the base class data set is established as shown in the following formula (2) : Indicates the image input of m categories in the base class data set; Indicates the corresponding label or expected result, which is known; As shown in the following formula (3), the basic model is specifically as follows:

[0115] Figure 2 As shown in the following formula (3), the basic model is specifically as follows:

[0116]

[0117] Wherein, Indicates the image to be detected; Indicates the prediction result; Indicates the weight parameter of the feature extraction layer; Indicates the weight parameter of the classifier layer; Indicates the feature extraction operation; Indicates the flattening operation; Indicates the softmax classification operation;

[0118] The feature extraction layer formula is specifically as follows:

[0119]

[0120] Wherein, Indicates the output of the feature extraction layer;

[0121] The classifier layer formula is specifically as follows:

[0122] ​​

[0123] wherein, represents the output of the classifier layer.

[0124] In one embodiment, the selected neural network model is a residual convolutional neural network.

[0125] S2, freeze the weight parameters of the feature extraction layer of the base model, parse and rearrange the base model through the base class dataset, and obtain an initial model including an initial classifier layer as a target model;

[0126] As shown in Figure 3 , the initial model includes an input layer, a feature extraction layer, an initial classifier layer, and an output layer; the structure of the initial model is the same as that of the base model except for the classifier layer and the initial classifier layer; the initial classifier layer of the initial model includes a rearrangement layer and a fully connected layer connected in sequence;

[0127] The input layer receives image data in the base class dataset and inputs the image data in the base class dataset to the feature extraction layer;

[0128] The feature extraction layer extracts features from the input image data to obtain a feature vector;

[0129] In the initial classifier layer, the feature vector passes through the rearrangement layer, outputs a feature expansion vector to the fully connected layer, and the fully connected layer outputs a prediction result vector to the output layer according to the feature expansion vector;

[0130] The output layer calculates the prediction result vector according to a preset softmax function and outputs each prediction result corresponding to the base class dataset;

[0131] Step S2 specifically includes the following steps:

[0132] S2.1, input image data in the base class dataset to the feature extraction layer, and the feature extraction layer extracts features from the input image data to obtain a feature vector , specifically as follows:

[0133]

[0134] wherein: , represents the total number of images in the base class dataset; represents the flattened feature dimension;

[0135] S2.2, input the feature vector to the rearrangement layer for feature expansion and rearrangement to obtain a feature expansion vector , specifically as follows:

[0136] S2.2.1, on the feature vector dimensional, to obtain a feature matrix , shape ( , 1, );

[0137] S2.2.2, transpose the feature matrix , to obtain a transposed feature matrix , shape ( , , 1);

[0138] S2.2.3, perform feature expansion, to obtain an expansion matrix , shape ( , , ), as follows:

[0139]

[0140] S2.2.4, rearrange the expansion matrix , to obtain a feature expansion vector , shape ( , );

[0141] In one embodiment, the rearrangement process is as shown in Figure 4 , as follows:

[0142] Take the upper triangular elements of the expansion matrix , and place the elements of each row in turn in the first row in order of rows, to form a feature expansion vector ; the upper triangular elements include all elements on the main diagonal and to the right of the main diagonal of the matrix.

[0143] S2.3, determine the weight parameters of the fully connected layer from the feature expansion vector and the class labels in the base class dataset, shape ( , ), as follows:

[0144]

[0145]

[0146] wherein is the transposed matrix of ; denotes the identity matrix; denotes the inverse operation of a matrix; is a regularization term coefficient, in one embodiment, Take 0.1; This is the autocorrelation matrix; express Corresponding category tags;

[0147] S2.4, Based on the weight parameters of the fully connected layer The initial model is determined using the following formula:

[0148]

[0149] →

[0150]

[0151] →

[0152]

[0153] in, This represents the output of the feature extraction layer. Indicates to The feature matrix obtained by dimensionality increase Indicates according to The expansion matrix obtained by performing feature expansion, Indicates to The feature expansion vector is obtained by rearranging the vectors.

[0154] S3. Determine if there are multiple new class datasets to be incrementally learned. If so, set the training order for each new class dataset and execute step S4; otherwise, execute step S6.

[0155] The new class dataset includes new class image data and corresponding category labels for the new class image data;

[0156] In one embodiment, establish A new class of datasets , , << , There are no images of the same category in the new class datasets; Indicates the first Image input from n classes in a new dataset; express The corresponding label or expected result It is known that there are no images of the same category between the new class dataset and the base class dataset.

[0157] S4, feature extraction and feature fusion are performed on each new class data set to obtain a feature fusion vector corresponding to each new class data set, including the following steps:

[0158] S4.1, according to the neural network model selected in step S1, using a new class data set, respectively using model back propagation training method to train the neural network model, obtaining a target base model corresponding to a new class data set;

[0159] S4.2, let ;

[0160] S4.3, randomly sampling the th new class data set to obtain the th feature extraction data set, the class distribution of the th feature extraction data set is consistent with the class distribution of the th new class data set, and the data amount of the th feature extraction data set is one tenth of the th new class data set;

[0161] S4.4, input the th feature extraction data set into the target base model corresponding to the th new class data set to obtain the feature vector of each image, and then average it to obtain the th target vector , specifically as follows:

[0162]

[0163]

[0164] Wherein: represents the 1st image in the th feature extraction data set; represents the feature vector of the 1st image in the th feature extraction data set; represents the feature vector of the th image in the th feature extraction data set; represents the total number of images in the th feature extraction data set;

[0165] S4.5, according to the th target vector , the feature fusion vector corresponding to the th new class data set is obtained, specifically as follows:

[0166] ;

[0167] in, For the corresponding number Feature fusion vectors of a new class dataset In one embodiment, the preset fusion coefficient is used. Take 0.9, For the first A target vector, For the corresponding number Feature fusion vectors of new class datasets; when hour, This corresponds to the feature fusion vector of the first new class dataset in the training order. For the first A target vector, =0;

[0168] S4.6, if , The calculation ends when the number of new class datasets (any positive integer other than 1) is reached, yielding the feature fusion vectors for each new class dataset; otherwise, let... Return to step S4.3.

[0169] S5. According to the training order, perform multiple rounds of class incremental learning on the initial model based on the feature fusion vectors of the new class dataset and the corresponding new class datasets, and output the class incremental learning model of the last round as the target model.

[0170] The class incremental learning model includes an input layer, a feature extraction layer, a class incremental classifier layer, and an output layer. Except for the initial classifier layer and the class incremental classifier layer, the structure of the initial model is the same as that of the class incremental learning model. The class incremental classifier layer of the class incremental learning model includes a sequentially connected class incremental rearrangement layer and a fully connected layer.

[0171] The input layer receives new class image data from the new class dataset and inputs the new class image data from the new class dataset into the feature extraction layer;

[0172] The feature extraction layer extracts features from the input new class image data to obtain the new class feature vector;

[0173] In the class increment classifier layer, the new class feature vector passes through the class increment rearrangement layer, outputting a new class feature expansion vector to the fully connected layer. The fully connected layer outputs the class increment prediction result vector to the output layer based on the new class feature expansion vector.

[0174] The output layer calculates the class-incremental prediction result vector according to a preset softmax function, and outputs each prediction result corresponding to the new class data set;

[0175] Step S5 includes the following steps:

[0176] S5.1, in the first round, the new class image data in the first new class data set in the training sequence is input into the initial model to perform feature extraction, and a first new class feature vector is obtained , specifically as follows:

[0177]

[0178] wherein, ; denotes the total number of images of the first new class data set; denotes the flattened feature dimension;

[0179] S5.2, the first new class feature vector is input into the class-incremental rearrangement layer to perform feature expansion and feature fusion, and a first new class feature expansion vector is obtained through rearrangement, specifically as follows:

[0180] S5.2.1, the first new class feature vector is dimensioned to obtain a first new class feature matrix , with a shape of ( , 1, );

[0181] S5.2.2, the first new class feature matrix is transposed to obtain , with a shape of ( , , 1);

[0182] S5.2.3, the first new class feature fusion vector is dimensioned to obtain a first new class feature fusion matrix , with a shape of ( , 1, );

[0183] S5.2.4, feature expansion and feature fusion are performed to obtain a first new class expansion matrix , with a shape of ( , , ), specifically as follows:

[0184]

[0185] wherein, is a fusion coefficient, in one embodiment, is taken as 0.95;

[0186] S5.2.5, the first new class expansion matrix is rearranged (the rearrangement process is shown in Figure 4 ), to obtain the first new class feature expansion vector , which has a shape of ( , );

[0187] S5.3, the weight parameters of the full connection layer are expanded, to obtain the expanded weight parameters of the full connection layer , which has a shape of ( , );

[0188] S5.4, according to the expanded weight parameters of the full connection layer , the full connection weight of the first class incremental learning model is obtained , which has a shape of ( , ), and then the first class incremental learning model is obtained, which is specifically as follows:

[0189]

[0190]

[0191] wherein, is the first new class feature expansion vector, is the transpose of the first new class feature expansion vector, is the class label corresponding to the new class image data in the first new class data set; is the first new class autocorrelation matrix;

[0192] The first class incremental learning model is as follows:

[0193]

[0194] →

[0195]

[0196] →

[0197]

[0198] wherein, represents the feature fusion vector of the 1st new class data set the 1st new class feature fusion matrix obtained by dimension increasing, represents the feature expansion vector of the 1st new class data set according to and the 1st new class expansion matrix obtained by feature expansion and feature fusion; represents the feature expansion vector of the 1st new class expansion matrix the 1st new class feature expansion vector obtained by rearrangement.

[0199] S5.5, let ;

[0200] S5.6, in the 1st round, the new class image data in the 1st new class data set in the training sequence is input into the feature extraction layer of the initial model to perform feature extraction, and the 1st new class feature vector is obtained;

[0201] S5.7, the 1st new class feature vector is input into the class increment rearrangement layer to perform feature expansion and feature fusion, and then the 1st new class feature expansion vector is obtained by rearrangement, which is specifically as follows:

[0202] S5.7.1, the 1st new class feature vector is dimensioned to obtain the 1st new class feature matrix with a shape of ( , , ), represents the total number of images of the 1st new class data set;

[0203] S5.7.2, the 1st new class feature matrix is transposed to obtain ;

[0204] S5.7.3, the feature fusion vector of the 1st new class data set is dimensioned to obtain the 1st new class feature fusion matrix ;

[0205] S5.7.4, feature expansion and feature fusion are performed to obtain the 1st new class expansion matrix with a shape of ( , , ), as detailed below:

[0206]

[0207] in, The fusion coefficient;

[0208] S5.8, regarding the first Fully connected weights in an incremental learning model for each class Perform dimension expansion to obtain the expanded dimension of the [missing information]. Fully connected weights in an incremental learning model for each class ;

[0209] S5.9, According to the expanded dimension of the first... Fully connected weights in an incremental learning model for each class , find the first Fully connected weights in an incremental learning model for each class Its shape is ( , ), and thus obtain the first The incremental learning models for each class are as follows:

[0210]

[0211]

[0212] in, For the first A new class feature expansion vector, For the first The transpose of the new class feature expansion vector For the corresponding number Category labels for new class image data in a new class dataset; For the first A new class of autocorrelation matrices;

[0213] S5.10, if , The calculation ends when the number of new class datasets (which is any positive integer other than 1) is reached, yielding the result of the calculation. The class-based incremental learning model, i.e., the class-based incremental learning model in the last round, otherwise let Return to step S5.6;

[0214] The formula for the final round of incremental learning model is as follows:

[0215]

[0216] →

[0217]

[0218] →

[0219]

[0220] in, For the first Fully connected weights of incremental learning models for each class; Indicates the first Feature fusion vector of a new class dataset The first dimension obtained by dimensional upgrading A new class feature fusion matrix, Indicates according to and The first feature obtained by feature expansion and feature fusion A new class expansion matrix; Indicates the first A new type of expansion matrix The result of rearranging A new class feature expansion vector.

[0221] S6. Input the image to be detected into the target model to obtain the image's classification label; return to step S3.

[0222] An analytical incremental learning system based on an FPGA development board includes an FPGA development board and a target model deployed on the FPGA development board, which is obtained according to an analytical incremental learning method based on an FPGA development board.

[0223] In one embodiment, such as Figure 5 As shown, the target model is quantized to obtain the quantized weight values ​​and the shift factors of the activation values. An FPGA development board is selected, and a global architecture adapted to the network model is designed based on the quantized weight values, the shift factors of the activation values, and the preset FPGA development board. The relevant hardware resource configurations are added to the device tree, and the bit stream of the PL (Programmable Logic) end is integrated with the Linux kernel to generate a complete Linux system image that can run on the FPGA development board. The generated Linux system image is burned into the NAND Flash of the FPGA development board to obtain the target FPGA development board with the target model deployed.

[0224] When updating the model parameters of the target FPGA development board with the target model deployed, the parameters to be updated are loaded from NAND Flash into the BRAM (Block Random Access Memory) at the PL end. Then, images from the new dataset are transmitted one by one to the FPGA development board. The new dataset is then used to perform [processing] on the target FPGA development board with the target model deployed. The incremental learning of wheel classes is represented by the following formula:

[0225]

[0226]

[0227] in, for The novel autocorrelation matrix in the first round of incremental learning; For the first A new class of autocorrelation matrices; This is the feature expansion vector in round 1. For the feature expansion vector The transpose of the matrix; For the first The novel autocorrelation matrix in the round, For the first A novel autocorrelation matrix in each round; For the first Feature expansion vectors in each round; For the feature expansion vector The transpose of the matrix; For the corresponding Adaptive scalar, For the corresponding The adaptive scalar is as follows:

[0228]

[0229]

[0230] The acquired images to be detected are transmitted to the target FPGA development board with the target model deployed after the model parameters have been updated for image classification.

[0231] Since FPGAs do not require inversion operations, but only division operations, the difficulty of updating and deploying incremental learning models is greatly reduced, enabling incremental learning models deployed on FPGA development boards to easily achieve incremental learning.

[0232] Although the FPGA can perform matrix inversion operation, it will consume a large amount of computing resources due to the large amount of computation. If a single image is input, C becomes a scalar, and the matrix inversion operation becomes a division operation. Therefore, the solution idea of the embodiment of the present application is: there is an inversion algorithm in the target class incremental model → the FPGA inversion operation will consume a large amount of computing resources, and even the model cannot be deployed → if a single image is input (that is, the class incremental learning is performed on each image), the inversion operation becomes a division operation → reduce the difficulty of model deployment and update difficulty.

[0233] In this way, when the model needs to add image knowledge of new classes, class incremental learning can be easily performed on the FPGA development board, without the need to rely on the original data set for retraining and redeployment, thereby solving the problems of complicated training process, large training cost and low training efficiency in the prior art.

[0234] The method for analyzing class incremental learning based on the FPGA development board provided by the embodiment of the present application can effectively solve the problems described in the background art. Compared with the traditional method, the method for analyzing class incremental learning based on the FPGA development board provided by the embodiment of the present application not only can effectively alleviate catastrophic forgetting, but also can protect data privacy and ensure that the model can be updated without relying on the original data. The method proposed in the embodiment of the present application can maintain high reasoning ability and stability when processing continuously changing complex application scenarios, and provides stronger long-term adaptability for deep learning models. In addition, the class incremental learning system based on the FPGA development board provided by the embodiment of the present application proposes an innovative method, which changes the inversion operation in the class incremental learning model into a division operation, making it possible to deploy the class incremental learning model to a resource-limited FPGA platform. Although the FPGA can implement inversion operation, due to high computational complexity, large resource consumption, slow speed and possible precision loss, it is generally recommended to avoid direct complex inversion operation in hardware design.

[0235] In addition, the above-described figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not for limitation purposes. It is easy to understand that the processes shown in the above-described figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0236] It should be understood that the present application is not limited to the precise construction which has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. An FPGA development board-based parsing class incremental learning method, characterized in that, The method comprises the following steps: S1, selecting a neural network model, using a pre-set base class data set to train the neural network model, and obtaining a base model comprising an input layer, a feature extraction layer, a classifier layer and an output layer; S2, freezing the weight parameters of the feature extraction layer of the base model, and rearranging the base model through the base class data set to obtain an initial model comprising an initial classifier layer as a target model; Specifically, the method comprises the following steps: S2.1, image data in the base class dataset is converted into a feature vector input feature extraction layer, the feature extraction layer extracts features from the input image data ;​ S2.2, the feature vector The input rearrangement layer, feature expansion and rearrangement are performed to obtain a feature expansion vector ; Specifically as follows: S2.2.1, to the feature vector dimensional, to obtain the feature matrix ; S2.2.2, to the feature matrix transpose, to obtain the transposed feature matrix ; S2.2.3, feature expansion is performed to obtain an expansion matrix In particular, as follows: S2.2.4, on the expansion matrix rearranged to obtain the characteristic expansion vector ; take the expansion matrix elements, in the order of rows, place each row of elements in the first row to form the characteristic expansion vector ; the upper triangular elements include all elements on the main diagonal and to the right of the matrix S2.3, according to the feature expansion vector and the class labels in the base dataset to determine the weight parameters of the fully connected layer as follows: wherein, is the transpose matrix of denotes the identity matrix; denotes the inverse operation of a matrix; is a regularization term coefficient; is an autocorrelation matrix; denotes the corresponding class label; S2.4, the weight parameters of the fully connected layer determining an initial model; S3, determining whether there are multiple new class data sets to be incrementally learned, and if so, setting a training order for each new class data set, and executing step S4, otherwise executing step S6; S4, performing feature extraction and feature fusion on each new class data set to obtain a feature fusion vector corresponding to each new class data set; S5, performing multiple rounds of class incremental learning on the initial model according to the new class data set and the feature fusion vector corresponding to each new class data set according to the training order, and outputting the class incremental learning model of the last round as the target model; S6, inputting the image to be detected into the target model to obtain the classification label of the image; returning to step S3.

2. The method of claim 1, wherein the method is based on an FPGA development board. The base class data set comprises images of multiple categories for training the feature extraction layer and category labels corresponding to the images; The new class data set comprises images of multiple categories for class incremental learning and category labels corresponding to the images; There are no images of the same category between the new class data set and the base class data set; There are no images of the same category between the multiple new class data sets.

3. The method of claim 1, wherein the method is based on an FPGA development board. The initial model comprises an input layer, a feature extraction layer, an initial classifier layer and an output layer; the structure of the initial model is the same as that of the base model except for the classifier layer and the initial classifier layer; the initial classifier layer comprises a rearrangement layer and a full connection layer connected in sequence; The input layer receives image data in the base class data set and inputs the image data in the base class data set into the feature extraction layer; The feature extraction layer extracts features from the input image data to obtain a feature vector; In the initial classifier layer, the feature vector passes through the rearrangement layer and outputs a feature expansion vector to the full connection layer, and the full connection layer outputs a prediction result vector to the output layer according to the feature expansion vector; The output layer calculates the prediction result vector according to a pre-set softmax function and outputs each prediction result corresponding to the base class data set; The base class data set comprises image data of multiple categories and category labels corresponding to the image data.

4. The FPGA development board-based parsing incremental learning method according to claim 1, characterized in that, Step S4 comprises the following steps: S4.1, using each new class data set to train the neural network model according to the neural network model selected in step S1, and obtaining multiple target base models corresponding to each new class data set; S4.2, let ; S4.3, regarding the first Random sampling was performed on the new class dataset to obtain the first... The feature extraction dataset, the first The category distribution of the feature extraction dataset is similar to that of the first feature extraction dataset. The class distributions of the new class datasets are consistent; S4.4, input the first feature extraction data set to the target base model corresponding to the first new class data set, to obtain the first target vector; S4.5、according to the first target vector, a feature fusion vector of the corresponding first new class data set is obtained, specifically as follows: ； wherein, is a feature fusion vector corresponding to the i-th new class data set, is a feature fusion vector corresponding to the i-th new class data set, is a preset fusion coefficient, is the i-th target vector, is a feature fusion vector corresponding to the i-th new class data set, is a feature fusion vector corresponding to the i-th new class data set, is a feature fusion vector corresponding to the i-th new class data set; when is a feature fusion vector corresponding to the i-th new class data set, is a feature fusion vector corresponding to the i-th new class data set, is the i-th target vector, is the i-th target vector, is 0. S4.6, if , is the number of new class data sets, the calculation is ended, the feature fusion vectors corresponding to each new class data set are obtained, otherwise , return to step S4.

3.

5. The method of claim 1, wherein the method is based on an FPGA development board. In step S5, the class incremental learning model comprises an input layer, a feature extraction layer, a class incremental classifier layer and an output layer; the structure of the initial model is the same as that of the class incremental learning model except for the initial classifier layer and the class incremental classifier layer; the class incremental classifier layer comprises a class incremental rearrangement layer and a full connection layer connected in sequence; The input layer receives new class image data in the new class data set and inputs the new class image data in the new class data set into the feature extraction layer; The feature extraction layer extracts features of the input new class image data to obtain a new class feature vector; In the class incremental classifier layer, the new class feature vector passes through a class incremental rearrangement layer, and a new class feature expansion vector is output to a full connection layer. The full connection layer outputs a class incremental prediction result vector to the output layer according to the new class feature expansion vector; The output layer calculates the class incremental prediction result vector according to a preset softmax function, and outputs each prediction result corresponding to the new class data set. The new class data set includes new class image data of multiple categories and category labels corresponding to the new class image data.

6. The FPGA development board-based parsing incremental learning method according to claim 5, characterized in that, Step S5 includes the following steps: S5.1, in the first round, input the new class image data in the first new class data set in the training sequence into the feature extraction layer of the initial model to extract features and obtain the first new class feature vector; S5.2, input the first new class feature vector into the class incremental rearrangement layer to perform feature expansion and feature fusion, and then rearrange to obtain the first new class feature expansion vector; S5.3, weight parameters of the fully connected layer dimension expansion is performed to obtain the weight parameters of the dimension expanded fully connected layer ; S5.4, weight parameters of the full connection layer after dimension expansion , obtain the full connection weight of the first class incremental learning model , and obtain the first class incremental learning model S5.5, let ; S5.6、in the first round, the new class image data in the first new class data set in the training sequence is input into the feature extraction layer of the initial model for feature extraction to obtain the first new class feature vector; S5.7, input the first new class feature vector into the class incremental rearrangement layer, perform feature expansion and feature fusion, and then obtain the first new class feature expansion vector through rearrangement S5.7, input the first new class feature vector into the class incremental rearrangement layer, perform feature expansion and feature fusion, and then obtain the first new class feature expansion vector through rearrangement S5.7, input the first new class feature vector into the class incremental rearrangement layer, perform feature expansion and feature fusion, and then obtain the first new class S5.8, the full connection weight of the first class incremental learning model is expanded in dimension to obtain the full connection weight of the first class incremental learning model after expansion in dimension ;​ S5.9、 according to the full connection weight of the expanded class incremental learning model S5.10、 according to the full connection weight of the expanded class incremental learning model S5.11、 according to the full connection weight of the expanded class incremental learning model S5.12、 according to the full connection weight of the expanded class incremental learning model S5.13、 according to the full connection weight of the expanded class incremental learning model S5.14、 according to the full connection weight of the expanded class incremental learning model S5.10, if , is the number of new class datasets, end the calculation to obtain the class incremental learning model, i.e. the class incremental learning model of the last round, otherwise let , return to step S5.

6.

7. The FPGA development board-based parsing incremental learning method according to claim 6, characterized in that, Step S5.2 is specifically as follows: S5.2.1, to the first new class feature vector dimensional, to obtain the first new class feature matrix ; S5.2.2, to the first new class feature matrix Transposing, we get ; S5.2.3, feature fusion vector of the first new class data set dimensionality is increased to obtain a first new class feature fusion matrix ; S5.2.4, feature expansion and feature fusion are performed to obtain a first new class expansion matrix , and specifically as follows: wherein, is a fusion coefficient; S5.2.5, to the first new class expansion matrix Re-arranging, get the first new class feature expansion vector ; In step S5.4, the weight parameters of the full connection layer after dimension expansion are obtained according to the weight parameters of the full connection layer before dimension expansion , and the full connection weight of the first class incremental learning model is obtained , and the full connection weight of the first class incremental learning model is obtained wherein, is a first new class feature expansion vector, is a transpose of the first new class feature expansion vector, is a class label corresponding to a new class image data in the first new class data set; is a first new class autocorrelation matrix; Step S5.7 is specifically as follows: S5.7.1, to the first new class feature vector dimensional, to obtain the first new class feature matrix ; S5.7.2, to the first new class feature matrix transposed to obtain ; S5.7.3, to the feature fusion vector of the first new class data set dimensional ascension, to obtain the first new class feature fusion matrix ; S5.7.4, feature expansion and feature fusion are performed to obtain a new class expansion matrix , as follows: ​ wherein, is a fusion coefficient; In step S5.9, the full connection weight of the i-th class incremental learning model is calculated according to the extended dimension, and the full connection weight of the i-th class incremental learning model is calculated, specifically as follows: ​​​​ wherein, is the th new class feature expansion vector, is the transpose of the th new class feature expansion vector, is the class label corresponding to the new class image data in the th new class data set; is the th new class autocorrelation matrix.

8. An FPGA development board-based parsing class incremental learning system, characterized in that, The target model obtained by the parsing class incremental learning method based on the FPGA development board according to any one of claims 1-7 is deployed on the FPGA development board.

Images