Analytic incremental learning method and system based on FPGA development board

Abstract

The invention discloses an analysis class incremental learning method and system based on an FPGA development board. The method comprises the following steps: training a base class data set by adopting a model back propagation training method to obtain a basic model; extracting weight parameters, and constructing an initial model as a target model; if a plurality of new-class data sets to be subjected to incremental learning exist, setting a training sequence for each new-class data set; performing feature extraction and feature fusion on each new class data set to obtain a feature fusion vector corresponding to each new class data set; performing multiple rounds of class incremental learning on the initial model according to a training sequence, and outputting a class incremental learning model of the last round as a target model; and inputting a to-be-detected image into the target model to obtain a classification label of the image. The class incremental learning process does not need to depend on data samples of old classes, data privacy is effectively protected, and meanwhile the method can be rapidly deployed on an FPGA development board by optimizing an input mode.

Description

Technical Field

[0001] The present application relates to the field of image detection technology, and in particular to an incremental learning method and system for analytical applications based on an FPGA development board. Background Art

[0002] In recent years, artificial intelligence technology represented by deep learning has achieved rapid development, promoting innovation and change in many fields. In this context, FPGA (Field-Programmable Gate Array) has gradually become an ideal choice for deploying deep learning models with its flexible and reconfigurable hardware characteristics. FPGA can efficiently perform complex computing tasks, meet the requirements of real-time and low power consumption, and provide an effective solution for edge computing and applications that require efficient reasoning. Especially in the face of scenarios that require fast response and continuous computing, FPGA's parallel processing capabilities and low latency characteristics are particularly prominent. However, most current deep learning models are still statically trained and lack the ability to adapt to new data in real time. When new data appears, the model usually needs to be retrained and redeployed, which not only increases development costs but also brings maintenance challenges. Quasi-incremental learning is a solution that can gradually adapt to new data without global retraining. However, it faces the problem of "catastrophic forgetting". That is, when learning new knowledge, the model may forget the old knowledge it has mastered, resulting in a decrease in overall performance. To meet this challenge, researchers have proposed a variety of quasi-incremental learning strategies, such as those based on regularization, replay mechanism, and bias correction. However, these methods often have certain limitations in practical applications. Some of them perform poorly, some infringe on data privacy, and some are difficult to deploy directly on resource-constrained hardware platforms such as FPGAs due to their high computational complexity.

[0003] In the existing technology, most FPGA accelerators use static model deployment. The model is first trained on the CPU / GPU and then ported to the FPGA for inference. These accelerators cannot dynamically adapt to new or unseen data. They can only detect trained data and have no ability to continuously learn. To adapt to new data, the model needs to be retrained on the PC and the updated model needs to be redeployed on the FPGA, which will cause time delays and complicate the deployment process (Nechi, Anouar, et al. "Fpga-based deep learning inference accelerators: Where are westanding?"). ACM Transactions on Reconfigurable Technology and Systems16.4(2023): 1-32.). Some prior arts, such as a deep learning training and inference system based on FPGA disclosed in Chinese patent document CN114781623A, mention training models on edge FPGAs, but only put part of the operations in the training process on the FPGA.

[0004] Most current deep learning models are still statically trained and lack the ability to adapt to new data in real time. When new data emerges, the model usually needs to be retrained and redeployed, which not only increases development costs but also brings maintenance challenges.

[0005] Class incremental learning has emerged as the times require, providing an effective way to solve this problem. Class incremental learning allows the model to gradually adapt to new data without global retraining, thereby efficiently updating and optimizing the model in a dynamic environment. This method effectively avoids frequent training and deployment, improves the adaptability of the model when processing dynamic data, and reduces computing resource consumption. However, class incremental learning faces the problem of "catastrophic forgetting", that is, when learning new knowledge, the model may forget the old knowledge it has mastered, resulting in overall performance degradation. In order to meet this challenge, researchers have proposed a variety of class incremental learning strategies, such as those based on regularization, replay mechanism, and bias correction. These methods have alleviated the forgetting problem to a certain extent, but there are still some limitations in practical applications. For example, methods based on the replay mechanism need to store old data or generate old class samples, which may bring high storage overhead or data privacy risks; although methods based on regularization do not need to save old data, they may not work well when facing large-scale categories and complex tasks; and bias correction methods also have unstable performance when dealing with class imbalance problems. In addition, these methods often have high requirements on hardware resources, especially some methods with high computational complexity, which are difficult to deploy directly on resource-constrained hardware platforms such as FPGAs, limiting the application of incremental learning in real-time and low-power scenarios. Summary of the invention

[0006] The present application provides an FPGA-based incremental learning method and system to solve one or more technical problems existing in the prior art.

[0007] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0008] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0009] A method for incremental learning of parsing based on an FPGA development board comprises the following steps: S1. Select a neural network model, use a pre-set base class data set, and adopt a model back propagation training method to train the neural network model to obtain a basic model including an input layer, a feature extraction layer, a classifier layer, and an output layer; S2, freezing the weight parameters of the feature extraction layer of the base model, parsing and rearranging the base model through the base class data set, and obtaining an initial model including an initial classifier layer as the target model; S3, determine whether there are multiple new class data sets to be incrementally learned, if so, set a training order for each new class data set and execute step S4, otherwise execute step S6; S4, performing feature extraction and feature fusion on each new class data set to obtain a feature fusion vector corresponding to each new class data set; S5, performing multiple rounds of class incremental learning on the initial model according to the new class data sets and the feature fusion vectors corresponding to each new class data set in the training order, and outputting the class incremental learning model of the last round as the target model; S6. Input the image to be detected into the target model to obtain the classification label of the image; return to step S3.

[0010] Furthermore, the base class dataset includes images of various categories and category labels corresponding to the images for training the feature extraction layer; The new class dataset includes images of multiple categories and their corresponding category labels for class incremental learning; There are no images of the same category between the new class dataset and the base class dataset; There are no images of the same category between multiple new class datasets.

[0011] Furthermore, in step S1, the basic model is specifically as follows:

[0012] in, represents the image to be detected; Indicates the prediction result; Represents the weight parameters of the feature extraction layer; Represents the weight parameters of the classifier layer; Represents feature extraction operation; Represents a flattening operation; Represents softmax classification operation; The feature extraction layer formula is as follows:

[0013] in, Represents the output of the feature extraction layer; The classifier layer formula is as follows:

[0014] in, Represents the output of the classifier layer.

[0015] Further, in step S2, the initial model includes an input layer, a feature extraction layer, an initial classifier layer and an output layer; except for the classifier layer and the initial classifier layer, the structure of the initial model is the same as that of the basic model; the initial classifier layer of the initial model includes a rearrangement layer and a fully connected layer connected in sequence; The input layer receives the image data in the base class data set, and inputs the image data in the base class data set to the feature extraction layer; The feature extraction layer extracts features from the input image data to obtain a feature vector; In the initial classifier layer, the feature vector passes through the rearrangement layer, outputs the feature expansion vector to the fully connected layer, and the fully connected layer outputs the prediction result vector to the output layer according to the feature expansion vector; The output layer calculates the prediction result vector according to a preset softmax function, and outputs each prediction result corresponding to the base class data set; The base class data set includes image data of multiple categories and category labels corresponding to the image data; Step S2 specifically includes the following steps: S2.1. Input the image data in the base class data set into the feature extraction layer. The feature extraction layer processes the input image data. Perform feature extraction and obtain feature vector ; S2.2. The feature vector Input the rearrangement layer, perform feature expansion and rearrangement, and obtain the feature expansion vector ; S2.3. Expand vector based on features and the category labels in the base class dataset determine the weight parameters of the fully connected layer , as follows:

[0016]

[0017] in, for The transposed matrix of represents the identity matrix; Represents the inverse operation of a matrix; is the regularization term coefficient; is the autocorrelation matrix; express The corresponding category label; S2.4. According to the weight parameters of the fully connected layer Determine the initial model.

[0018] Furthermore, step S2.2 is specifically as follows: S2.2.1. For feature vectors Perform dimension increase to obtain the feature matrix ; S2.2.2. Transpose the feature matrix , get the transposed feature matrix ; S2.2.3. Perform feature expansion to obtain the expansion matrix , as follows:

[0019] S2.2.4. For the expansion matrix Rearrange and get the feature expansion vector .

[0020] Further, step S4 includes the following steps: S4.1, according to the neural network model selected in step S1, use each new class data set and respectively adopt the model back propagation training method to train the neural network model to obtain multiple target basic models corresponding to each new class data set; S4.2. Order ; S4.3, for Randomly sample the new class data set and get the feature extraction dataset, The category distribution of the feature extraction dataset is similar to that of the The category distribution of the new class dataset is consistent. The data volume of the feature extraction dataset is One tenth of the new class dataset; S4.4. The feature extraction dataset is input into the corresponding The target base model of the new class dataset is obtained target vector; S4.5. According to The target vector is obtained corresponding to The feature fusion vector of the new class data set is as follows: ; in, For the corresponding The feature fusion vector of the new class dataset, is the preset fusion coefficient, For the target vector, For the corresponding feature fusion vector of a new class of data set; hour, is the feature fusion vector of the first new class data set in the corresponding training sequence, For the target vector, is 0; S4.6, if , The calculation ends when the number of new class data sets (any positive integer except 1) is reached, and the feature fusion vector corresponding to each new class data set is obtained. Otherwise, , return to step S4.3.

[0021] Furthermore, the class incremental learning model includes an input layer, a feature extraction layer, a class incremental classifier layer and an output layer; except for the initial classifier layer and the class incremental classifier layer, the structure of the initial model is the same as that of the class incremental learning model; the class incremental classifier layer of the class incremental learning model includes a class incremental rearrangement layer and a fully connected layer connected in sequence; The input layer receives the new class image data in the new class data set, and inputs the new class image data in the new class data set to the feature extraction layer; The feature extraction layer extracts features from the input new class image data to obtain a new class feature vector; In the class increment classifier layer, the new class feature vector passes through the class increment rearrangement layer, outputs the new class feature expansion vector to the fully connected layer, and the fully connected layer outputs the class increment prediction result vector to the output layer according to the new class feature expansion vector; The output layer calculates the class incremental prediction result vector according to a preset softmax function, and outputs each prediction result corresponding to the new class data set; The new class data set includes new class image data and category labels corresponding to the new class image data; Step S5 includes the following steps: S5.1. In the first round, the new class image data in the first new class data set in the training sequence is input into the feature extraction layer of the initial model for feature extraction to obtain the first new class feature vector; S5.2, inputting the first new class feature vector into the class increment rearrangement layer, performing feature expansion and feature fusion, and then obtaining the first new class feature expansion vector by rearrangement; S5.3. Weight parameters for the fully connected layer Expand the dimension and obtain the weight parameters of the fully connected layer after the expansion ; S5.4. According to the weight parameters of the fully connected layer after dimensional expansion , find the fully connected weight of the first class incremental learning model , and then get the first class incremental learning model; S5.5. Order ; S5.6, in In the round, the first The new class image data in the new class dataset is input into the feature extraction layer of the initial model for feature extraction, and the first New class feature vectors; S5.7. The feature vector of the new class is input into the class incremental rearrangement layer for feature expansion and feature fusion, and then the first class is obtained by rearrangement. New class feature expansion vector; S5.8, for Fully connected weights of the incremental learning model for each class Expand the dimension and get the first Fully connected weights of the incremental learning model for each class ; S5.9, according to the expanded Fully connected weights of the incremental learning model for each class , find the Fully connected weights of the incremental learning model for each class , and then get the Class incremental learning model; S5.10: If , The number of new class data sets is calculated and the first The incremental learning model of each class is the incremental learning model of the last round, otherwise , return to step S5.6.

[0022] Furthermore, step S5.2 is specifically as follows: S5.2.1. For the first new class feature vector Perform dimension increase to obtain the first new class feature matrix ; S5.2.2. For the first new class feature matrix Transpose and get ; S5.2.3. Feature fusion vector for the first new class dataset Perform dimension increase to obtain the first new class feature fusion matrix ; S5.2.4. Perform feature expansion and feature fusion to obtain the first new class expansion matrix , as follows:

[0023] in, is the fusion coefficient; S5.2.5. Expanding the matrix for the first new class Rearrange (see the rearrangement process for details) Figure 4 ), get the first new class feature expansion vector ; In step S5.4, according to the weight parameters of the fully connected layer after dimensional expansion , find the fully connected weight of the first class incremental learning model , as follows:

[0024]

[0025] in, is the first new class feature expansion vector, is the transpose of the first new class feature expansion vector, is the category label of the new class image data corresponding to the first new class data set; is the autocorrelation matrix of the first new class.

[0026] Further, step S5.7 is specifically as follows: S5.7.1. New class feature vector By upgrading the dimension, we get New class feature matrix ; S5.7.2, for New class feature matrix Transpose and get ; S5.7.3, for The feature fusion vector of the new class dataset By upgrading the dimension, we get New class feature fusion matrix ; S5.7.4, perform feature expansion and feature fusion to obtain New class expansion matrix , as follows:

[0027] in, is the fusion coefficient; In step S5.9, according to the expanded dimension Fully connected weights of the incremental learning model for each class , find the Fully connected weights of the incremental learning model for each class , as follows:

[0028]

[0029] in, For the New class feature expansion vector, For the The transpose of the feature expansion vector of the new class, For the corresponding Category labels of new class image data in new class datasets; For the A new class of autocorrelation matrices.

[0030] A parsing incremental learning system based on an FPGA development board comprises an FPGA development board and a target model deployed on the FPGA development board and obtained according to a parsing incremental learning method based on the FPGA development board.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows: The present application constructs an initial model through the weight parameters of the feature extraction layer in the extracted basic model and the preset classifier layer parameters, and performs the first round of class incremental learning on the new class data set ranked first in multiple new class data sets according to the initial model. In class incremental learning, the target basic model obtained by training the new class data set input in the current round using the model back-propagation training method can obtain the feature fusion vector, and class incremental learning can be performed without relying on the base class data set, and then by continuously updating the parameters of the next round of class incremental classifier layer, the class incremental learning model of the next round is continuously output, and finally the class incremental learning model of the last round is output, so as to complete the iterative update of the class incremental learning model. The class incremental learning model of the present application solves the problem that the model cannot learn the features of new category images after the weights of the feature extraction layer are frozen by means of feature fusion. In the subsequent class incremental learning process, there is no need to rely on data samples of old categories, thereby effectively protecting data privacy. At the same time, the quasi-incremental learning model of the present application can be quickly deployed on the FPGA development board by optimizing the input method, thereby solving the problem in related technologies that method-like models with high computational complexity are difficult to be deployed in FPGAs, and provides an efficient solution for the application of quasi-incremental learning in real-time and low-power scenarios.

[0032] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The present invention is a flowchart of an incremental learning method for parsing based on an FPGA development board in an embodiment of the present invention.

[0034] Figure 2 This is a model architecture diagram of the basic model in an embodiment of the present invention.

[0035] Figure 3 This is a model architecture diagram of an initial model or a quasi-incremental learning model in an embodiment of the present invention.

[0036] Figure 4 Schematic diagram of expansion and rearrangement in an embodiment of the present invention.

[0037] Figure 5 The figure is a logical diagram of a class incremental learning process performed on an FPGA in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0039] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the present application.

[0040] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or micro-control node devices.

[0041] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0042] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0043] This embodiment provides an incremental learning method for parsing based on an FPGA development board, such as Figure 1 As shown, the following steps are included: S1. Select a neural network model, use a pre-set base class data set, and adopt a model back propagation training method to train the neural network model to obtain a basic model including an input layer, a feature extraction layer, a classifier layer, and an output layer, such as Figure 2 As shown; the base class data set includes image data of multiple categories and category labels corresponding to the image data; In one embodiment, a base class data set is created , Represents the image input of m categories in the base class dataset; express The corresponding label or expected result, is known; like Figure 2 As shown, the basic model is as follows:

[0044] in, represents the image to be detected; Indicates the prediction result; Represents the weight parameters of the feature extraction layer; Represents the weight parameters of the classifier layer; Represents feature extraction operation; Represents a flattening operation; Represents softmax classification operation; The feature extraction layer formula is as follows:

[0045] in, Represents the output of the feature extraction layer; The classifier layer formula is as follows:

[0046] in, Represents the output of the classifier layer.

[0047] In one embodiment, the selected neural network model is a residual convolutional neural network.

[0048] S2, freezing the weight parameters of the feature extraction layer of the base model, parsing and rearranging the base model through the base class data set, and obtaining an initial model including an initial classifier layer as the target model; like Figure 3 As shown, the initial model includes an input layer, a feature extraction layer, an initial classifier layer and an output layer; except for the classifier layer and the initial classifier layer, the structure of the initial model is the same as that of the basic model; the initial classifier layer of the initial model includes a rearrangement layer and a fully connected layer connected in sequence; The input layer receives the image data in the base class data set, and inputs the image data in the base class data set to the feature extraction layer; The feature extraction layer extracts features from the input image data to obtain a feature vector; In the initial classifier layer, the feature vector passes through the rearrangement layer, outputs the feature expansion vector to the fully connected layer, and the fully connected layer outputs the prediction result vector to the output layer according to the feature expansion vector; The output layer calculates the prediction result vector according to a preset softmax function, and outputs each prediction result corresponding to the base class data set; Step S2 specifically includes the following steps: S2.1. Image data in the base class dataset Input feature extraction layer, the feature extraction layer input image data Perform feature extraction and obtain feature vector , as follows:

[0049] in: , Represents the total number of images in the base class dataset; Represents the feature dimension after flattening; S2.2. The feature vector Input the rearrangement layer, perform feature expansion and rearrangement, and obtain the feature expansion vector , as follows: S2.2.1. For feature vectors Perform dimension increase to obtain the feature matrix , the shape is ( ,1, ); S2.2.2. Transpose the feature matrix , get the transposed feature matrix , the shape is ( , , 1); S2.2.3. Perform feature expansion to obtain the expansion matrix , the shape is ( , , ), as follows:

[0050] S2.2.4. For the expansion matrix Rearrange and get the feature expansion vector , the shape is ( , ); In one embodiment, the rearrangement process is as follows Figure 4 As shown, the details are as follows: Take the expansion matrix The upper triangular elements of the , in row order, are placed in the first row to form the feature expansion vector ; The upper triangular elements include all elements on the main diagonal of the matrix and to its right.

[0051] S2.3. Expand vector based on features and the category labels in the base class dataset determine the weight parameters of the fully connected layer , the shape is ( , ), as follows:

[0052]

[0053] in, for The transposed matrix of represents the identity matrix; Represents the inverse operation of a matrix; is the regularization term coefficient. In one embodiment, Take 0.1; is the autocorrelation matrix; express The corresponding category label; S2.4. According to the weight parameters of the fully connected layer Determine the initial model, the formula is as follows:

[0054] →

[0055]

[0056] →

[0057]

[0058] in, represents the output of the feature extraction layer, Express The feature matrix obtained by dimension upgrading is: Indicates based on The expansion matrix obtained by feature expansion is Express Rearrange the feature vector.

[0059] S3, determine whether there are multiple new class data sets to be incrementally learned, if so, set a training order for each new class data set and execute step S4, otherwise execute step S6; The new class data set includes new class image data and category labels corresponding to the new class image data; In one embodiment, establishing New Class Dataset , , << , There are no images of the same category between the new class datasets; Indicates Image input of n categories in a new category dataset; express The corresponding label or expected result, It is known that there are no images of the same category between the new class dataset and the base class dataset; S4, performing feature extraction and feature fusion on each new class data set to obtain a feature fusion vector corresponding to each new class data set, including the following steps: S4.1. According to the neural network model selected in step S1, use New class data sets are used to train the neural network model using the model back propagation training method. The new class dataset corresponds to A target base model; S4.2. Order ; S4.3, for Randomly sample the new class data set and get the feature extraction dataset, The category distribution of the feature extraction dataset is similar to that of the The category distribution of the new class dataset is consistent. The data volume of the feature extraction dataset is One tenth of the new class dataset; S4.4. The feature extraction dataset is input into the corresponding The target basic model of the new class data set is used to obtain the feature vector of each image, and then the average is calculated to obtain the first Target vector , as follows:

[0060]

[0061] in: Indicates The first image in the feature extraction dataset; Indicates Extract the feature vector of the first image in the feature extraction dataset; Indicates The feature extraction dataset The feature vector of the image; Indicates The total number of images in the feature extraction dataset; S4.5. According to Target vector Get the corresponding The feature fusion vector of the new class dataset , as follows: ; in, For the corresponding The feature fusion vector of the new class dataset, is a preset fusion coefficient. In one embodiment, Take 0.9, For the target vector, For the corresponding feature fusion vector of a new class of data set; hour, is the feature fusion vector of the first new class data set in the corresponding training sequence, For the target vector, is 0; S4.6, if , The calculation ends when the number of new class data sets (any positive integer except 1) is reached, and the feature fusion vector corresponding to each new class data set is obtained. Otherwise, , return to step S4.3.

[0062] S5, performing multiple rounds of class incremental learning on the initial model according to the new class data sets and the feature fusion vectors corresponding to each new class data set in the training order, and outputting the class incremental learning model of the last round as the target model; The class incremental learning model includes an input layer, a feature extraction layer, a class incremental classifier layer and an output layer; except for the initial classifier layer and the class incremental classifier layer, the structure of the initial model is the same as that of the class incremental learning model; the class incremental classifier layer of the class incremental learning model includes a class incremental rearrangement layer and a fully connected layer connected in sequence; The input layer receives the new class image data in the new class data set, and inputs the new class image data in the new class data set to the feature extraction layer; The feature extraction layer extracts features from the input new class image data to obtain a new class feature vector; In the class increment classifier layer, the new class feature vector passes through the class increment rearrangement layer, outputs the new class feature expansion vector to the fully connected layer, and the fully connected layer outputs the class increment prediction result vector to the output layer according to the new class feature expansion vector; The output layer calculates the class incremental prediction result vector according to a preset softmax function, and outputs each prediction result corresponding to the new class data set; Step S5 includes the following steps: S5.1. In the first round, the new class image data in the first new class dataset in the training sequence Input the feature extraction layer of the initial model to extract features and obtain the first new class feature vector , as follows:

[0063] in, ; Represents the total number of images in the first new class dataset; Represents the feature dimension after flattening; S5.2. The first new class feature vector Input class incremental rearrangement layer, perform feature expansion and feature fusion, and then obtain the first new class feature expansion vector through rearrangement , as follows: S5.2.1. For the first new class feature vector Perform dimension increase to obtain the first new class feature matrix , the shape is ( ,1, ); S5.2.2. For the first new class feature matrix Transpose and get , the shape is ( , , 1); S5.2.3. Feature fusion vector for the first new class dataset Perform dimension increase to obtain the first new class feature fusion matrix , the shape is ( ,1, ); S5.2.4. Perform feature expansion and feature fusion to obtain the first new class expansion matrix , the shape is ( , , ), as follows:

[0064] in, is the fusion coefficient. In one embodiment, Take 0.95; S5.2.5. Expanding the matrix for the first new class Rearrange (see the rearrangement process for details) Figure 4 ), get the first new class feature expansion vector , the shape is ( , ); S5.3. Weight parameters for the fully connected layer Expand the dimension and obtain the weight parameters of the fully connected layer after the expansion , the shape is ( , ); S5.4. According to the weight parameters of the fully connected layer after dimensional expansion , find the fully connected weight of the first class incremental learning model , whose shape is ( , ), and then get the first class incremental learning model, as follows:

[0065]

[0066] in, is the first new class feature expansion vector, is the transpose of the first new class feature expansion vector, is the category label of the new class image data corresponding to the first new class data set; is the first new class autocorrelation matrix; The first class incremental learning model has the following formula:

[0067] →

[0068]

[0069] →

[0070]

[0071] in, Represents the feature fusion vector of the first new class data set The first new class feature fusion matrix obtained by dimension upgrading is: According to and The first new type expansion matrix is ​​obtained by performing feature expansion and feature fusion; Represents the expansion matrix for the first new class The first new class feature expansion vector is obtained by rearrangement.

[0072] S5.5. Order ; S5.6, in In the round, the first The new class image data in the new class dataset is input into the feature extraction layer of the initial model for feature extraction, and the first New class feature vectors; S5.7. The feature vector of the new class is input into the class incremental rearrangement layer for feature expansion and feature fusion, and then the first class is obtained by rearrangement. The new class feature expansion vector is as follows: S5.7.1. New class feature vector By upgrading the dimension, we get New class feature matrix , the shape is ( , , ), Indicates The total number of images in the new class dataset; S5.7.2, for New class feature matrix Transpose and get ; S5.7.3, for The feature fusion vector of the new class dataset By upgrading the dimension, we get New class feature fusion matrix ; S5.7.4, perform feature expansion and feature fusion to obtain New class expansion matrix , the shape is ( , , ), as follows:

[0073] in, is the fusion coefficient; S5.8, for Fully connected weights of the incremental learning model for each class Expand the dimension and get the first Fully connected weights of the incremental learning model for each class ; S5.9, according to the expanded Fully connected weights of the incremental learning model for each class , find the Fully connected weights of the incremental learning model for each class , whose shape is ( , ), and then get the The incremental learning model of each class is as follows:

[0074]

[0075] in, For the New class feature expansion vector, For the The transpose of the feature expansion vector of the new class, For the corresponding Category labels of new class image data in new class datasets; For the A new class of autocorrelation matrices; S5.10: If , The calculation ends with the number of new class data sets (any positive integer except 1), and the first The incremental learning model of each class is the incremental learning model of the last round, otherwise , return to step S5.6; The final round of incremental learning model is as follows:

[0076] →

[0077]

[0078] →

[0079]

[0080] in, For the The fully connected weights of the incremental learning model for each class; Indicates The feature fusion vector of the new class dataset The first New class feature fusion matrix, Indicates based on and The first A new class of expansion matrices; Indicates New class expansion matrix Rearrange the obtained The feature expansion vector of the new class.

[0081] S6. Input the image to be detected into the target model to obtain the classification label of the image; return to step S3.

[0082] A parsing incremental learning system based on an FPGA development board comprises an FPGA development board and a target model deployed on the FPGA development board and obtained according to a parsing incremental learning method based on the FPGA development board.

[0083] In one embodiment, Figure 5As shown, the target model is quantized to obtain the quantized quantization weight value and the shift factor of the activation value; an FPGA development board is selected, and the global architecture of the adaptive network model is designed according to the quantization weight value, the shift factor of the activation value and the preset FPGA development board, and the relevant hardware resource configuration is added to the device tree, and the bit stream of the PL (Progarmmable Logic) end is integrated with the Linux kernel to generate a complete Linux system image that can be run on the FPGA development board, and the generated Linux system image is burned into the NAND Flash of the FPGA development board to obtain the target FPGA development board with the target model deployed; When the target FPGA development board with the target model deployed updates the model parameters, the parameters to be updated are loaded from the NAND Flash to the BRAM (Block Random Access Memory) on the PL side, and then the images in the new data set are transferred to the FPGA development board one by one. The incremental learning of the round is reflected by the following formula:

[0084]

[0085] in, for The new autocorrelation matrix in the first round of the quasi-incremental learning of the round; For the A new class of autocorrelation matrices; is the feature expansion vector in the first round, is the feature expansion vector The transposed matrix of For the The new autocorrelation matrix in the round, For the A new autocorrelation matrix in rounds; For the Feature expansion vector in round; is the feature expansion vector The transposed matrix of For the corresponding The adaptive scalar For the corresponding The adaptive scalar is as follows:

[0086]

[0087] The collected images to be detected are transmitted to the target FPGA development board where the target model is deployed after the model parameters are updated for image classification.

[0088] Since FPGA does not need to perform inverse operations but only needs to perform division operations, the difficulty of updating and deploying the quasi-incremental learning model is greatly reduced, making it possible for the quasi-incremental learning model deployed on the FPGA development board to easily achieve incremental learning.

[0089] Although FPGA can perform matrix inversion operations, due to the large amount of calculations, a lot of computing resources will be consumed during execution. If a single image is input, C becomes a scalar, and the matrix inversion operation becomes a division operation. Therefore, the solution idea of ​​the embodiment of the present application is: there is an inversion algorithm in the target class incremental model → FPGA inversion operation will consume a lot of computing resources, and even the model cannot be deployed → if a single image is input (that is, class incremental learning is performed image by image), the inversion operation becomes a division operation → reduce the difficulty of model deployment and update.

[0090] In this way, whenever the model needs to add new categories of image knowledge, it can easily perform incremental learning on the FPGA development board without relying on the original data set for retraining and redeployment, thus solving the problems of cumbersome training process, high training cost and low training efficiency in the existing technology.

[0091] The analytical incremental learning method based on FPGA development board provided in the embodiment of the present application can effectively solve the problems described in the background technology. Compared with the traditional method, the analytical incremental learning method based on FPGA development board provided in the embodiment of the present application can not only effectively alleviate catastrophic forgetting, but also protect data privacy and ensure that the model can be updated without relying on the original data. The method proposed in the embodiment of the present application can maintain high reasoning ability and stability when dealing with complex application scenarios that are constantly changing, and provides stronger long-term adaptability for deep learning models. In addition, the analytical incremental learning system based on FPGA development board provided in the embodiment of the present application proposes an innovative method to convert the inversion operation in the incremental learning model into a division operation, making it possible to deploy the incremental learning model to a resource-constrained FPGA platform. Although FPGA can implement inversion operations, due to high computational complexity, large resource consumption, slow speed and possible loss of precision, it is usually recommended to avoid directly performing complex inversion operations in hardware design.

[0092] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0093] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be performed without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A parsing incremental learning method based on FPGA development board, characterized in that: The following steps are involved: S1. Select a neural network model, use a preset base class data set to train the neural network model, and obtain a basic model including an input layer, a feature extraction layer, a classifier layer, and an output layer; S2, freezing the weight parameters of the feature extraction layer of the base model, parsing and rearranging the base model through the base class data set, and obtaining an initial model including an initial classifier layer as the target model; S3, determine whether there are multiple new class data sets to be incrementally learned, if so, set a training order for each new class data set and execute step S4, otherwise execute step S6; S4, performing feature extraction and feature fusion on each new class data set to obtain a feature fusion vector corresponding to each new class data set; S5, performing multiple rounds of class incremental learning on the initial model according to the new class data sets and the feature fusion vectors corresponding to each new class data set in the training order, and outputting the class incremental learning model of the last round as the target model; S6. Input the image to be detected into the target model to obtain the classification label of the image; return to step S3.

2. The incremental learning method for parsing based on FPGA development board according to claim 1, characterized in that: The base class dataset includes images of various categories and the corresponding category labels of the images for training the feature extraction layer; The new class dataset includes images of multiple categories and their corresponding category labels for class incremental learning; There are no images of the same category between the new class dataset and the base class dataset; There are no images of the same category between multiple new class datasets.

3. The incremental learning method for parsing based on FPGA development board according to claim 1, characterized in that: In step S2, the initial model includes an input layer, a feature extraction layer, an initial classifier layer and an output layer; except for the classifier layer and the initial classifier layer, the structure of the initial model is the same as that of the basic model; the initial classifier layer includes a rearrangement layer and a fully connected layer connected in sequence; The input layer receives the image data in the base class data set, and inputs the image data in the base class data set to the feature extraction layer; The feature extraction layer extracts features from the input image data to obtain a feature vector; In the initial classifier layer, the feature vector passes through the rearrangement layer, outputs the feature expansion vector to the fully connected layer, and the fully connected layer outputs the prediction result vector to the output layer according to the feature expansion vector; The output layer calculates the prediction result vector according to a preset softmax function, and outputs each prediction result corresponding to the base class data set; The base class data set includes image data of multiple categories and category labels corresponding to the image data.

4. The incremental learning method for parsing based on FPGA development board according to claim 3, characterized in that: Step S2 specifically includes the following steps: S2.

1. Image data in the base class dataset Input feature extraction layer, the feature extraction layer input image data Perform feature extraction and obtain feature vector ; S2.

2. The feature vector Input the rearrangement layer, perform feature expansion and rearrangement, and obtain the feature expansion vector ; S2.

3. Expand vector based on features and the category labels in the base class dataset determine the weight parameters of the fully connected layer , as follows: in, for The transposed matrix of represents the identity matrix; Represents the inverse operation of a matrix; is the regularization term coefficient; is the autocorrelation matrix; express The corresponding category label; S2.

4. According to the weight parameters of the fully connected layer Determine the initial model.

5. The incremental learning method for parsing based on FPGA development board according to claim 4, characterized in that: Step S2.2 is as follows: S2.2.

1. For feature vectors Perform dimension increase to obtain the feature matrix ; S2.2.

2. Transpose the feature matrix , get the transposed feature matrix ; S2.2.

3. Perform feature expansion to obtain the expansion matrix , as follows: S2.2.

4. For the expansion matrix Rearrange and get the feature expansion vector .

6. The incremental learning method for parsing based on FPGA development board according to claim 1, characterized in that: Step S4 includes the following steps: S4.1, according to the neural network model selected in step S1, use each new class data set to train the neural network model respectively, and obtain multiple target basic models corresponding to each new class data set; S4.

2. Order ; S4.3, for Randomly sample the new class data set and get the feature extraction dataset, The category distribution of the feature extraction dataset is similar to that of the The category distribution of the new class dataset is consistent; S4.

4. The feature extraction dataset is input into the corresponding The target base model of the new class dataset is obtained target vector; S4.

5. According to The target vector is obtained corresponding to The feature fusion vector of the new class data set is as follows: ； in, For the corresponding The feature fusion vector of the new class dataset, is the preset fusion coefficient, For the target vector, For the corresponding feature fusion vector of a new class of data set; hour, is the feature fusion vector of the first new class data set in the corresponding training sequence, For the target vector, is 0; S4.6, if , The number of new class data sets is calculated to obtain the feature fusion vector corresponding to each new class data set. Otherwise, , return to step S4.

3.

7. The incremental learning method for parsing based on FPGA development board according to claim 1, characterized in that: In step S5, the class incremental learning model includes an input layer, a feature extraction layer, a class incremental classifier layer and an output layer; except for the initial classifier layer and the class incremental classifier layer, the structure of the initial model is the same as the structure of the class incremental learning model; the class incremental classifier layer includes a class incremental rearrangement layer and a fully connected layer connected in sequence; The input layer receives the new class image data in the new class data set, and inputs the new class image data in the new class data set to the feature extraction layer; The feature extraction layer extracts features from the input new class image data to obtain a new class feature vector; In the class increment classifier layer, the new class feature vector passes through the class increment rearrangement layer, outputs the new class feature expansion vector to the fully connected layer, and the fully connected layer outputs the class increment prediction result vector to the output layer according to the new class feature expansion vector; The output layer calculates the class incremental prediction result vector according to a preset softmax function, and outputs each prediction result corresponding to the new class data set; The new class data set includes new class image data of multiple categories and category labels corresponding to the new class image data.

8. The incremental learning method for parsing based on FPGA development board according to claim 7, characterized in that: Step S5 includes the following steps: S5.

1. In the first round, the new class image data in the first new class data set in the training sequence is input into the feature extraction layer of the initial model for feature extraction to obtain the first new class feature vector; S5.2, inputting the first new class feature vector into the class increment rearrangement layer, performing feature expansion and feature fusion, and then obtaining the first new class feature expansion vector by rearrangement; S5.

3. Weight parameters for the fully connected layer Expand the dimension and obtain the weight parameters of the fully connected layer after the expansion ; S5.

4. According to the weight parameters of the fully connected layer after dimensional expansion , find the fully connected weight of the first class incremental learning model , and then get the first class incremental learning model; S5.

5. Order ; S5.6, in In the round, the first The new class image data in the new class dataset is input into the feature extraction layer of the initial model for feature extraction, and the first New class feature vectors; S5.

7. The feature vector of the new class is input into the class incremental rearrangement layer for feature expansion and feature fusion, and then the first class is obtained by rearrangement. New class feature expansion vector; S5.8, for Fully connected weights of the incremental learning model for each class Expand the dimension and get the first Fully connected weights of the incremental learning model for each class ; S5.9, according to the expanded Fully connected weights of the incremental learning model for each class , find the Fully connected weights of the incremental learning model for each class , and then get the Class incremental learning model; S5.10: If , The number of new class data sets is calculated and the first The incremental learning model of each class is the incremental learning model of the last round, otherwise , return to step S5.

6.

9. The incremental learning method for parsing based on FPGA development board according to claim 8, characterized in that: Step S5.2 is as follows: S5.2.

1. For the first new class feature vector Perform dimension increase to obtain the first new class feature matrix ; S5.2.

2. For the first new class feature matrix Transpose and get ; S5.2.

3. Feature fusion vector for the first new class dataset Perform dimension increase to obtain the first new class feature fusion matrix ; S5.2.

4. Perform feature expansion and feature fusion to obtain the first new class expansion matrix , as follows: in, is the fusion coefficient; S5.2.

5. Expanding the matrix for the first new class Rearrange and get the first new class feature expansion vector ; In step S5.4, according to the weight parameters of the fully connected layer after dimensional expansion , find the fully connected weight of the first class incremental learning model , as follows: in, is the first new class feature expansion vector, is the transpose of the first new class feature expansion vector, is the category label of the new class image data corresponding to the first new class data set; is the first new class autocorrelation matrix; Step S5.7 is as follows: S5.7.

1. New class feature vector By upgrading the dimension, we get New class feature matrix ; S5.7.2, for New class feature matrix Transpose and get ; S5.7.3, for The feature fusion vector of the new class dataset By upgrading the dimension, we get New class feature fusion matrix ; S5.7.4, perform feature expansion and feature fusion to obtain New class expansion matrix , as follows: in, is the fusion coefficient; In step S5.9, according to the expanded dimension Fully connected weights of the incremental learning model for each class , find the Fully connected weights of the incremental learning model for each class , as follows: in, For the New class feature expansion vector, For the The transpose of the feature expansion vector of the new class, For the corresponding Category labels of new class image data in new class datasets; For the A new class of autocorrelation matrices.

10. An incremental learning system for analytical applications based on an FPGA development board, characterized in that: It comprises an FPGA development board and a target model obtained by the analytic incremental learning method based on the FPGA development board according to any one of claims 1 to 9 and deployed on the FPGA development board.

Images