A continuous image classification model training method and system for achieving zero-sample replay and minimalistic expansion

Abstract

The present invention belongs to the technical field related to image classification, and discloses a continuous image classification model training method and system for realizing zero-sample replay and simple expansion. The training method comprises: by solving the optimization objective function of the unit basis vector for the current classification task, allocating the optimal equal-angle basis vector matrix of the current classification task from the unit hypersphere and adding it to the equal-angle basis vector matrix set storing the equal-angle basis vector matrix of the historical classification task to update the set; in the forward propagation of the training, determining the optimal global output representation of each hidden layer by solving the problem of minimizing statistical dependence; in the backward propagation of the training, using the gradient descent method to feedback and adjust the network parameters of each hidden layer. The present invention can achieve a balance between stability and plasticity in the continuous image classification task, has the characteristics of zero-sample replay and simple expansion, and greatly improves the training efficiency by improving the optimization problem of each step.

Description

Technical Field

[0001] The present invention belongs to the technical field related to image classification, and more specifically, relates to a continuous image classification model training method and system for realizing zero-sample replay and simplified expansion. Background Art

[0002] Artificial intelligence technology led by deep learning has achieved remarkable success in classification tasks such as image classification, speech recognition and natural language processing, generating huge economic and social benefits. However, the key condition for its success is to place the model in a relatively closed and static environment. A common closed environment assumption is that the training set and the test set are from the same probability distribution, that is, the independent and identically distributed condition. However, the data flow in the real world causes catastrophic forgetting in traditional single-classification task learning methods, that is, they fall into the dilemma of stability (ability to retain knowledge of old classification tasks) and plasticity (ability to learn knowledge of new classification tasks). Continuous learning effectively avoids the dependence of training and test data on independent and identically distributed conditions, and has made extensive progress in processing continuous image classification tasks.

[0003] Continuous learning is to continue training new classification tasks based on the historically trained model, so that the model can retain the existing classification function and increase the classification function of the new classification task. The main methods of continuous learning are replay-based continuous learning method and expansion-based continuous learning method.

[0004] Replay-based continuous learning methods assume that a small amount of original samples of old categories can be stored or rely on additional generation modules to generate pseudo samples for joint training when learning new categories. Among them, original replay is a direct and effective way, whether used alone or combined with other methods, and has achieved competitive classification accuracy. However, the performance of such methods is heavily dependent on the quantity and quality of replay samples. For example, a small buffer of old samples means a lack of information specific to the classification task. Moreover, always storing old samples also involves practical issues such as violating data privacy, and privacy protection is increasingly valued.

[0005] Continuous learning methods based on expansion gradually increase the size of the network (such as the backbone or branches) to accommodate new categories. When resources are not limited, the classification accuracy can be pushed to the upper bound of continuous learning, that is, close to the performance of offline batch learning. However, if the final network size is comparable to multiple single-classification task learning networks, that is, the final model size is equivalent to integrating several (e.g., equal to the number of classification tasks) base models, this goes against the original intention of continuous learning and should be calculated into the memory budget for fair comparison. At the same time, the model capacity required to learn new categories must be carefully balanced with the number of parameters introduced, especially considering that the number of classification tasks / categories that the model needs to learn is usually unknown in advance, and base networks that are too large or too small cannot match the requirements of resource conservation in the real world. Summary of the invention

[0006] In response to the above defects or improvement needs of the prior art, the present invention provides a continuous image classification model training method and system that realizes zero-sample replay and simplified expansion, which aims to solve the technical problems that the existing methods need to access old classification task samples and consume storage resources without restriction during the continuous image classification training process.

[0007] To achieve the above object, the present invention provides a continuous image classification model training method for achieving zero-sample replay and simplified expansion, which comprises:

[0008] By solving the optimization objective function for the unit basis vector of the current classification task, the optimal C is allocated from the unit hypersphere t unit basis vectors and form the optimal equal-angle basis vector matrix W for the current classification task t (t)’ , t is the classification task number, C t Equal to the number of label categories of the current classification task t, the matrix W (t)’ Add the equal-angle basis vector matrix set that stores the equal-angle basis vector matrix of the historical classification task to obtain an updated equal-angle basis vector matrix set;

[0009] In the forward propagation of training, the optimal global output representation Z of each hidden layer b is determined by solving the problem of minimizing statistical dependence b ’ , subscript b is the hidden layer index, b=1,2,…,B, B is the number of hidden layers; the problem of minimizing statistical dependence is to minimize the weighted sum of multiple factors, the multiple factors include a first factor for calculating the correlation between the output representation of each hidden layer and the input representation of the current classification task training sample, a second factor for calculating the inverse of the correlation between the output representation of each hidden layer and the label of the current classification task training sample, and a third factor for calculating the product between the gradient mapping matrix and the hidden layer history representation determined by the previous training;

[0010] In the back propagation of training, obtain the updated equal-angle basis vector matrix set and the best global output representation Z b ’ , the gradient descent method is used to determine the parameter update amount of the network parameters, and the batch learning form is used to iteratively calculate the update value of the gradient mapping matrix. Based on the obtained parameter update amount and the update value of the gradient mapping matrix, the network parameters of each hidden layer are adjusted to obtain the training results.

[0011] Optionally, the problem of minimizing statistical dependence is specifically:

[0012] ,

[0013] Where, X (t) Y is the input representation of the training sample of the current classification task t, (t) is the label of the training sample of the current classification task t, Z b is the output representation of the hidden layer b in the forward propagation of training, uHSIC is the function for calculating the correlation, R b uHSIC =α(A b T A b +αI) -1 is the gradient mapping matrix, α is a constant less than 1, I is the unit matrix, A b It is composed of the historical representation of the hidden layer b determined by the previous training, β is the balance factor of the correlation factor, and μ is the regularization strength coefficient.

[0014] Optionally, in the forward propagation, the first factor and the second factor perform an unbiased estimation of the correlation based on the Hilbert-Schmidt independence criterion, and the calculation formula for calculating any correlation between E and F is:

[0015] ,

[0016] Where uHSIC(E,F) represents the correlation between E and F, n is the number of training samples, tr(·) is the trace operator, and the matrix is the intermediate parameter calculated based on the kernel matrix H, the matrix The elements in the matrix is the intermediate parameter calculated based on the kernel matrix G, the matrix The elements in δ ij is the Kronecker coefficient, the kernel matrix H is obtained by performing nonlinear feature transformation on the representation E in the Hilbert space, and the kernel matrix G is obtained by performing nonlinear feature transformation on the representation F in the Hilbert space. The subscripts i and j are the row and column indices of the matrix, i, j = 1, 2, 3, ..., n, I n is the n×n identity matrix.

[0017] Optionally, 400≤β≤600, 9000≤μ≤15000, and 0.005≤α≤0.015.

[0018] Optionally, in the back propagation of training, the parameter update amount of the network parameters is determined by using the gradient descent method, the update value of the gradient mapping matrix is ​​iteratively calculated by using the batch learning form, and the network parameters of each hidden layer are adjusted based on the obtained parameter update amount and the update value of the gradient mapping matrix, including:

[0019] First obtain the updated equal angle basis vector matrix set and the best global output representation Z b ’ , construct and solve the optimization objective function that makes the classification result close to the label, and obtain the parameter update amount of the network parameter gradient descent;

[0020] Then update the network parameters θ obtained by training the current classification task t based on the network parameter update formula b (t) ;

[0021] Among them, the network parameter update formula is:

[0022] ,

[0023] In the formula, θ b (t-1) is the network parameter obtained from the last classification task t-1 training, is the parameter update amount of the current classification task t, λ is the learning rate, M b uHSIC is the updated value of the gradient mapping matrix;

[0024] Where M b uHSIC Determined through the following process:

[0025] Determine the initial value of k and extract the best global representation Z b ’ The feature vectors corresponding to the first k training samples in the initialization representation Z b ’ (k)=[z b 1 ,z b 2 ,…,z b k ], where z b k Represents Z b ’ The feature vector corresponding to the kth training sample;

[0026] Determine M buHSIC The initial value of the iteration M b uHSIC (k+1), M b uHSIC (k+1)=α(Z b ’ (k)+Z b ’T (k)+ z b k+1 (z b k+1 ) T +

[0027] αI) -1 , α is a constant less than 1, and I is the unit matrix;

[0028] In each iteration, k=k+1 is set and iterative calculation is performed based on the iterative formula. After executing the preset number of iterations, the final output is M b uHSIC (k+1) is the update amount M of the gradient mapping matrix b uHSIC , and its iteration formula is:

[0029] .

[0030] Optionally, constructing an optimization objective function that makes the classification result approach the label includes: constructing the optimization objective function of the current classification task t in the form of minimizing the negative log-likelihood function.

[0031] Optionally, the optimization objective function for the unit basis vector of the current classification task is specifically:

[0032] ,

[0033] Where W (t) is the equal angle basis vector matrix of the current classification task t, is the unit basis vector w i Normalized value, is the unit basis vector w j The normalized value of , γ∈[0,1) is the common angle used to separate the unit basis vectors in pairs;

[0034] After obtaining the optimal equal-angle basis vector matrix W (t)’ Then, the obtained matrix W (t)’ Add and save to the equal angle basis vector matrix set to obtain the updated equal angle basis vector matrix set. The updated equal angle basis vector matrix set is Ω 1:t =[W (1)’ , W (2)’ ,…, W (t-1)’ , W(t)’ ].

[0035] The present invention also provides a continuous image classification model training system for achieving zero-sample replay and simplified expansion, which comprises:

[0036] The equal angle basis vector matrix update unit is used to solve the optimization objective function for the unit basis vector of the current classification task and allocate the optimal C from the unit hypersphere. t unit basis vectors and form the optimal equal-angle basis vector matrix W for the current classification task t (t)’ , t is the classification task number, C t Equal to the number of label categories of the current classification task t, the matrix W (t)’ Add the equal-angle basis vector matrix set that stores the equal-angle basis vector matrix of the historical classification task to obtain an updated equal-angle basis vector matrix set;

[0037] The forward propagation control unit is used to determine the optimal global output representation Z of each hidden layer b by solving the problem of minimizing statistical dependence in the forward propagation of training. b ’ , subscript b is the hidden layer index, b=1,2,…,B, B is the number of hidden layers; the problem of minimizing statistical dependence is to minimize the weighted sum of multiple factors, the multiple factors include a first factor for calculating the correlation between the output representation of each hidden layer and the input representation of the current classification task training sample, a second factor for calculating the inverse of the correlation between the output representation of each hidden layer and the label of the current classification task training sample, and a third factor for calculating the product between the gradient mapping matrix and the hidden layer history representation determined by the previous training;

[0038] Back propagation control unit, in the back propagation of training, obtains the updated equal angle basis vector matrix set and the best global output representation Z b ’ , the gradient descent method is used to determine the parameter update amount of the network parameters, and the batch learning form is used to iteratively calculate the update value of the gradient mapping matrix. Based on the obtained parameter update amount and the update value of the gradient mapping matrix, the network parameters of each hidden layer are adjusted to obtain the training results.

[0039] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0040] The present invention also provides a computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the steps of any of the above methods are implemented.

[0041] In general, compared with the prior art, the above technical solution conceived by the present invention mainly has the following beneficial effects.

[0042] 1. The present invention can reduce representation bias by constructing a minimized statistical dependency problem during forward propagation to avoid using a cross entropy loss function like a traditional continuous image classification algorithm, and the gradient mapping matrix therein can learn all possible representations from previous tasks, which means that representations with weaker discriminability for the current task are also fully considered, thereby improving the accuracy of training. Moreover, the present invention uses batch learning to iteratively calculate the updated value of the gradient mapping matrix, and adjusts the network parameters of each hidden layer based on the obtained parameter update amount and the updated value of the gradient mapping matrix. This process does not require storage and replay of historical task samples, thus achieving zero-sample replay and ensuring the data privacy of historical task samples.

[0043] 2. The minimization of statistical dependence problem constructed by the present invention includes calculating the third factor of the product between the gradient mapping matrix and the hidden layer historical representation determined by the previous training. Compared with the bottleneck problem that may arise from setting the third factor equal to 0 in the constraint condition and having no solution, the minimization of statistical dependence problem constructed by the present invention is easier to solve during calculation.

[0044] 3. Different from the existing methods which usually determine all possible output categories by experience before training, which has a large amount of calculation and affects the training speed, the present invention adopts equal-angle basis vectors to achieve the scalability of classification tasks. Moreover, each time a new task is trained, it is only necessary to allocate equal-angle basis vectors on the unit hypersphere according to the number of categories of the current task, and then the newly added equal-angle basis vectors and the equal-angle basis vectors determined by the historical training tasks are combined into the equal-angle basis vector set of the model. The number of equal-angle basis vectors in this set is consistent with the total number of label categories of all tasks so far. In the continuous image classification process, the network has the advantage of efficient storage, which can better meet the actual needs of limited storage and realize the simple expansion of model output.

[0045] 4. Furthermore, in one embodiment of the present invention, a specific calculation formula for calculating the correlation is given. Calculating the correlation based on the calculation formula can overcome the deviation value existing in the existing HSIC-based biased empirical evaluation method, thereby better evaluating the statistical correlation between the neural network representation learning layers and improving the quality of the global representation.

[0046] 5. Furthermore, in one embodiment of the present invention, M in the network parameter update formula is given. b uHSIC In the process of determining M, since there is a third factor in the problem of minimizing statistical dependence, which is the product of the gradient mapping matrix and the hidden layer history representation determined by the previous training, buHSIC When M is obtained, it is only necessary to iterate the calculation for a preset number of times. b uHSIC Then, we can determine M b uHSIC The process is simpler and no additional constraints need to be set for M b uHSIC The solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flowchart of the steps of a continuous image classification model training method in one embodiment of the present invention.

[0048] Figure 2 In one embodiment of the present invention, b uHSIC The change of rank as the number of tasks increases. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0050] The present invention proposes a continuous image classification model training method that realizes zero-sample replay and simplified expansion, such as Figure 1 The figure is a flow chart of the steps of a continuous image classification model training method in one embodiment of the present invention. It should be noted that the sequence of the steps is only an example, but is not limited to this. Each step is described in detail below.

[0051] Step S1: By solving the optimization objective function for the unit basis vector of the current classification task, the optimal C is allocated from the unit hypersphere t unit basis vectors and form the optimal equal-angle basis vector matrix W for the current classification task t (t)’ , t is the classification task number, C t Equal to the number of label categories of the current classification task t, the matrix W (t)’ The equal-angle basis vector matrix set is added to the equal-angle basis vector matrix set that stores the equal-angle basis vector matrices of the historical classification tasks to obtain an updated equal-angle basis vector matrix set.

[0052] For example, 5 training sample sets are obtained, each sample set has several training samples, and each sample set corresponds to a classification task. After 5 consecutive trainings, the image classification model is required to be able to identify 5 classification tasks. Different classification tasks have different requirements. For example, one classification task is to identify animals, and the other classification task is to identify plants. Through continuous learning, the model is required to have the ability to identify both animals and plants.

[0053] Specifically, according to the number of output label categories C of the current classification task t t , determine the need to incrementally allocate C on the unit hypersphere t d-dimensional unit basis vectors ; The basis vectors need to be separated in pairs by a common angle γ∈[0,1), that is, they belong to the set Any basis vector of Needs to be satisfied ,in represents the Euclidean norm.

[0054] In order to make the network expansion close to the initial model in the continuous image classification process, a strategy of pre-defining the weights of the neural network classification head is adopted. In the present invention, a set of equal-angle basis vector matrices is specifically defined, which contains the equal-angle basis vector matrices obtained after each historical classification task training. When training the current classification task t, the current set of equal-angle basis vector matrices already contains the optimal equal-angle basis vector matrices corresponding to the historical classification tasks 1, 2, ..., t-1. The current set of equal-angle basis vector matrices can be denoted as Ω 1:t =[W (1)’ , W (2)’ ,…, W (t-1)’ , W (t)’ ]. When training the current classification task t, it is only necessary to determine the optimal equal angle basis vector matrix W for the current classification task. (t)’ And add it to the equal angle basis vector matrix set Ω 1:t-1 In the above example, we obtain the set of equal-angle basis vector matrices Ω containing the optimal equal-angle basis vector matrices corresponding to the historical classification tasks 1, 2, …, t-1, t. 1:t , the updated equal-angle basis vector matrix set Ω 1:t Then the classification of 1~t total t classification tasks can be achieved.

[0055] Among them, the optimal equal-angle basis vector matrix W for the current classification task t is determined by solving the optimization objective function for the unit basis vector of the current classification task (t)’ , the optimization objective function for the unit basis vector of the current classification task is as follows:

[0056] ,

[0057] Where W (t) is the equal angle basis vector matrix of the current classification task t, is the unit basis vector w i The normalized value of , is the unit basis vector w j The normalized value of , are all unit basis vectors on the unit hypersphere, and γ∈[0,1) is the common angle used to separate the unit basis vectors in pairs.

[0058] By optimizing the objective function above, we can quickly find a solution that satisfies the current task t. C t basis vectors, forming the optimal equal-angle basis vector matrix W for the current classification task t (t)’ .

[0059] In solving the optimal equal angle basis vector matrix W for the current classification task t (t)’ Then, add it to the equal angle basis vector matrix set Ω 1:t-1 In the above example, we get the updated equal-angle basis vector matrix set Ω 1:t ,Right now:

[0060] Ω 1:t =[W (1)’ , W (2)’ ,…, W (t-1)’ , W (t)’ ].

[0061] This step is to determine the equal-angle basis vector matrix set Ω used in the output layer 1:t , in determining the equal angle basis vector matrix set Ω 1:t After that, we can use Ω 1:t Calculate the probability of the input image corresponding to each category, and take the category corresponding to the maximum probability as the final classification result.

[0062] For example, for the i-th input sample x of task t i (t) , denote the representation learning layer network f θ The output is z i (t) Based on the matrix Ω 1:t The isoangular basis vectors w in c ’ Calculate z i (t) is identified as the output sample y i (t) The probability is:

[0063] ,

[0064] in represents w after L2 normalization c ’ , w c ’ is the matrix Ω 1:t The cth equal-angle basis vector in , represents the number of label categories seen so far, and τ represents the temperature coefficient.

[0065] It should be noted that although there is a way to expand the number of output task categories by setting basis vectors, it is necessary to set as many basis vectors as possible at one time before each training to accommodate the output categories required by the new task, that is, it is necessary to optimize and solve a number much larger than The optimization process of C is very computationally intensive. This paper proposes for the first time to solve C by optimizing the problem in each training task. t The basis vectors can be combined with the basis vectors obtained from the historical training tasks, which can greatly reduce the computational complexity and improve the training efficiency.

[0066] Step S2: In the forward propagation of training, the optimal global output representation Z of each hidden layer b is determined by solving the problem of minimizing statistical dependence b ’ , subscript b is the hidden layer index, b=1,2,…,B, B is the number of hidden layers; the problem of minimizing statistical dependence is to minimize the weighted sum of multiple factors, the multiple factors include a first factor for calculating the correlation between the output representation of each hidden layer and the input representation of the current classification task training sample, a second factor for calculating the inverse of the correlation between the output representation of each hidden layer and the label of the current classification task training sample, and a third factor for calculating the product between the gradient mapping matrix and the hidden layer history representation determined by the previous training.

[0067] Generally, a neural network model has an input layer, a representation learning layer, and an output layer.

[0068] f θ Contains B hidden layers, and adjusts the representation learning layer f through model training θ Each hidden layer b has a corresponding activation function S b (·), nonlinear mapping through activation function , get the output representation of hidden layer b , where b=1,2,…,B,d b-1 and d b denote the output dimensions of hidden layers b-1 and b respectively, and n is the number of training samples. bIt represents the parameters that need to be updated in the hidden layer b. The output representation of the hidden layer b can be expressed as Z b = S b (Z b-1 θ b ), where Z b-1 and θ b are the input representation and parameters of the hidden layer b respectively.

[0069] In the present invention, the optimal global output representation Z of each hidden layer b is determined by solving the problem of minimizing statistical dependence in the forward propagation. b ’ . Specifically:

[0070] The first factor can be expressed as: uHSIC(Z b , X (t) );

[0071] The second factor can be expressed as: uHSIC(Z b , Y (t) );

[0072] The third factor can be expressed as: R b uHSIC A b ;

[0073] Where, X (t) Y is the input representation of the training sample of the current classification task t, (t) is the label of the training sample of the current classification task t, Z b is the output representation of the hidden layer b in the forward propagation of training, uHSIC is the function for calculating the correlation, R b uHSIC =α(A b T A b +αI) -1 is the gradient mapping matrix, α is a constant less than 1, I is the unit matrix, A b It is composed of the historical representation of the hidden layer b determined by the previous training.

[0074] In a specific embodiment, the problem of minimizing statistical dependence is specifically expressed as:

[0075] ,

[0076] Where β is the balance factor of the correlation factor, and μ is the regularization strength coefficient, both of which are weighted coefficients that can be adjusted according to actual conditions.

[0077] In specific operations, you can choose 400≤β≤600, 9000≤μ≤15000, 0.005≤α≤0.015, for example, α=0.01, μ=10000, β=500.

[0078] By solving the problem of minimizing statistical dependence, we can obtain the optimal global output representation Z of the output of each hidden layer during forward propagation. b ’ .

[0079] In a specific embodiment, the first factor and the second factor perform an unbiased estimate of the correlation based on the Hilbert-Schmidt independence criterion. For any representations E and F, the calculation formula for the correlation is as follows:

[0080] ,

[0081] In the formula, is the correlation between E and F, n is the number of training samples, tr(·) is the trace operator, and the matrix is the intermediate parameter calculated based on the Gaussian kernel matrix H, the matrix Elements in ,matrix is the intermediate parameter calculated based on the Gaussian kernel matrix G, the matrix Elements in , δ ij is the Kronecker coefficient, and the Gaussian kernel matrix H is the representation of E in the Hilbert space The nonlinear feature h is transformed in ij =h(x i ,x j ), x i 、x j They represent the output representation corresponding to the i-th training sample in representation E and the output representation corresponding to the j-th training sample, respectively. The Gaussian kernel matrix is ​​the representation F in the Hilbert space The nonlinear feature g is transformed in G ij =g(y i ,y j ), y i ,y j They represent the output representation corresponding to the i-th training sample and the output representation corresponding to the j-th training sample in representation F, respectively. The subscripts i and j are the row and column indices of the matrix, i, j = 1, 2, 3, ..., n, I n is the n×n identity matrix.

[0082] The minimization statistical dependence problem constructed in this step includes at least three factors. It should be emphasized here that the third factor is placed in the minimization statistical dependence problem and the minimization statistical dependence problem is solved. Compared with using the third factor as a constraint in the process of iteratively calculating the update value of the gradient mapping matrix using the batch learning form, the present invention places the third factor in the minimization statistical dependence problem for solution. When the update value of the gradient mapping matrix is ​​subsequently iteratively calculated using the batch learning form, only simple iterations are required. It is easier to solve under the framework of the present invention, thereby further improving the training efficiency.

[0083] Step S3: In the back propagation of training, obtain the updated equal-angle basis vector matrix set and the best global output representation Z b ’ , the gradient descent method is used to determine the parameter update amount of the network parameters, and the batch learning form is used to iteratively calculate the update value of the gradient mapping matrix. Based on the obtained parameter update amount and the update value of the gradient mapping matrix, the network parameters of each hidden layer are adjusted to obtain the training results.

[0084] In step S1, the equal-angle basis vector matrix set Ω is determined. 1:t And through step S2, the optimal global output representation Z of each hidden layer determined by the forward propagation is obtained b ’ After that, the network parameters of each hidden layer can be adjusted based on the regularized gradient descent method to obtain the training results.

[0085] Specifically, step S3 can be divided into two steps. First, the optimal network parameter update amount for each layer is calculated. , and then based on the network parameters θ obtained from the previous task training b (t-1) And the network parameter update amount determined by this training Calculate the network parameters after this training and complete this training.

[0086] Calculate the optimal update amount of network parameter gradient descent for each layer .

[0087] The goal of back propagation is to adjust the network parameters so that the classification results output by the network are close to the real labels. By adopting the network parameter gradient descent method, the update amount of the network parameter gradient descent is optimized. In a specific embodiment, the optimization objective function of the current classification task t can be constructed in the form of minimizing the negative log-likelihood function. The optimization objective function can be specifically expressed as:

[0088] ,

[0089] Among them, |D t| is the number of training samples for task t, z i (t) The representation learning layer network f obtained after the i-th input sample in task t is input into the network θ The output of , whose initial value of optimization is the best global output representation Z of the last hidden layer calculated by the forward propagation in S2 B ’ The output corresponding to the i-th input sample.

[0090] Solving the above optimization objective function, we can calculate the network parameter update amount that makes the classification result of the network output close to the real label .

[0091] Based on the network parameters θ obtained from the previous task training b (t-1) And the network parameter update amount determined by this training , substitute into the network parameter update formula to calculate the network parameter θ after this training b (t) .

[0092] Among them, the network parameter update formula is:

[0093] ,

[0094] In the formula, θ b (t-1) is the network parameter obtained from the last classification task t-1 training, is the parameter update amount of the current classification task t, λ is the learning rate, for example, it can be 0.1, M b uHSIC is the updated value of the gradient mapping matrix, which is different from the gradient mapping matrix used to minimize the statistical dependence problem in step S2 and needs to be determined through the following iterative process.

[0095] Step S31: Determine the initial value of k and extract the best global representation Z b ’ The feature vectors corresponding to the first k training samples in the initialization representation Z b ’ (k)=[z b 1 ,z b 2 ,…,z b k ], where z b k Represents Z b ’ The feature vector corresponding to the kth training sample.

[0096] Specifically, when learning task t, the best global representation Z b ’ The first k feature vectors of are considered as a batch, denoted as Z b ’ (k)=[z b 1 ,z b 2 ,…,z b k ], where Z0 ’ (k) = X (t) (k) represents treating the input of the current task t as the original feature. Given Z b ’ (k) The new vector z b k+1 , can be obtained.

[0097] Step S32: Determine the initial value M of the iteration b uHSIC (k+1), M b uHSIC (k+1)=α(Z b ’ (k)+Z b ’T (k)+ z b k+1 (z b k +1 ) T +αI) -1 , α is a constant less than 1, and I is the unit matrix.

[0098] Step S33: In each iteration, k=k+1 is set and iterative calculation is performed based on the iterative formula. After executing the preset number of iterations, the final output is M b uHSIC (k+1) is the update amount M of the gradient mapping matrix b uHSIC , and its iteration formula is:

[0099] .

[0100] Understandably, each iteration requires a new call to z b k+1 , iterates at most until the last Z is called b ’The output representation of the last sample in the training. In specific operations, the number of iterations can be set according to the actual situation. The larger the number of iterations, the more accurate the result, but the larger the amount of calculation, the slower the training speed. The number of iterations can be determined by comprehensively balancing accuracy and training speed. It should be noted that the present invention considers the third factor of the product between the calculated gradient mapping matrix and the hidden layer historical representation determined by the previous training in advance when constructing the problem of minimizing statistical dependence. The study found that when the third factor is considered in the early construction of the problem of minimizing statistical dependence, the M b uHSIC When solving , only simple iterations are required, and the more iterations there are, the more accurate the result will be. There is no need to add strong constraints, which can improve training efficiency.

[0101] like Figure 2 The figure shows an embodiment of the present invention. b uHSIC The change of rank with the increase of tasks, M b uHSIC The rank decreases slowly as the number of tasks increases, indicating that the present invention not only focuses on privacy protection in the process of continuous image classification but also maintains sufficient model plasticity to learn new tasks.

[0102] When M is determined by the above method b uHSIC Then, M b uHSIC ,θ b (t-1) , Substitute the network parameters

[0103] Update formula , we can get the network parameters obtained from the current task training, update the network parameters, and end the training.

[0104] The present invention also proposes a continuous image classification model training system that realizes zero-sample replay and simplified expansion, which includes an equal-angle basis vector matrix update unit, a forward propagation control unit, and a back-propagation control unit.

[0105] The equal angle basis vector matrix update unit is used to solve the optimization objective function for the unit basis vector of the current classification task and allocate the optimal C from the unit hypersphere. t unit basis vectors and form the optimal equal-angle basis vector matrix W for the current classification task t (t)’ , t is the classification task number, C t Equal to the number of label categories of the current classification task t, the matrix W (t)’ The equal-angle basis vector matrix set is added to the equal-angle basis vector matrix set that stores the equal-angle basis vector matrices of the historical classification tasks to obtain an updated equal-angle basis vector matrix set.

[0106] The forward propagation control unit is used to determine the optimal global output representation Z of each hidden layer b by solving the problem of minimizing statistical dependence in the forward propagation of training. b ’ , subscript b is the hidden layer index, b=1,2,…,B, B is the number of hidden layers; the problem of minimizing statistical dependence is to minimize the weighted sum of multiple factors, the multiple factors include a first factor for calculating the correlation between the output representation of each hidden layer and the input representation of the current classification task training sample, a second factor for calculating the inverse of the correlation between the output representation of each hidden layer and the label of the current classification task training sample, and a third factor for calculating the product between the gradient mapping matrix and the hidden layer history representation determined by the previous training.

[0107] Back propagation control unit, in the back propagation of training, obtains the updated equal angle basis vector matrix set and the best global output representation Z b ’ , the gradient descent method is used to determine the parameter update amount of the network parameters, and the batch learning form is used to iteratively calculate the update value of the gradient mapping matrix. Based on the obtained parameter update amount and the update value of the gradient mapping matrix, the network parameters of each hidden layer are adjusted to obtain the training results.

[0108] It can be understood that the system is used to implement the continuous image classification model training method introduced above, and each unit therein is used to implement the corresponding steps in the method. The specific details can be referred to the above introduction and will not be repeated here.

[0109] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above method when executed by a processor.

[0110] Specifically, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0111] The present invention also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the method of the above embodiment of the present invention.

[0112] In general, the present invention can achieve a balance between stability and plasticity in continuous image classification tasks, has the characteristics of zero-sample replay and simplified expansion, and greatly improves the training efficiency by improving the optimization problems of each step. It has very important potential value for the development and deployment of embedded systems and mobile devices.

[0113] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. It should be noted that "in one embodiment", "for example", "for example", etc. of the present invention are intended to illustrate the present invention, rather than to limit the present invention.

[0114] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.

Claims

1. A continuous image classification model training method that realizes zero-sample replay and simplified expansion, characterized in that: include: By solving the optimization objective function for the unit basis vector of the current classification task, the optimal C is allocated from the unit hypersphere t unit basis vectors and form the optimal equal-angle basis vector matrix W for the current classification task t (t)’ , t is the classification task number, C t Equal to the number of label categories of the current classification task t, the matrix W (t)’ Add the equal-angle basis vector matrix set that stores the equal-angle basis vector matrix of the historical classification task to obtain an updated equal-angle basis vector matrix set; In the forward propagation of training, the optimal global output representation Z of each hidden layer b is determined by solving the problem of minimizing statistical dependence b ’ , subscript b is the hidden layer index, b=1,2,…,B, B is the number of hidden layers; the problem of minimizing statistical dependence is to minimize the weighted sum of multiple factors, the multiple factors include a first factor for calculating the correlation between the output representation of each hidden layer and the input representation of the current classification task training sample, a second factor for calculating the inverse of the correlation between the output representation of each hidden layer and the label of the current classification task training sample, and a third factor for calculating the product between the gradient mapping matrix and the hidden layer history representation determined by the previous training; In the back propagation of training, obtain the updated equal-angle basis vector matrix set and the best global output representation Z b ’ , using the gradient descent method to determine the parameter update amount of the network parameters, using batch learning to iteratively calculate the update value of the gradient mapping matrix, and adjusting the network parameters of each hidden layer based on the obtained parameter update amount and the update value of the gradient mapping matrix to obtain the training results; The problem of minimizing statistical dependence is specifically: Where, X (t) Y is the input representation of the training sample of the current classification task t, (t) is the label of the training sample of the current classification task t, Z b is the output representation of the hidden layer b in the forward propagation of training, uHSIC is the function for calculating the correlation, R b uHSIC =α(A b T A b +αI) -1 is the gradient mapping matrix, α is a constant less than 1, I is the unit matrix, A b It is composed of the historical representation of the hidden layer b determined by the previous training, β is the balance factor of the correlation factor, and μ is the regularization strength coefficient; In the back propagation of training, the gradient descent method is used to determine the parameter update amount of the network parameters, and the update value of the gradient mapping matrix is ​​iteratively calculated using the batch learning form. Based on the obtained parameter update amount and the update value of the gradient mapping matrix, the network parameters of each hidden layer are adjusted by feedback, including: First obtain the updated equal angle basis vector matrix set and the best global output representation Z b ’ , construct and solve the optimization objective function that makes the classification result close to the label, and obtain the parameter update amount of the network parameter gradient descent; Then update the network parameters θ obtained by training the current classification task t based on the network parameter update formula b (t) ; Among them, the network parameter update formula is: i b (t) =θ b (t-1) -λM b uHSIC Dth b (t) , In the formula, θ b (t-1) is the network parameter obtained from the last classification task t-1 training, Δθ b (t) is the parameter update amount of the current classification task t, λ is the learning rate, M b uHSIC is the updated value of the gradient mapping matrix; Where M b uHSIC Determined through the following process: Determine the initial value of k and extract the best global representation Z b ’ The feature vectors corresponding to the first k training samples in the initialization representation Z b ’ (k) = [z b 1 ,z b 2 ,…,z b k ], where z b k Represents Z b ’ The feature vector corresponding to the kth training sample; Determine M b uHSIC The initial value of the iteration M b uHSIC (k+1), M b uHSIC (k+1)=α(Z b ’ (k)+Z b ’T (k)+z b k+1 (z b k+1 ) T +αI) -1 , α is a constant less than 1, and I is the unit matrix; In each iteration, k=k+1 is set and iterative calculation is performed based on the iterative formula. After executing the preset number of iterations, the final output is M b uHSIC (k+1) is used as the update amount M of the gradient mapping matrix b uHSIC , and its iteration formula is:

2. The continuous image classification model training method according to claim 1, characterized in that: In the forward propagation, the first factor and the second factor perform an unbiased estimate of the correlation based on the Hilbert-Schmidt independence criterion, and the calculation formula for calculating any correlation between E and F is: Where uHSIC(E,F) represents the correlation between E and F, n is the number of training samples, tr(·) is the trace operator, and the matrix is the intermediate parameter calculated based on the kernel matrix H, the matrix Elements in matrix is the intermediate parameter calculated based on the kernel matrix G, the matrix Elements in δ ij is the Kronecker coefficient, and the kernel matrix H is the representation of E in the Hilbert space The kernel matrix G is obtained by performing nonlinear feature transformation on F in the Hilbert space. The subscripts i and j are the row and column indices of the matrix, i, j = 1, 2, 3, ..., n, I n is the n×n identity matrix.

3. The continuous image classification model training method according to claim 1, characterized in that: 400≤β≤600, 9000≤μ≤15000, 0.005≤α≤0.

015.

4. The continuous image classification model training method according to claim 1, characterized in that: Constructing an optimization objective function that makes the classification result close to the label, including: constructing the optimization objective function of the current classification task t in the form of minimizing the negative log-likelihood function.

5. The continuous image classification model training method according to claim 1, characterized in that: The optimization objective function for the unit basis vector of the current classification task is specifically: Where W (t) is the equal angle basis vector matrix of the current classification task t, is the unit basis vector w i Normalized value, is the unit basis vector w j The normalized value of , γ∈[0,1) is the common angle used to separate the unit basis vectors in pairs; After obtaining the optimal equal-angle basis vector matrix W (t)’ Then, the obtained matrix W (t)’ Add and save to the equal angle basis vector matrix set to obtain the updated equal angle basis vector matrix set. The updated equal angle basis vector matrix set is Ω 1:t =[W (1)’ ,W (2)’ ,…,W (t-1)’ ,W (t)’ ].

6. A continuous image classification model training system that implements zero-sample replay and simplified expansion, characterized in that: include: The equal angle basis vector matrix update unit is used to solve the optimization objective function for the unit basis vector of the current classification task and allocate the optimal C from the unit hypersphere. t unit basis vectors and form the optimal equal-angle basis vector matrix W for the current classification task t (t)’ , t is the classification task number, C t Equal to the number of label categories of the current classification task t, the matrix W (t)’ Add the equal-angle basis vector matrix set that stores the equal-angle basis vector matrix of the historical classification task to obtain an updated equal-angle basis vector matrix set; The forward propagation control unit is used to determine the optimal global output representation Z of each hidden layer b by solving the problem of minimizing statistical dependence in the forward propagation of training. b ’ , subscript b is the hidden layer index, b=1,2,…,B, B is the number of hidden layers; the problem of minimizing statistical dependence is to minimize the weighted sum of multiple factors, the multiple factors include a first factor for calculating the correlation between the output representation of each hidden layer and the input representation of the current classification task training sample, a second factor for calculating the inverse of the correlation between the output representation of each hidden layer and the label of the current classification task training sample, and a third factor for calculating the product between the gradient mapping matrix and the hidden layer history representation determined by the previous training; Back propagation control unit, in the back propagation of training, obtains the updated equal angle basis vector matrix set and the best global output representation Z b ’ , using the gradient descent method to determine the parameter update amount of the network parameters, using batch learning to iteratively calculate the update value of the gradient mapping matrix, and adjusting the network parameters of each hidden layer based on the obtained parameter update amount and the update value of the gradient mapping matrix to obtain the training results; The problem of minimizing statistical dependence is specifically: Where, X (t) Y is the input representation of the training sample of the current classification task t, (t) is the label of the training sample of the current classification task t, Z b is the output representation of the hidden layer b in the forward propagation of training, uHSIC is the function for calculating the correlation, R b uHSIC =α(A b T A b +αI) -1 is the gradient mapping matrix, α is a constant less than 1, I is the unit matrix, A b It is composed of the historical representation of the hidden layer b determined by the previous training, β is the balance factor of the correlation factor, and μ is the regularization strength coefficient; In the back propagation of training, the gradient descent method is used to determine the parameter update amount of the network parameters, and the update value of the gradient mapping matrix is ​​iteratively calculated using the batch learning form. Based on the obtained parameter update amount and the update value of the gradient mapping matrix, the network parameters of each hidden layer are adjusted by feedback, including: First obtain the updated equal angle basis vector matrix set and the best global output representation Z b ’ , construct and solve the optimization objective function that makes the classification result close to the label, and obtain the parameter update amount of the network parameter gradient descent; Then update the network parameters θ obtained by training the current classification task t based on the network parameter update formula b (t) ; Among them, the network parameter update formula is: i b (t) =θ b (t-1) -λM b uHSIC Dth b (t) , In the formula, θ b (t-1) is the network parameter obtained from the last classification task t-1 training, Δθ b (t) is the parameter update amount of the current classification task t, λ is the learning rate, M b uHSIC is the updated value of the gradient mapping matrix; Where M b uHSIC Determined through the following process: Determine the initial value of k and extract the best global representation Z b ’ The feature vectors corresponding to the first k training samples in the initialization representation Z b ’ (k) = [z b 1 ,z b 2 ,…,z b k ], where z b k Represents Z b ’ The feature vector corresponding to the kth training sample; Determine M b uHSIC The initial value of the iteration M b uHSIC (k+1), M b uHSIC (k+1)=α(Z b ’ (k)+Z b ’T (k)+z b k+1 (z b k+1 ) T +αI) -1 , α is a constant less than 1, and I is the unit matrix; In each iteration, k=k+1 is set and iterative calculation is performed based on the iterative formula. After executing the preset number of iterations, the final output is M b uHSIC (k+1) is the update amount M of the gradient mapping matrix b uHSIC , and its iteration formula is:

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images