New category learning and lightweight update migration method for mobile terminals based on parameter freezing

By adopting parameter freezing technology on mobile devices, the computing and communication resource limitations of new category learning and model updates on mobile devices are solved, and efficient new category learning and lightweight model updates are achieved, which significantly improves accuracy and efficiency. Reduced resource consumption.

CN118172599BActive Publication Date: 2024-11-12SHANGHAI JIAOTONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410289526.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-14
Publication Date
2024-11-12
Estimated Expiration
2044-03-14

AI Technical Summary

Technical Problem

When learning new categories and model updates are performed on mobile devices, there are computational and communication resource limitations, resulting in large amounts of old categories data forgotten and model updates, making it difficult to efficiently handle new categories and transmission model updates.

Method used

Using a parameter freezing method, freeze the model parameters of the feature extractor and old category classification header on the server side, and use the new category data to perform lightweight model updates on mobile devices, reducing memory overhead by selectively freezing and thawing parameters. and model updates to realize lightweight model transmission across devices.

Benefits of technology

It significantly reduces the forgetting of old category data, reduces training time and memory overhead, and the number of model updates is very light, improving the accuracy and efficiency of learning in new categories, and reducing the cost of computing and communication.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118172599B_ABST
    Figure CN118172599B_ABST
Patent Text Reader

Abstract

A method for learning new categories and lightweight update migration on a mobile terminal based on parameter freezing, first completely freezes the model parameters of the feature extractor in the image classification model on the server side, then selectively freezes the model parameters of the old category classification head in the image classification model and sends them to the mobile device to be used for incremental training; after the new and old category classification heads are selectively frozen again based on pre-training on the mobile terminal, the lightweight model update is performed using the new category data collected locally, and finally the cross-device transmission of the updated part of the classification head is completed. The present invention can reduce the forgetting of the old category data while maintaining the ability to learn new categories, and reduce the memory overhead of training, and also enables training to be performed in a short time and the model update migration amount is very lightweight, which overall solves the problem of learning a small number of new categories on mobile devices with limited computing and communication resources and transmitting model updates to other mobile devices performing the same task.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a technology in the field of neural network applications, specifically a new category learning and lightweight update migration method for a mobile terminal based on parameter freezing. Background Art

[0002] In the process of visual recognition tasks in open environments undertaken by mobile devices, new categories may continue to appear as the environment continues to change and usually represent unexpected situations. In order to mitigate potential risks, it is necessary to be able to incrementally learn new categories on mobile devices and transmit the new category recognition capabilities to other devices performing the same task. However, whether it is the coordination of the conflict between learning new categories and forgetting old categories in the training phase, or the transmission of model updates in the communication phase, there are severe challenges under the limitations of the computing and communication resources of mobile devices. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a new category learning and lightweight update migration method for a mobile terminal based on parameter freezing, which selectively freezes most of the model parameters before training, thereby reducing the forgetting of old category data while maintaining the ability to learn new categories, and reducing the memory overhead of training. It also enables training to be carried out in a short time and the model update migration amount is very lightweight, which overall solves the problem of learning a small number of new categories that appear on mobile devices with limited computing and communication resources and transmitting model updates to other mobile devices performing the same tasks.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a new category learning and lightweight update migration method for a mobile terminal based on parameter freezing. First, the model parameters of a feature extractor in an image classification model are completely frozen on the server side, and then the model parameters of an old category classification head in the image classification model are selectively frozen and sent to a mobile device to perform incremental training. After the new and old category classification heads are selectively frozen again based on pre-training on the mobile terminal, lightweight model updates are performed using new category data collected locally, and finally the updated part of the classification head is transmitted across devices.

[0006] The complete freezing means: for the feature extractor, the old category classification head and the new category classification header For the image classification model Θ, all shallow parts except the classification head will remain unchanged during the learning process of new categories.

[0007] The selective freezing of the model parameters of the old category classification head refers to: using a random sampling subset D of the old category training data stored in the server f, calculate the diagonal value f of the Fisher information matrix for the old classification head, and for each parameter j in the old classification head, calculate the corresponding value of the diagonal element of the Fisher information matrix according to its value Freeze all corresponding values ​​f by proportion α j The larger element, where: 0≤α<1.

[0008] The sending means: the server creates a bit mask corresponding to the old category classification header parameters of the model, uses the bit mask to record the frozen parameters and sends them to the mobile device to perform incremental training, wherein the corresponding bit mask of the frozen parameters is set to 0, otherwise it is set to 1.

[0009] The selective freezing based on pre-training refers to: the mobile device uses the new category data to quickly train the image classification model including the selectively frozen old category classification head and the unfrozen new category classification head, and decides whether to freeze the parameter according to the absolute value of the change of each parameter j during the fast training process. Specifically, the elements with larger corresponding values ​​in the old classification head will be frozen according to the proportion β old The corresponding elements with larger values ​​in the new classification header will be retained according to the proportion β new retain, all other elements will be frozen, where: 0≤β old ≤1-α,0<β new ≤1.

[0010] The lightweight model update means that the mobile device uses the locally collected new category data to train the local image classification model, so that the unfrozen network parameters in the old category classification head and the new category classification head in the model are updated.

[0011] The cross-device transmission means that after a mobile device performs local incremental training based on new category data, the generated lightweight model update is transmitted to other devices; after receiving the model update, other mobile devices superimpose the model update on the new and old category classification heads of their local classification models, specifically: in: They are the local new and old category classification heads, Respectively, for the lightweight model updates received.

[0012] Technical Effects

[0013] The present invention freezes model parameters by decoupling the server and the mobile device. The feature extractor is frozen on the server, and the Fisher information is calculated and the parameters with higher corresponding Fisher information values ​​in the old model classification head are frozen by using the old category data. The new category data is used on the mobile device to freeze the parameters with smaller changes in the unfrozen parts of the new category classification head and the old category classification head by pre-training the new category data. Compared with the prior art, the present invention significantly reduces the forgetting of the old category data while maintaining the ability to learn new categories, reduces the memory overhead of training, and enables training to be performed in a shorter time. The model update migration amount is very light. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the process of the present invention;

[0015] Figure 2 A schematic diagram of selectively freezing model parameters of old category classification heads on a server according to the present invention;

[0016] Figure 3 A schematic diagram of the present invention using new category data for pre-training on a mobile device and further using the pre-training results to selectively freeze new and old category classification heads;

[0017] Figure 4 It is a schematic diagram comparing the classification accuracy of the present invention and the classic incremental learning methods EWC, LwF, fine-tuning all parameters (FT) and fine-tuning feature extractor (FFE) on new and old categories;

[0018] Figure 5 A schematic diagram showing the comparison of tensor memory usage between the present invention and EWC, LwF, FT and FFE;

[0019] Figure 6 Schematic diagram of the comparison of the running time of the present invention and EWC, LwF, FT and FFE on two embedded devices, NVIDIA Jetson TX2 and NVIDIA JetsonNANO;

[0020] Figure 7 It is a schematic diagram comparing the model updating migration volume between the present invention and EWC, LwF, FT and FFE. DETAILED DESCRIPTION

[0021] like Figure 1As shown, this embodiment relates to a new category learning and lightweight update migration method for a mobile terminal based on parameter freezing. This embodiment is performed on the classification task of the CIFAR100 data set. CIFAR100 contains 60,000 three-channel color images of 100 classes, and each class contains 600 images (500 images for training and 100 images for testing). This embodiment simulates the actual data situation with more old categories and fewer new categories by using the first 50 categories as old category data and the 51st-52nd categories as new category data, and simulates the distribution location of new and old category samples in the real situation by placing all the old category data on the server and all the new categories on the mobile device. For the image classification model, this embodiment adopts the Resnet18 classic architecture, which includes 11.2M parameters, of which the old classification head has 25,650 parameters and the new classification head has 1026 parameters. The method specifically includes the following steps:

[0022] Step 1: Initialize the mobile device and edge server: assign the 51st and 52nd categories of the CIFAR100 dataset as new category data to the mobile device, and assign the 1st to 50th categories as old category data to the edge server. Both the edge server and the mobile device contain image classification models trained on the 50 old categories.

[0023] The image classification model includes: a feature extractor, an old category classification head, and a new category classification head created by an edge server and a mobile device respectively, wherein: the new category classification head is initialized with a random seed.

[0024] Step 2: Freeze parameters on the edge server: The edge server uses old category data to select model parameters to be frozen, including:

[0025] 2.1) The edge server keeps all shallow parts of the image classification model except the classification head unchanged during the learning process of new categories, that is, completely frozen, thereby significantly reducing the memory usage of mobile devices during training and the size of model updates.

[0026] 2.2) The edge server randomly selects 5 images from each of all old categories to form a sample set D f , through the old category classification head The diagonal value f of the empirical Fisher information matrix is ​​calculated, and the parameters corresponding to the highest part are selected to freeze.

[0027] The highest portion accounts for a ratio of α to all diagonal values, and α=0 in this embodiment.

[0028] 2.3) The edge server sends the partial freezing result, i.e., the bit mask, to the mobile device.

[0029] Step 3: Freeze local parameters of mobile devices: When the mobile device receives partial freezing results and encounters new category data, it uses the new category data to selectively freeze the unfrozen parameters based on pre-training again until the new category test accuracy reaches 0.8, and obtains the absolute value of each model parameter update at this time.

[0030] The re-selective freezing based on pre-training means that the elements with larger corresponding values ​​in the old classification head will be frozen in proportion β old The corresponding elements with larger values ​​in the new classification header will be retained according to the proportion β new Retain, in this embodiment β old =0.1,β new =0.3.

[0031] Step 4: Mobile devices learn new categories and complete cross-device model update transmission: Use samples of the new category to train the model on resource-constrained mobile devices, and only update the unfrozen parameters during the training process. After the new category is learned, the mobile device transmits the updated part of the classification head to other mobile devices. Those devices will add the update to the local image classification model and can recognize the new category of data.

[0032] After specific practical experiments, the above method was run on a server with INTEL i9-9900K CPU and NVIDIA RTX-2080Ti GPU, under Python 3.8.12, and on two embedded devices, NVIDIAJetson TX2 and NVIDIAJetson NANO.

[0033] like Figure 4 As shown in FIG. 1 , the average accuracy of the model trained by the present invention on the test set is 17.49% and 3.84% higher than that of the classic algorithms EWC and LwF (74.72% vs. 57.23% and 70.88%, respectively). Figure 5 As shown in FIG. 1 , the maximum tensor memory usage of the training process of the present invention is 141 MB, which is 67.95% and 59.84% less than that of EWC and LwF, respectively. Figure 6 As shown, the training time for new category learning of the present invention is 11.33 seconds and 42.33 seconds on NVIDIA Jetson TX2 and NVIDIA Jetson NANO, respectively.

[0034] Store the model updates in FP32 format, such as Figure 7 As shown, the model update migration volume generated by the present invention during the new category learning process is 11.3KB, accounting for only 0.026% of the total model volume; it is reduced by 99.974% (11.3KB vs. 43,800KB) compared with EWC and LwF.

[0035] Compared with the prior art, the present invention achieves the purpose of preventing catastrophic forgetting of old categories, learning new categories and reducing computational and communication costs through a unified parameter freezing framework. In the evaluation of the classification task based on the CIFAR100 dataset, the present invention achieved an average test accuracy that was 3.85% higher than the classic incremental learning methods EWC and LwF, and reduced tensor memory usage by 59.94%, training time by 54.68%, and model update migration by 99.974%. In addition, the present invention does not need to store old category data locally on the mobile device through the server-mobile device decoupling design, which greatly reduces the related privacy, security and legal risks.

[0036] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A new category learning and lightweight update migration method for mobile terminals based on parameter freezing, characterized in that: First, the model parameters of the feature extractor in the image classification model are completely frozen on the server side, and then the model parameters of the old category classification head in the image classification model are selectively frozen and sent to the mobile device to perform incremental training; after the new and old category classification heads are selectively frozen again based on pre-training on the mobile side, the lightweight model is updated using the new category data collected locally, and finally the updated part of the classification head is transmitted across devices; The complete freezing means: for the feature extractor, the old category classification head and the new category classification header The image classification model Θ, all shallow parts except the classification head, will remain unchanged during the learning process of new categories; The selective freezing of the model parameters of the old category classification head refers to: using a random sampling subset D of the old category training data stored in the server f , calculate the diagonal value f of the Fisher information matrix for the old category classification head, and for each parameter j according to its corresponding value in the diagonal element of the Fisher information matrix Freeze all corresponding values ​​f by proportion α j Larger elements, where: 0≤α<1; The sending means: the server creates a bit mask corresponding to the old category classification header parameter of the model, uses the bit mask to record the frozen parameters and sends them to the mobile device to perform incremental training, wherein the corresponding bit mask of the frozen parameters is set to 0, otherwise it is set to 1; The pre-training-based selective freezing again means that the mobile device uses new category data to quickly train the image classification model including the selectively frozen old category classification head and the unfrozen new category classification head, and decides whether to freeze the parameter according to the absolute value of the change of each parameter j during the rapid training process.

2. According to the method for new category learning and lightweight update migration based on parameter freezing on mobile terminals according to claim 1, it is characterized in that: The selective freezing based on pre-training means that the elements with larger corresponding values ​​in the old classification head will be frozen in proportion β old The corresponding elements with larger values ​​in the new classification header will be retained according to the proportion β new retain, all other elements will be frozen, where: 0≤β old ≤1-α,0<β new ≤1.

3. According to the parameter freezing-based mobile terminal new category learning and lightweight update migration method according to claim 1, it is characterized in that: The lightweight model update means that the mobile device uses the locally collected new category data to train the local image classification model, so that the unfrozen network parameters in the old category classification head and the new category classification head in the model are updated.

4. The method for new category learning and lightweight update migration based on parameter freezing on a mobile terminal according to claim 1 is characterized in that: The cross-device transmission means that after a mobile device performs local incremental training based on new category data, the generated lightweight model update is transmitted to other devices; after receiving the model update, other mobile devices superimpose the model update on the new and old category classification heads of their local classification models, specifically: in: They are the local new and old category classification heads, Respectively, for the lightweight model updates received.