Task adaptation and federated learning method and system for edge-side visual analysis

Abstract

The present application provides a task adaptation and federated learning method and system for edge side visual analysis, for task adaptation, in the scene of federated learning, by preventing negative transfer to ensure the high precision of the model in visual analysis, so that the model can be trained through local samples, and can also interact with other edge devices to learn the relevant task information of other edge devices, thereby improving the precision of the model in visual analysis of the edge device; at the same time, the overhead in federated learning is reduced, the communication overhead is large when the edge device interacts, the resources of the edge device are limited, by using local task knowledge accumulation, the communication overhead and calculation overhead of the whole training process in response to task adaptation are reduced under the condition of ensuring high learning precision, and the communication size of the edge device will not increase with the increase of the task.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and particularly relates to a task self-adaptation and federated learning method and system for edge-side visual analysis. BACKGROUND

[0002] Nowadays, with hundreds of millions of mobile and Internet of Things (IoT) devices generating tens of zettabytes of data at the edge of the network, there is an opportunity to deploy artificial intelligence on edge devices, and the artificial intelligence applications (such as deep learning networks) on the edge devices can avoid transmitting raw data to protect the privacy of the devices, while at the same time, this also brings new challenges: the scenarios encountered by edge devices are constantly changing, which requires the deployed deep neural network model to capture and adapt to these changes in real time, and in real life, these devices need to handle a series of tasks over time, including different classifications, and since different clients not only have different tasks, but also carry heterogeneous training samples, when training network models on a large number of clients according to constantly changing tasks, this challenge will be further exacerbated, therefore, how to handle each task sequence on the edge with limited resources and accurately and efficiently transmit task information between devices becomes the key to solving the above difficulties.

[0003] As discussed above, the sequence of tasks on each client requires the model to constantly learn and not forget the past tasks, and each task needs to communicate with other clients during the learning process to effectively obtain task-related information and avoid situations where a single client cannot handle it, and such communication requires security and stability, because the data of the client is mostly private data, which requires strong privacy protection methods to avoid transmitting raw data samples, and at the same time, due to network limitations, the content transmitted by each client cannot be too large, therefore, in order to efficiently handle new tasks, the main methods include:

[0004] Setting regularization parameters: the regularization parameters are a set of vectors that limit the model bias, for each task, the task parameters after training are calculated to determine the contribution of the parameters to the task, and the result of the determination is the regularization parameter. By setting these regularization parameters, the model can prevent excessive bias to new tasks and cause forgetting of already learned tasks; however, the method of setting regularization parameters leads to a continuous decline in the ability to remember old tasks. In federated learning, a series of tasks carried by edge devices is endless, and the method of setting regularization parameters uses a model to process an infinite sequence of tasks without additional storage. This makes it more likely that earlier tasks will be forgotten as tasks continue, and the model can only accept tasks within a certain range. Early tasks will be discarded, making the model unable to work. In addition, the learning ability of new tasks is disturbed. In order to remember old tasks, many parameters of the model are limited, and when learning new tasks, the model convergence speed will slow down, greatly increasing the training time. At the same time, the limited parameters will reduce the training accuracy of the model;

[0005] Sample storage: the method of sample storage is simply to store part of the sample for each task. When learning new tasks, these old task samples can be used for learning together; however, its applicability in the federated scenario is limited. In the federated scenario, the computing and storage capabilities of each edge device are limited. However, this method requires continuous storage of task samples as tasks continue, so it cannot store multiple samples for each task. After the storage of samples is limited, the model's training ability for past tasks decreases, resulting in overfitting problems, which ultimately leads to catastrophic forgetting;

[0006] Dynamic model architecture: the dynamic model architecture method adds new network parameters for each task, which makes each task have its own independent parameters and reduces the interference of other tasks; however, the edge limits the model size, and the dynamic model architecture needs to continuously expand the size of the model. On various resource-constrained edge devices, as tasks continue to arrive, this will make the model unable to expand unlimitedly; in addition, the expanded model brings communication problems. Each edge device needs to transmit these model parameters to the server and other edge devices for aggregation as the model size increases continuously. There are two problems here: first, since each edge device experiences different tasks, the size of the expanded model may differ, which makes the number of model parameters transmitted to the server different and unable to be aggregated; second, as the size of the model increases, the amount of communication also increases. When the number of model parameters is particularly large, it will cause the federated framework to stall due to communication.

[0007] Based on the technical problems existing in the prior art as described above, the present application provides a task adaptation and federated learning method and system for edge side visual analysis. SUMMARY

[0008] The present application provides a task adaptation and federated learning method and system for edge side visual analysis.

[0009] The present application adopts the following technical solutions:

[0010] In one aspect, the present application provides a task adaptation and federated learning method for edge side visual analysis, comprising:

[0011] Step 1, the server transmits global model parameters to selected clients to obtain task knowledge;

[0012] Step 2, the gradient generation module calculates all past task gradients according to the local data samples of the client and the stored task knowledge, judges whether the number of current task gradients exceeds the set upper limit, if yes, calculates the similarity between the current task gradient and the past task gradient in turn, selects the gradient with the largest difference to input into the gradient aggregation module, otherwise step 2 ends;

[0013] Step 3, the gradient aggregation module calculates the included angle between the current task gradient and the past task gradient; if the included angle is not greater than 90°, the current task gradient meets the requirement of the gradient aggregation module, and the model is updated using the current task gradient; otherwise, the gradient is selected, and the current task gradient is rotated to a minimum extent, so that the current task gradient can meet the condition that the included angle with each past task gradient is less than or equal to 90 degrees, and the model is updated using the aggregated gradient;

[0014] Step 4, the client uploads its model parameters and training sample number to the server, and the server obtains the aggregated task parameters by weighted average of the training sample numbers of each client;

[0015] Step 5, the knowledge extraction module retains the model parameters with the largest 5%-15% weight values in the current model, sets the remaining parameters to 0, adjusts the model, and stores the adjusted model parameters to the knowledge storage module as the task knowledge of the current task.

[0016] Further, step 1 comprises:

[0017] Step 1.1, according to the number of clients participating in training and the set selection ratio, select clients to participate in training;

[0018] Step 1.2, the server sends parameters to each selected client.

[0019] Further, in step 2, the task gradient experienced by all task knowledge obtained from the sample includes:

[0020] Step 2.1, select a part of the sample from the current task, and the model uses the selected sample to perform an inference;

[0021] Step 2.2, obtain all past task knowledge from the knowledge storage module, and each task knowledge performs an inference according to the sample;

[0022] Step 2.3, calculate the loss according to the sample data label and the inference result of the model in step 2.1, thereby obtaining the current task gradient, and at the same time, calculate the gradient of each past task according to the inference result in step 2.2.

[0023] Further, in step 5, the model adjustment includes:

[0024] Step 5.1: randomly select part of the data sample from the current task data set, and the model performs an inference according to these data samples;

[0025] Step 5.2, calculate the loss using the data label and the inference result;

[0026] Step 5.3, calculate the gradient according to the loss, and update the parameters that are not set to 0 using the gradient;

[0027] Step 5.4, determine whether the loss reaches the loss threshold or the number of adjustments reaches the upper limit, if the loss does not reach the loss threshold or the number of adjustments does not reach the upper limit, return to step 5.2, otherwise, go to step 5.3.

[0028] Further, in step 1.1, after the client training is completed, the accuracy, loss, and memory occupation of each client model are tested.

[0029] Further, in step 2.2, the cross-entropy loss function is used as the loss function, and the loss value is calculated according to the inference probability and the true label (i.e. the greater the probability of correctly predicted class, the smaller the loss), and the past task gradient calculation formula is as follows:

[0030]

[0031] In formula (1), g i represents the gradient corresponding to the i-th task, loss represents the cross-entropy loss function, f(W, X m+1 ) represents the inference result according to the model parameter W on the data sample X m+1 , and f(W i , X m+1 ) represents the inference result of the i-th task knowledge on the sample X m+1 .

[0032] Further, in step 3, g m+1 represents the current task gradient, G = {g1,..., g m represents the past task gradient set, through g i *g m+1 <0 indicates that the angle between the two gradients is greater than 90 degrees.

[0033] Further, in step 3, the gradient rotation problem is converted into a quadratic programming optimization problem, and the optimization objective is as follows:

[0034]

[0035] s.t.Gg'≥0......(3),

[0036] In formula (2)-(3), g' is the final aggregated gradient, and it can be obtained that:

[0037]

[0038] Omitting the constant part in it:

[0039]

[0040] In order to reduce the amount of calculation, the rotation amount v of the gradient is directly calculated, and formula (5) is converted into a dual form as follows:

[0041]

[0042] s.t.v≥0......(7),

[0043] The final gradient is:

[0044] g' = G T v + g m+1 *g m+1 ......(8),

[0045] The model parameters are updated using g'.

[0046] On the other hand, the present application further provides a task adaptive and federated learning system for edge side visual analysis, comprising:

[0047] A parameter aggregation module receives multiple sets of edge side model parameters, and transmits the combined parameters to each participating edge device through an internal aggregation algorithm;

[0048] A knowledge extraction module identifies and extracts the most relevant parameter information of the task from the trained model as a task knowledge of the task; after each task switching, the knowledge extraction module extracts the task knowledge and stores it;

[0049] a knowledge storage module, receiving the task knowledge extracted by the knowledge extraction module and saving the task knowledge;

[0050] a gradient generation module, generating the gradient of all past tasks using the current training sample and the task knowledge from the knowledge storage module for the model to train, so as to adapt to the task; the gradient generation module distinguishes the part of the past task gradient with the largest difference from the gradient of the current task, and transmits the selected gradient to the gradient aggregation module;

[0051] a gradient aggregation module, used to find a suitable aggregated gradient to reduce the loss of the new task without increasing the loss of the previous task.

[0052] Further, the gradient aggregation module makes the included angle between the aggregated gradient and the past task gradient an acute angle.

[0053] Compared with the prior art, the superior effect of the present application is that:

[0054] 1. The task self-adaptation and federated learning method for edge-side visual analysis in the present application, for task self-adaptation, prevents negative transfer in the scene of federated learning to ensure high precision of the model in visual analysis, so that the model can be trained by local samples and can also interact with other edge devices to learn relevant task information of other edge devices, thereby improving the precision of the model in visual analysis of the edge device;

[0055] 2. The task self-adaptation and federated learning method for edge-side visual analysis in the present application reduces the overhead in federated learning. When interacting on the edge side, the communication overhead is large, and the resources of the edge device are limited. By using local (local) task knowledge accumulation, the communication overhead and computational overhead for task self-adaptation in the entire training process are reduced under the condition of ensuring high learning precision, and the communication size on the edge side will not increase with the increase of the task. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is the overall flowchart of the method of the embodiment of the present application;

[0057] Figure 2 is the step 1 flowchart of the method of the embodiment of the present application;

[0058] Figure 3 is the step 2 flowchart of the method of the embodiment of the present application;

[0059] Figure 4 is the step 3 flowchart of the method of the embodiment of the present application;

[0060] Figure 5 is the step 4 flowchart of the method of the embodiment of the present application;

[0061] Figure 6 is a flow chart of step 5 of the method described in the embodiments of the present application;

[0062] Figure 7 is a time and accuracy comparison chart of the method described in the embodiments of the present application and other methods on edge devices;

[0063] Figure 8 is a time comparison chart of the method described in the embodiments of the present application and other methods on different bandwidth sizes;

[0064] Figure 9 is an accuracy comparison chart of the method described in the embodiments of the present application and other methods in large-scale scenarios;

[0065] Figure 10 is an accuracy comparison chart of the method described in the embodiments of the present application and other methods in multi-task scenarios;

[0066] Figure 11 is an accuracy comparison chart of the method described in the embodiments of the present application and other methods on multiple typical classification networks;

[0067] Figure 12 is an accuracy comparison chart of the method described in the embodiments of the present application and other methods under different storage ratios. DETAILED DESCRIPTION

[0068] In order to enable a clearer understanding of the above-mentioned purposes, features and advantages of the present application, the present application will be further described in detail below in conjunction with the drawings and specific embodiments, and it should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0069] EMBODIMENT

[0070] As shown in Figures 1-6 , the task adaptation and federated learning method for edge-oriented visual analysis includes:

[0071] Step 1, the server transmits parameters to the client:

[0072] Step 1.1, according to the number of edge devices participating in training and the set selection ratio, select a part of the clients from all the clients to participate in training;

[0073] Step 1.2, the server sends parameters to each selected client, and step 1 ends;

[0074] Step 2, the gradient generation module trains the model with the gradients of all past tasks to adapt to the tasks:

[0075] Step 2.1.2, obtain all past task knowledge from the knowledge storage module, each task knowledge is inferred according to the sample;

[0076] Step 2.1.1, select a part of samples from the current task, and the model uses the selected samples to perform an inference;

[0077] Step 2.1.2, obtain all past task knowledge from the knowledge storage module, each task knowledge is inferred according to the sample;

[0078] Step 2.1.3, calculate the loss according to the sample data label and the inference result of the model in step 2.1.1, thereby obtaining the current task gradient, and calculating the gradient of each past task according to the inference result in step 2.1.2;

[0079] Step 2.2, judge whether the number of current task gradient exceeds the set upper limit, if it exceeds, enter step 2.3, otherwise, it means that the system is sufficient to cope with all gradients, then step 2 ends and enters step 3;

[0080] Step 2.3, calculate the similarity between the current task gradient and the past task gradient in turn, and select the part of the gradient with the largest difference to pass into the gradient aggregation module;

[0081] Step 3, the gradient aggregation module finds the appropriate aggregated gradient:

[0082] Step 3.1, calculate the angle between the current task gradient and the past task gradient;

[0083] Step 3.2, judge whether there is a violation, calculate the angle between each past task gradient and the current task gradient in turn, and judge whether there is an angle greater than 90 degrees, if there is no angle greater than 90 degrees, it means that the current task gradient meets the requirements of the gradient aggregation module, at this time, directly use the current task gradient to update the model, step 3 ends and enters step 4; Otherwise, enter step 3.3;

[0084] Step 3.3, select the gradient, if there is an angle greater than 90 degrees, it means that the current task gradient violates the requirements of the gradient aggregation module, at this time, the current task gradient is rotated to the minimum extent, so that it can meet the condition that the angle with each past task gradient is less than or equal to 90 degrees;

[0085] Step 3.4, update the model using the aggregated gradient;

[0086] Step 4, the server aggregates the model parameters:

[0087] Step 4.1, the client participating in the training uploads its own model parameters and the number of training samples to the server;

[0088] Step 4.2, server aggregates parameters, after receiving the parameters from the trained clients, the server obtains the aggregated task parameters by weighted average according to the number of training samples of each client;

[0089] Step 5, the knowledge extraction module extracts task knowledge and stores:

[0090] Step 5.1, according to the current model parameters, select the part with the largest weight, and set the remaining parameters to 0;

[0091] Step 5.2, model fine-tuning:

[0092] Step 5.2.1: randomly select a part of data samples from the current task data set, and the model performs inference according to these data samples;

[0093] Step 5.2.2, calculate the loss using the data label and the inference result;

[0094] Step 5.2.3, calculate the gradient according to the loss, and update the parameters that are not set to 0 using the gradient;

[0095] Step 5.2.4, judge whether the loss reaches the loss threshold or the number of fine-tuning reaches the upper limit, if not, return to step 5.2.2, otherwise enter 5.3;

[0096] Step 5.3, store the fine-tuned model parameters in the knowledge storage module.

[0097] In the above embodiment, the concept of task knowledge is defined: in actual situations, the model on the edge device needs to continuously cope with different tasks, in order to ensure that the model on the edge device will not forget the task information that has been learned, therefore, the concept of task knowledge is defined, which is used to store the key information of each task, and the task knowledge of each task is composed of model parameters after training the task, and is stored in each edge device.

[0098] The task self-adaptation and federated learning system for edge-side visual analysis comprises:

[0099] The parameter aggregation module receives multiple sets of edge-side model parameters, and transmits the combined parameters to each edge device participating in training through an internal aggregation algorithm;

[0100] The knowledge extraction module identifies and extracts the most relevant parameter information of the task from the trained model as the task knowledge of the task; after each task switching, the knowledge extraction module extracts and stores the task knowledge;

[0101] The knowledge storage module receives and saves the task knowledge extracted by the knowledge extraction module;

[0102] A gradient generation module utilizes all task knowledge from the knowledge storage module and current training samples to generate gradients of all past tasks for the model to train, so as to adapt to the task; the gradient generation module distinguishes a part of past task gradients with the largest difference from the current task gradient, and transmits the selected gradient to the gradient aggregation module.

[0103] A gradient aggregation module is used to find a suitable aggregated gradient to reduce the loss of the new task without increasing the loss of the previous task.

[0104] Specifically, in the above embodiment, a total of 4 platform architectures are selected: 8 Jetson TX2s have 256-core NVIDIA Pascal GPUs and 8 GB of memory; 8 Jetson Nanos are NVIDIA-Maxwell architecture, with 128 NVIDIA CUDA cores and 4 GB of memory; 4 Jetson Xavier NXs have 384-core NVIDIA Volta GPUs, with a total of 48 Tensor cores and 16 GB of memory; 4 Jetson AGXs have 512-core Volta GPUs and 32 GB of memory, all Jetson platforms are Ubuntu 18.04.5 LTS, support PyTorch 1.9.0 (Python 3.6.9) as the target edge device, and deploy multiple typical models in each edge device, and the data set considers multiple classic image classification data sets, including: Cifar100, FC100, CORe50, MiniImageNet and TinyImageNet. For Cifar100, FC100 and MiniImageNet, they are split into 200 private task sequences and distributed to 20 clients, for CORe50, 220 private task sequences are split and distributed to 20 clients, and for TinyImageNet, 400 task sequences are split and distributed to 20 clients.

[0105] In step 1 of the above embodiment, when training the network on the edge device, the server selects 40% of the clients to participate in training in each round, that is, 20*0.4=8 edge devices are trained, and for a task, a total of 10 rounds of training are performed, and after 10 rounds, the model of each edge device is tested, for example, including: accuracy, loss, memory occupation.

[0106] In step 2 of the above embodiment, the gradient generation module selects 20% of the samples from the existing samples to participate in gradient calculation each time. In step 2.1.1, the model inference result is the probability of predicting each class, and the loss function used in step 2.1.2 is the cross-entropy loss function, which calculates the loss value according to the inferred probability and the true label, that is, the greater the probability of correctly predicting the class, the smaller the loss. For past task gradients, the calculation formula is as follows:

[0107]

[0108] In the above formula, g i represents the gradient corresponding to the i-th task, loss represents the cross-entropy loss function described above, f(W, X m+1 ) represents the inference result of the model parameter W on the data sample X m+1 , and f(W i , X m+1 ) represents the inference result of the i-th task on the sample X m+1 using the task knowledge of the i-th task; in step 2.1.3, if the number of tasks is particularly large, set k = 5, which means only 5 most different past gradients are selected to be transmitted to the gradient aggregation module, ensuring that the training time does not increase with the number of tasks, reducing the computational load.

[0109] In step 3 of the above embodiment, the generated gradient needs to be aggregated. To facilitate the description of the specific process, g m+1 represents the current task gradient, G = {g1,..., g m} represents the set of past task gradients, and g i *g m+1 is calculated in step 3.1 to determine whether there is an obtuse angle, that is, g i *g m+1 < 0, which means the angle between the two gradients is greater than 90 degrees. If the above condition does not exist, the model parameter is updated using g m+1 , otherwise the gradient needs to be rotated according to step 3.2. According to the description of step 3.3, rotating the gradient is converted into a quadratic programming optimization problem, and the optimization objective is as follows:

[0110]

[0111] s.t.Gg'≥0,

[0112] g' is the final aggregated gradient, so:

[0113]

[0114] The constant part is omitted:

[0115]

[0116] In order to reduce the amount of calculation, the rotation amount v of the gradient is directly calculated, so the above formula is converted into a dual form as follows:

[0117]

[0118] s.t.v 0,

[0119] At this time, the final gradient is used:

[0120] g'=G T v+g m+1 *g m+1 ,

[0121] The model parameters are updated using g'.

[0122] In step 4 of the above embodiment, the 8 clients trained will transmit the trained parameters to the server, and for Cifar100, FC100, MiniImageNet, and MiniImageNet, the number of samples trained by each client each time is 250, and for TinyImageNet, it is 825, and the server aggregates the parameters by weighted average, and then sends the parameters to the next round of clients.

[0123] In step 5 of the above embodiment, when a task is trained, the client extracts the relevant task parameters as task knowledge, in step 5.1, 10% of the maximum weight parameters are selected for saving, and the remaining parameters are set to 0, in step 5.2, 20% of the samples are selected to fine-tune the selected parameters, and the maximum number of fine-tuning is set to 10, and the error value threshold is set to one percent of the previous error value, and then the trained parameters are stored in the knowledge storage.

[0124] In order to verify the superior technical effects of the method described in the above embodiment, the method performs well in the scene of federated learning and task adaptation by testing on multiple data sets. Figure 7 In the resource-constrained multiple edge device scene, the method improves the accuracy by 33.26% compared to the general federated learning, and improves the accuracy by 77.35% compared to the general task adaptation method, and in practice, the communication network bandwidth is limited from 50KB to 10MB, and the method described in the application has good performance, and the communication time of the method is only 10% of the training time when the bandwidth is 1MB, which is 34.28% lower than the latest algorithm Figure 8 ) communication time.

[0125] By testing on extreme conditions, mainly including: large-scale clients and a large number of tasks, the method still maintains the highest accuracy and less training time, first for large-scale clients, 50 and 100 clients are selected for testing. Figure 9 In the case of large-scale clients, the method still maintains the highest accuracy, compared with some of the latest methods, the accuracy is improved by 43.1%, and for the multi-task case, as shown in Figure 10 In the case of large-scale clients, the method still maintains the highest accuracy, compared with some of the latest methods, the accuracy is improved by 43.1%, and for the multi-task case, as shown in

[0126] By testing on different typical image classification networks, the results reflect the applicability of the method, in the test, 8 kinds of latest neural networks are selected, and MiniImageNet is tested, in Figure 11 In the case of large-scale clients, the method still maintains the highest accuracy, compared with some of the latest methods, the accuracy is improved by 43.1%, and for the multi-task case, as shown in

[0127] Finally, by setting different parameter ratios for testing, in the test, the task adaptive algorithm for saving samples is selected for comparison, as shown in Figure 12 In the case of large-scale clients, the method still maintains the highest accuracy, compared with some of the latest methods, the accuracy is improved by 43.1%, and for the multi-task case, as shown in

[0128] The application is not limited by the above embodiments, the above embodiments and the description are only to illustrate the principles of the application, various changes and improvements of the application can be made without departing from the spirit and scope of the application, and these changes and improvements all fall within the scope of the claimed application. The scope of protection of the application is defined by the appended claims.

Claims

1. A method for task adaptation and federated learning for edge-side visual analysis, characterized in that, Comprise: Step 1, the server transmits global model parameters to selected clients to obtain task knowledge; Step 2, the gradient generation module calculates all past task gradients according to the local data samples of the client and the stored task knowledge, the data samples are used for image classification, and it is judged whether the number of current task gradients exceeds the set upper limit, if it exceeds, the similarity between the current task gradient and the past task gradient is calculated in turn, the gradient with the largest difference is selected and transmitted to the gradient aggregation module, otherwise step 2 ends; Step 3, the gradient aggregation module calculates the included angle between the current task gradient and the past task gradient; if the included angle is not greater than 90°, the current task gradient meets the requirement of the gradient aggregation module, and the model is updated using the current task gradient; otherwise, the gradient is selected, and the current task gradient is rotated to the minimum extent, so that the current task gradient can meet the condition that the included angle with each past task gradient is less than or equal to 90 degrees, and the model is updated using the aggregated gradient; Step 4, the client uploads its own model parameters and training sample number to the server, and the server obtains the aggregated task parameters by weighted average of the training sample numbers of each client; Step 5, the knowledge extraction module retains the model parameters with the largest 5%-15% weight value in the current model, and sets the remaining parameters to 0, adjusts the model, and stores the adjusted model parameters in the knowledge storage module as the task knowledge of the current task.

2. The edge-side visual analysis oriented task adaptation and federated learning method according to claim 1, characterized in that, Step 1 includes: Step 1.1, select clients to participate in training according to the number of participating clients and the set selection ratio; Step 1.2, the server sends parameters to each selected client.

3. The edge-side visual analysis oriented task adaptation and federated learning method according to claim 1, characterized in that, In step 2, all task gradients experienced by the task knowledge obtained according to the samples include: Step 2.1, select a part of samples from the current task, and the model performs inference using the selected samples; Step 2.2, obtain all past task knowledge from the knowledge storage module, and perform inference according to the samples for each task knowledge; Step 2.3, calculate the loss according to the sample data label and the inference result of the model in step 2.1 to obtain the current task gradient, and calculate the gradient of each past task according to the inference result in step 2.

2.

4. The edge-side visual analysis oriented task adaptation and federated learning method according to claim 1, characterized in that, In step 5, the model adjustment includes: Step 5.1: randomly select part of the data samples from the current task data set, and the model performs inference according to these data samples; Step 5.2, calculate the loss using the data label and the inference result; Step 5.3, calculate the gradient according to the loss, and update the parameters that are not set to 0 using the gradient; Step 5.4, judge whether the loss reaches the loss threshold or the number of adjustments reaches the upper limit, if not, return to step 5.2, otherwise go to step 5.

3.

5. The edge-side visual analysis oriented task adaptation and federated learning method according to claim 2, characterized in that, In step 1.1, after the client training is completed, the accuracy, loss and memory occupation of each client model are tested.

6. The edge-side visual analysis oriented task adaptation and federated learning method according to claim 3, characterized in that, In step 2.2, the cross-entropy loss function is used as the loss function, and the loss value is calculated according to the inference probability and the true label, and the past task gradient calculation formula is as follows: ……（1）， denotes a gradient corresponding to the i-th task, denotes a cross-entropy loss function, denotes a cross-entropy loss function, denotes a cross-entropy loss function, denotes a cross-entropy loss function, denotes a cross-entropy loss function, denotes a cross-entropy loss function, denotes a cross-entropy loss function, denotes a cross-entropy loss function.

7. The edge-side visual analysis oriented task adaptation and federated learning method according to claim 6, characterized in that, In step 3, use denotes the current task gradient, denotes the set of past task gradients, by determines that the angle between the two gradients is greater than 90 degrees.

8. The edge-side visual analysis oriented task adaptation and federated learning method according to claim 7, characterized in that, In step 3, the gradient rotation problem is converted into a quadratic programming optimization problem, and the optimization objective is as follows: ， ， In formulas (2)-(3), The final gradient of polymerization is obtained as: ， Omit the constant part in it: ， To reduce the amount of calculation, the amount of rotation of the gradient is directly calculated Equation (5) is converted to a dual form as follows: ， ， The final gradient is: ， Using updating the model parameters.

Images