Basic model neuron granularity adaptation method for edge side data drift

Abstract

The invention relates to an edge side data drift-oriented basic model neuron granularity adaptation method, which comprises the following steps of: 1, when edge side input data is detected to drift, quickly calculating the importance of neurons in a basic model by an agent neuron extraction module by taking the basic model and new environment data as input, and calculating the granularity of the neurons in the basic model; extracting the most important neurons in the basic model; step 2, a neuron index construction module takes the basic model and the proxy neurons as the output in the step 1 as the input, constructs a neuron index between each proxy neuron and the basic model, and outputs the neuron index; step 3, the proxy neuron updating module uses the new environment data to update the proxy neuron; and the like. The method has the advantages that the most important neurons in the basic model are extracted for updating under limited resources on the edge side, the basic model does not need to be compressed, and adaptation can be carried out under the condition that generalization of the basic model is not lost, so that the adaptation speed is increased, and the adaptation precision is improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a basic model neuron granularity adaptation method for edge-side data drift. Background Art

[0002] With the development of deep learning technology, the introduction of foundation models marks a major change in the field of artificial intelligence. Foundation models usually have hundreds of millions or even hundreds of billions of parameters, can efficiently capture contextual information, and have strong representation and reasoning capabilities. Foundation models include large language models (such as ChatGPT), large visual models, and large multimodal models. They have emerged in tasks such as human-computer dialogue, object detection, speech recognition, and picture-text question and answer. Complex capabilities and characteristics that traditional non-foundation models do not have have emerged, showing human-like thinking and intelligence, so they are called the cornerstone of the next generation of artificial intelligence. An important difference between foundation models and traditional non-foundation models is that foundation models are usually pre-trained on cloud servers using massive data, but after pre-training, they only store highly generalized knowledge and perform poorly on data in specific tasks and fields. At this time, the foundation model needs to be "adapted" / "fine-tuned" on the data of the target task and field so that the foundation model can better adapt to the data distribution of the target task and field before it can be used normally.

[0003] The adaptation / fine-tuning here mainly refers to the process of updating all or part of the parameters in the pre-trained basic model on a specific data set to improve its accuracy on the specific data set. For example, the Vision Transformer, a basic visual model pre-trained on massive image data, lacks accurate reasoning capabilities for specific fields (such as traffic scene pictures). At this time, it is necessary to use the traffic scene picture data set to fine-tune all or part of the key parameters of the basic model. The common term "training" involves a wider range of fields. In addition to adaptation / fine-tuning, it can also refer to pre-training process, etc. The present invention mainly focuses on the field of adaptation / fine-tuning of the basic model.

[0004] Most of the calls to the basic model come from users or devices on the edge side (such as mobile phones and tablets). In order to respond to requests quickly, avoid network transmission delays, and protect data privacy, the basic model can be deployed on the edge side after the initial adaptation / fine-tuning. However, the environment on the edge side may continue to change dynamically, causing the distribution of model input data to continue to evolve, which is called the "data drift" phenomenon on the edge side, which will greatly reduce the accuracy of the basic model. For example, the edge-side unmanned vehicle may continue to move in different geographical locations (open roads, congested roads, and tunnels, etc.) and weather (cloudy, cloudy, and rainy, etc.). At this time, the recognition accuracy of the basic model in the unmanned vehicle may be greatly reduced, affecting the perception ability and driving safety of the unmanned vehicle. At this time, it is necessary to continuously use the data in the new environment to further adapt / fine-tune the basic model to maintain the high accuracy of the basic model.

[0005] Basic model adaptation / fine-tuning for edge data drift includes the following three mainstream methods:

[0006] (1) Efficient adaptation / fine-tuning method of basic model parameters. This type of method adds a small number of trainable model parameters to each network layer in the basic model, freezes other parameters during fine-tuning, and only updates the newly added parameters, that is, training at the "parameter" granularity, so as to speed up the fine-tuning speed and reduce resource usage. For example, add a low-rank matrix to each layer, or add a magnitude vector and direction matrix, and then only train these newly added matrices and vectors. This type of method has a key problem when fine-tuning the basic model under edge data drift: because the number of basic model parameters is in the hundreds of millions and the number is huge, it is impossible to quickly calculate the importance of each parameter based on the newly arrived data under limited resources, and select the most important parameters to update. This may cause parameters with very low accuracy to occupy the computing resources used for adaptation, reducing the adaptation efficiency of the basic model under limited resources on the edge, and low adaptation accuracy.

[0007] (2) Static small model generation and adaptation / fine-tuning methods. This type of method first compresses the basic model into a small model with a fixed structure and size before deployment, and then uses the new environment data to train all the parameters of the small model to speed up training and reduce resource usage. For example, in the new environment, the basic model knowledge is distilled to the small model, and the drift of label distribution can be considered to optimize the adaptation algorithm, and the small model that has been adapted in the new environment in the past can be stored and reused in the new environment in the future, reducing unnecessary adaptation costs.

[0008] (3) Dynamic small model generation and adaptation / fine-tuning methods. This type of method uses the dynamic scaling method of the model architecture in the inference phase to generate a small model from the base model before fine-tuning begins, and then uses the new environment data to train all the parameters of the small model. The dynamic scaling method of the model architecture in the inference phase refers to dynamically adjusting the model size according to the available resources or input data during inference, thereby adjusting the trade-off between model accuracy and resource usage under limited resources. For example, by allowing model inference to exit from the middle layer of the model early, generating a series of candidate models of different sizes in advance, searching for the best model structure in advance according to the hardware architecture and operating constraints, and dynamically increasing or decreasing the filters involved in the calculation in the model during runtime. The method described in the latest paper of this type of method, "ElasticDNN: On-Device Neural Network Remodeling for Adapting Evolving Vision Domains at Edge", first shrinks the original base model to a small model when the environment changes, then trains the small model using new environment data, and finally feeds the knowledge learned by the small model back to the large model.

[0009] The above static or dynamic small model generation and adaptation / fine-tuning methods all scale and train the base model at the "model granularity", that is, compress the entire base model into a small model and then train the small model. However, the compression of the base model will damage the strong generalization performance of the base model, making the compressed small model slow to learn and have limited learning capacity during training, ultimately resulting in low accuracy in the adaptation / fine-tuning of the base model.

[0010] In summary, the existing basic model adaptation / fine-tuning methods for edge-side data drift belong to the "parameter" granularity or "model" granularity. The parameter granularity is too fine, and the overhead of finding the most important parameters is too high, so it is impossible to quickly perceive the newly arrived data and make targeted adaptations to the most important parameters; the model granularity is too coarse, and the generalization performance of the basic model cannot be effectively retained, and the learning capacity is limited.

[0011] For example, the patent application document with Chinese invention patent application number CN202411141783.5 discloses a large model fine-tuning method, device, equipment, medium and program product, which constructs a re-parameterization factor for each self-attention layer in the large model, and constructs a static multi-head self-attention weight based on the re-parameterization factor. Based on the static multi-head self-attention weight and the initial self-attention map, the first fine-tuning module is determined, and the second fine-tuning module is determined based on the target feedforward neural network layer and the low-rank adapter parameters. Based on the first fine-tuning module and the second fine-tuning module, some parameters of the large model are adjusted on the training data. This method belongs to the parameter granularity basic model parameter efficient adaptation / fine-tuning method, which cannot quickly perceive the dynamic input data on the edge side to make targeted fine-tuning of the most important parameters, and the efficiency of learning new data is limited, and the adaptation / fine-tuning accuracy is low under the limited resources on the edge side.

[0012] For another example, the patent application document with Chinese invention patent application number CN202410480665.0 discloses a training method and device for a large language model in the vertical field of e-commerce. According to the data ratio and pre-training data set, the LoRA method is used to continue pre-training the existing open source large language model to obtain the basic model, and the fine-tuning data set is used to fine-tune the basic model with multi-task instructions to obtain the fine-tuned basic model, i.e., the large language model in the vertical field of e-commerce. This method belongs to the parameter granularity basic model parameter efficient adaptation / fine-tuning method, which cannot quickly perceive the dynamic input data on the edge side to make targeted fine-tuning of the most important parameters, the efficiency of learning new data is limited, and the adaptation / fine-tuning accuracy is low under the limited resources on the edge side.

[0013] For another example, the patent application document with Chinese invention patent application number CN202411120027.4 discloses a multimodal large model training optimization method and device in the power vertical field. After the cloud server compresses the basic model, the compressed small model is updated and tuned using a low-rank fine-tuning method. This method belongs to the static small model generation and adaptation / fine-tuning method at the model granularity. The compression of the basic model on the cloud server impairs its generalization performance for dynamic input data on the edge side, and the learning capacity of the small model is limited, so the adaptation / fine-tuning accuracy is low under the limited resources on the edge side. Summary of the invention

[0014] In view of the defects of the prior art in this technical field, the present invention proposes a basic model neuron granularity adaptation method for edge-side data drift, and proposes a "neuron granularity" training method. The neuron granularity is between the parameter granularity and the model granularity, achieving a better balance between adaptation / fine-tuning accuracy and overhead. The neuron is the most basic computing unit for feature extraction in the basic model. A neuron contains multiple model parameters, and multiple neurons constitute a basic model. The model parameters in a neuron have similar important characteristics.

[0015] The basic model neuron granularity adaptation method for edge-side data drift described in the present invention includes the following two key data structures:

[0016] Proxy neurons: A small number of neurons in the base model that are most important for accuracy in a new environment. Training these neurons has the greatest benefit for the accuracy of the base model and are identified and extracted for separate training.

[0017] Neuron index: It is built between the base model and the proxy neurons and is used to feed back the update amount of the proxy neurons to the base model to update the neurons related to the base model, so as to achieve fast adaptation / fine-tuning of the base model without compromising the generalization of the base model.

[0018] The basic model neuron granularity adaptation method for edge-side data drift described in the present invention proposes a neuron granularity training method, which includes the following functional modules:

[0019] Proxy neuron extraction module: According to the data distribution in the new environment, first select the most representative data sample in the new environment, and then quickly calculate the importance of each neuron in the basic model in the new environment based on the sample. The higher the importance, the greater its contribution to the accuracy, and the more worthy of updating in fine-tuning. Then, extract a small number of the most important neurons in the basic model as proxy neurons;

[0020] Neuron index building module: builds the neuron index between the base model and each proxy neuron, where each neuron index is a one-to-many index that connects a proxy neuron to all neurons in the same layer of the base model, and defines how the model parameters in the neurons of the base model are mapped to the model parameters in the corresponding proxy neurons through linear combination operations;

[0021] Agent neuron update module: Based on the contrastive learning loss function, each agent neuron is individually updated with new environment data;

[0022] Basic model adaptation module: Through neuron indexing, the update amount of the proxy neuron is quickly fed back to the basic model to update the relevant neurons of the basic model, thereby achieving rapid fine-tuning of the basic model without compromising the generalization of the basic model.

[0023] The basic model neuron granularity adaptation method for edge-side data drift of the present invention comprises the following steps:

[0024] Step 1: When the edge input data drifts, the proxy neuron extraction module takes the basic model and new environment data as input, quickly calculates the importance of neurons in the basic model, and extracts the most important neurons in the basic model as the output proxy neurons;

[0025] Step 2: The neuron index building module takes the basic model and the proxy neuron output in step 1 as input, builds the neuron index between each proxy neuron and the basic model and outputs it;

[0026] Step 3, the agent neuron update module updates the agent neuron using the new environment data;

[0027] Step 4: The basic model adaptation module uses the knowledge learned by the proxy neurons in step 3 in the new environment to update the corresponding neurons in the basic model based on the neuron index, thereby achieving the effect of basic model adaptation.

[0028] Furthermore, the proxy neuron extraction module in step 1 extracts the proxy neuron, specifically including:

[0029] Step 1.1, given v data samples in the new environment to Calculate the i-th sample Information entropy

[0030]

[0031] In the above formula, represent The probability of being predicted as category c. The information entropy of a data sample represents the amount of information it contains. The sample with higher information entropy contains more information in the new environment and is more representative of the new environment. After calculating the information entropy of all v data samples, this step selects the sample with the highest information entropy as the most representative sample.

[0032] Step 1.2: The most representative sample selected in step 1.1 Input the basic model for forward propagation, during which the jth neuron in the i-th layer of the basic model is calculated Importance As shown below:

[0033]

[0034] In the above formula, Fi represents the input of the i-th layer of the basic model in the forward propagation, and FBS (Feature Boosting and Suppression) represents the feature boosting and compression module. The importance score of each neuron in each layer is calculated according to the input of each layer. In this step, the FBS module is based on F i As input, after being processed by the average pooling layer AvgPool, the linear layer Linear and the activation function layer ReLU, the final output is a vector The jth element in That is, the jth neuron in the i-th layer of the basic model Importance score Apply this step to all neurons in the base model and calculate the importance scores of all neurons in the base model;

[0035] Step 1.3: Extract neurons with importance scores in the top ρ% from each layer of the base model as proxy neurons of the base model. is extracted as the proxy neuron, then the proxy neuron is represented as

[0036] Furthermore, the neuron index construction module in step 2 constructs a neuron index between each proxy neuron and the basic model, specifically including:

[0037] Step 2.1, for a proxy neuron and all neurons in layer i of the base model (w is the number of neurons in the i-th layer of the basic model), a neuron index is established between them, and the neuron index stores a set of weight coefficients {Γ i,j,1 ,Γ i,j,2 ,…,Γ i,j,w}, where Γ i,j,k represent and The correlation between these two factors defines the basic model neuron. To the agent neuron The mapping relationship is as shown below:

[0038]

[0039] Based on the above mapping relationship, the basic model adaptation module can quickly feed back the proxy neuron update amount to the basic model neuron, apply this step to all proxy neurons, and initialize the neuron index of all proxy neurons;

[0040] Step 2.2, use the loss function and gradient descent algorithm to solve the weight coefficient in the neuron index:

[0041]

[0042] Where L is the number of network layers in the base model, and step 2.2 is applied to all neurons.

[0043] Furthermore, in step 3, the proxy neuron update module uses the new environment data to train the proxy neuron for several iterations, each of which specifically includes:

[0044] Step 3.1: Substitute the v data samples from step 1.1 to Perform data enhancement and input the basic model for forward propagation to obtain the input of the i-th layer of the basic model in this process, which is recorded as

[0045] Step 3.2, for the agent neuron The F obtained in step 1.2 i and step 3.1 Input the neuron respectively and get the neuron For F i and Output O i,j and

[0046] Step 3.3, calculate O in step 3.2 i,j and The difference:

[0047]

[0048] The smaller the difference, the better the proxy neuron The more similar features can be extracted from the same category of data in different environments, for example, similar features can be extracted from pictures of the same car in sunny and rainy days, which means that the neuron has stronger generalization ability. In addition, the deeper the proxy neuron, the stronger the generalization ability and the more robust to the input data. Therefore, if the externally given hyperparameter learning rate is l max , the learning rate l of the agent neuron i Follow the following formula to make adjustments:

[0049] l i = l max *i / L, that is, the deeper the proxy neuron is, the more similar features should be extracted for the input data of different environments, and the above difference is used as the proxy neuron The loss function is back-propagated with a learning rate of l iPerform gradient descent, update the weight of the proxy neuron, apply steps 3.2 and 3.3 to all proxy neurons, and complete an iterative update of all proxy neurons.

[0050] Furthermore, in step 4, the basic model adaptation module uses the knowledge learned by the proxy neuron in the new environment in step 3 to update the corresponding neurons in the basic model based on the neuron index, including:

[0051] make For the agent neuron The update amount in step 3 (i.e. the knowledge learned) is used to update the corresponding neurons of the basic model through the neuron index

[0052]

[0053] Among them, ← is an assignment operation, and the same operation is performed on all proxy neurons to complete the update of the basic model neurons and realize the adaptation of the basic model in the drift of new environment data.

[0054] The basic model neuron granularity adaptation method for edge-side data drift described in the present invention has the following superior technical effects:

[0055] 1. The basic model neuron granularity adaptation method for edge-side data drift described in the present invention is the first neuron granularity basic model adaptation method in the edge-side data drift scenario. It extracts the most important neurons in the basic model for updating under the limited resources on the edge side, and does not need to compress the basic model. It can be adapted without losing the generalization of the basic model, and uses the high generalization of the basic model to speed up the adaptation speed and improve the adaptation accuracy. However, the existing model granularity adaptation method needs to compress the basic model before adaptation / fine-tuning under the limited resources on the edge side, which loses the powerful generalization and learning capacity of the basic model, and the adaptation converges slowly and has low accuracy.

[0056] 2. The basic model neuron granularity adaptation method for edge-side data drift described in the present invention performs model adaptation at the neuron granularity, and the neuron granularity is between the parameter granularity and the model granularity, achieving a better balance between adaptation / fine-tuning accuracy and overhead; on the other hand, the neuron granularity is tens of thousands of times finer than the model granularity, so that the adaptation method can accurately perceive new environmental data and select the neurons that are most important for accuracy for targeted adaptation; on the other hand, since neurons are the most basic computing units for feature extraction in the basic model, and the multiple parameters contained in each neuron have similar importance, a lot of redundant calculations are saved when calculating the importance. Compared with the parameter granularity adaptation method, while ensuring the adaptation accuracy, it can significantly speed up the process of finding the most important part of the basic model. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Schematic diagram of the basic model neuron granularity fine-tuning method for edge-side data drift described in an embodiment of the present invention;

[0058] Figure 2 is a flowchart of step 1 of the method described in an embodiment of the present invention;

[0059] Figure 3 is a flowchart of step 2 of the method described in an embodiment of the present invention;

[0060] Figure 4 is a flowchart of step 3 of the method described in an embodiment of the present invention;

[0061] Figure 5 is a flowchart of step 4 of the method described in an embodiment of the present invention;

[0062] Figure 6 It is a flowchart of steps 1 to 4 in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the specific implementation methods of the specification. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0064] Example

[0065] like Figure 1 As shown, the four steps of the basic model neuron granularity fine-tuning method for edge-side data drift are run on the edge side, such as a smart phone, and specifically include:

[0066] Step 1: When the edge input data drifts, the proxy neuron extraction module takes the basic model and the new environment data as input, quickly calculates the importance of neurons in the basic model, and extracts the most important neurons in the basic model as the output proxy neurons, such as Figure 2 As shown:

[0067] Step 1.1, given v data samples in the new environment to Calculate the i-th sample Information entropy

[0068]

[0069] in, represent The probability of being predicted as category c. The information entropy of a data sample represents the amount of information it contains. The higher the information entropy, the more information it contains in the new environment and the more representative it is of the new environment. After calculating the information entropy of all v data samples, this step selects the sample with the highest information entropy as the most representative sample.

[0070] Step 1.2: The most representative sample selected in step 1.1 Input the basic model for forward propagation, during which the jth neuron in the i-th layer of the basic model is calculated Importance

[0071]

[0072] Among them, F i represents the input of the i-th layer of the basic model in the forward propagation, and FBS represents the feature excitation compression module, which can calculate the importance score of each neuron in the layer according to the input of each layer. In this step, the FBS module is based on F i As input, after being processed by the average pooling layer AvgPool, the linear layer Linear and the activation function layer ReLU, the final output is a vector The jth element in That is, the jth neuron in the i-th layer of the basic model Importance score Apply this step to all neurons in the base model and calculate the importance scores of all neurons in the base model;

[0073] Step 1.3: Extract neurons with importance scores in the top ρ% from each layer of the base model as proxy neurons of the base model. is extracted as the proxy neuron, then the proxy neuron is represented as

[0074] Step 2: The neuron index building module takes the basic model and the proxy neurons output in step 1 as input, builds the neuron index between each proxy neuron and the basic model and outputs it, such as Figure 3 As shown:

[0075] Step 2.1, for a proxy neuron and all neurons in layer i of the base model Here w is the number of neurons in the i-th layer of the basic model, and a neuron index is established between them. The neuron index stores a set of weight coefficients {Γ i,j,1 ,Γ i,j,2 ,…,Γ i,j,w}, where Γ i,j,k represent and The correlation between these two factors defines the basic model neuron. To the agent neuron The mapping relationship is as follows:

[0076]

[0077] Based on the above mapping relationship, the basic model adaptation module can quickly feed back the proxy neuron update amount to the basic model neuron, apply this step to all proxy neurons, and initialize the neuron index of all proxy neurons;

[0078] Step 2.2, use the following loss function and gradient descent algorithm to solve the weight coefficient in the neuron index:

[0079]

[0080] Where L is the number of network layers in the basic model, and this step is applied to all neurons;

[0081] Step 3, the agent neuron update module uses the new environment data to update the agent neuron, such as Figure 4 As shown:

[0082] Step 3.1: Substitute the v data samples from step 1.1 to Perform data enhancement and input the basic model for forward propagation to obtain the input of the i-th layer of the basic model in this process, which is recorded as

[0083] Step 3.2, for the agent neuron The F obtained in step 1.2 i and step 3.1 Input the proxy neurons respectively to get the proxy neurons For F i and Output O i,j and

[0084] Step 3.3, calculate O in step 3.2 i,j and The difference is as follows:

[0085]

[0086] In the formula, the smaller the difference, the more likely the proxy neuron is The more similar features can be extracted from the same category of data in different environments, for example, similar features can be extracted from pictures of the same car in sunny and rainy days, which means that the proxy neuron has stronger generalization. In addition, the deeper the proxy neuron, the stronger the generalization and the more robust it is to the input data. Therefore, if the externally given hyperparameter learning rate is l max , the learning rate l of the proxy neuron i Follow the following formula to make adjustments:

[0087] l i = l max *i / L,

[0088] That is, the deeper the proxy neurons are, the more similar features they should extract for input data from different environments, and the above differences are used as proxy neurons. The loss function is back-propagated with a learning rate of l i Perform gradient descent to update the weights of the proxy neurons, apply steps 3.2 and 3.3 to all proxy neurons, and complete an iterative update of all proxy neurons;

[0089] Step 4: The basic model adaptation module uses the knowledge learned by the proxy neurons in step 3 in the new environment to update the corresponding neurons in the basic model based on the neuron index, so as to achieve the effect of basic model adaptation. Figure 5 As shown:

[0090] make For the agent neuron The update amount in step 3, that is, the knowledge learned, is used to update the corresponding neurons of the basic model through the neuron index

[0091]

[0092] Among them, ← is an assignment operation, and the same operation is performed on all proxy neurons to complete the update of the basic model neurons and realize the adaptation of the basic model in the drift of new environment data.

[0093] Figure 6 The flowchart of steps 1 to 4 of the above embodiment is shown.

[0094] In the above embodiments, the edge side refers to the portable terminal devices close to the data source and the user, such as smart phones, tablet computers, etc.; data drift means that the location of the edge device changes, resulting in changes in the distribution of the data it collects. For example, the lighting and traffic density of the images collected by the unmanned vehicle camera change, and the accent volume of the voice collected by the smart phone changes. The original model cannot handle it well, resulting in a decrease in accuracy; adaptation / fine-tuning means that after the environment changes, the basic model needs to be further trained using data in the new environment to improve its own accuracy.

[0095] In the above embodiment, the popular visual basic model Vision Transformer is used, which has 110 million parameters and can be used for edge image classification visual analysis. It includes 12 Transformer layers. In the specific implementation, the image dataset GTA5 is used for pre-training. The Volta GPU chip with a processor of 384 cores and the edge device NVIDIA Xavier NX with 16GB of available memory are used as the target edge device, and the Vision Transformer and the method described in the present invention are deployed therein. The image dataset COCO is used as the new environment dataset, representing the computer vision analysis application running on the edge side.

[0096] In step 1 of the above embodiment, when the model inputs the dataset COCO representing the new environment, the proxy neuron extraction module first selects a batch of data in the dataset, calculates the information entropy of these samples, selects the sample with the largest information entropy and inputs it into the basic model, and obtains the output F of each layer in the basic model. i , calculate the importance of all neurons in the basic model, and finally extract a small part, such as ρ = 20%, with the most important neurons as proxy neurons. The total volume is about 40MB. The time consumed by this step is approximately equal to the time of one forward propagation of the basic model, which is less than 1 second.

[0097] In step 2 of the above embodiment, for example, the neurons in the first layer of the basic model are represented as A proxy neuron is represented as Then the neuron index established between them is expressed as {Γ 1,3,1 ,Γ 1,3,2 ,…,Γ 1,3,10}, the defined mapping relationship is This step is performed for all agent neurons and takes less than 2 seconds.

[0098] In step 3 of the above embodiment, the proxy neuron update module first generates enhanced data using data enhancement methods such as rotation, scaling, adding noise, blurring, and hue change on a batch of data in step 1, and then inputs it into the basic model to obtain the output of each layer. F i and Input each agent neuron and get two outputs respectively. Assume that for the agent neuron Output O after input F1 1,3 for:

[0099] O 1,3 =[0.1,3.4,5.7,2.1,3.6,9.1,1.1,2.3,4.5,0.2],

[0100] enter Output after for:

[0101]

[0102] The difference between these two inputs is 2.48, and this sum is the value of the loss function. After back propagation and gradient descent, the proxy neuron is updated. The smaller the sum, the more robust the proxy neuron is to data drift, the more it can enhance the adaptability of the basic model to data drift and improve the accuracy of the basic model. The learning rate l of the gradient descent is max The value is 1e-4, and 500 iterations are performed, that is, the proxy neuron is updated 500 times. This step is applied to all proxy neurons, and the total time consumed for updating all proxy neurons 500 times is 287 seconds.

[0103] In step 4 of the above embodiment, the basic model adaptation module uses the knowledge learned by the proxy neuron in the new environment to update the basic model based on the neuron index to achieve basic model adaptation / fine-tuning. For example, the proxy neuron The knowledge learned in step 3 is expressed as To use the knowledge of the proxy neurons to update the neurons in the first layer of the base model

[0104]

[0105] The above operations were performed on all proxy neurons, which took about 3 seconds to complete the adaptation / fine-tuning of the basic model. The accuracy of the basic model was improved from 0.13 to 0.55.

[0106] Through tests on a variety of basic models with billions of parameters and edge devices, the basic model neuron granularity fine-tuning method for edge-side data drift performs excellently in the basic model tuning scenario for edge-side data drift. Compared with the most advanced parameter granularity or model granularity adaptation algorithms of predecessors, the accuracy is improved by an average of 21.88% and a maximum of 35.80%, the training memory usage is reduced by an average of 27.14%, and the energy consumption is reduced by an average of 65.65%. This shows that the specific method of neuron granularity training of the basic model neuron granularity fine-tuning method for edge-side data drift described in the present invention can better utilize the available resources on the edge to complete the basic model fine-tuning and improve the accuracy of the basic model, and has higher availability on the edge with limited resources.

[0107] The present invention is not limited by the above-mentioned embodiments. The above-mentioned embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which shall all fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims.

Claims

1. A basic model neuron granularity adaptation method for edge-side data drift, comprising the following steps: Step 1: When the edge input data drifts, the proxy neuron extraction module takes the basic model and new environment data as input, quickly calculates the importance of neurons in the basic model, and extracts the most important neurons in the basic model as the output proxy neurons; Step 2: The neuron index building module takes the basic model and the proxy neuron output in step 1 as input, builds the neuron index between each proxy neuron and the basic model and outputs it; Step 3, the agent neuron update module updates the agent neuron using the new environment data; Step 4: The basic model adaptation module uses the knowledge learned by the proxy neurons in step 3 in the new environment to update the corresponding neurons in the basic model based on the neuron index, thereby achieving the effect of basic model adaptation.

2. According to the basic model neuron granularity adaptation method for edge-side data drift as claimed in claim 1, in step 1, the proxy neuron extraction module extracts the proxy neuron, specifically comprising: Step 1.1, given v data samples in the new environment to Calculate the i-th sample Information entropy In the above formula, represent The probability of being predicted as category c. The information entropy of a data sample represents the amount of information it contains. The sample with higher information entropy contains more information in the new environment and is more representative of the new environment. After calculating the information entropy of all v data samples, this step selects the sample with the highest information entropy as the most representative sample. Step 1.2: The most representative sample selected in step 1.1 Input the basic model for forward propagation, during which the jth neuron in the i-th layer of the basic model is calculated Importance As shown below: In the above formula, F i represents the input of the i-th layer of the basic model in the forward propagation, and FBS represents the feature excitation compression module. The importance score of each neuron in each layer is calculated according to the input of each layer. In this step, the FBS module is based on F i As input, after being processed by the average pooling layer AvgPool, the linear layer Linear and the activation function layer ReLU, the final output is a vector The jth element in That is, the jth neuron in the i-th layer of the basic model Importance score Apply this step to all neurons in the base model and calculate the importance scores of all neurons in the base model; Step 1.3: Extract neurons with importance scores in the top ρ% from each layer of the base model as proxy neurons of the base model. is extracted as the proxy neuron, then the proxy neuron is represented as 3. According to the basic model neuron granularity adaptation method for edge-side data drift according to claim 1, in step 2, the neuron index construction module constructs a neuron index between each proxy neuron and the basic model, specifically comprising: Step 2.1, for a proxy neuron and all neurons in layer i of the base model (w is the number of neurons in the i-th layer of the basic model), a neuron index is established between them, and the neuron index stores a set of weight coefficients {Γ i,j,1 ,Γ i,j,2 ,…,Γ i,j,w }, where Γ i,j,k represent and The correlation between these two factors defines the basic model neuron. To the agent neuron The mapping relationship is as shown below: Based on the above mapping relationship, the basic model adaptation module can quickly feed back the proxy neuron update amount to the basic model neuron, apply this step to all proxy neurons, and initialize the neuron index of all proxy neurons; Step 2.2, use the loss function and gradient descent algorithm to solve the weight coefficient in the neuron index: Where L is the number of network layers in the base model, and this step is applied to all neurons.

4. According to the basic model neuron granularity adaptation method for edge-side data drift as claimed in claim 1, in step 3, the proxy neuron update module uses the new environment data to train the proxy neuron for several iterations, and each iteration includes the following specific steps: Step 3.1: Substitute the v data samples from step 1.1 to Perform data enhancement and input the basic model for forward propagation to obtain the input of the i-th layer of the basic model in this process, which is recorded as Step 3.2, for the agent neuron The F obtained in step 1.2 i and step 3.1 Input the neuron respectively to get the neuron For F i and Output O i,j and Step 3.3, calculate O in step 3.2 i,j and The difference: As shown in the above formula, the smaller the difference, the stronger the proxy neuron. The more similar features can be extracted from the same category of data in different environments, the stronger the generalization of the proxy neuron. The deeper the proxy neuron, the stronger the generalization and the more robust to the input data. If the externally given hyperparameter learning rate is l max , then the learning rate of the proxy neuron is l i Follow the following formula to make adjustments: l i =l max *i / L, That is, the deeper the proxy neuron is, the more similar features should be extracted for the input data of different environments, and the above difference is used as the proxy neuron The loss function is back-propagated with a learning rate of l i Perform gradient descent, update the weights of the proxy neurons, apply steps 3.2 and 3.3 to all proxy neurons, and complete an iterative update of all proxy neurons.

5. According to the basic model neuron granularity adaptation method for edge-side data drift according to claim 1, in step 4, the basic model adaptation module uses the knowledge learned by the proxy neuron in the new environment in step 3 to update the corresponding neurons in the basic model based on the neuron index, including: make For the agent neuron The knowledge learned from the update amount in step 3 is used to update the corresponding neurons of the basic model through the neuron index Among them, ← is an assignment operation, and the same operation is performed on all proxy neurons to complete the update of the basic model neurons and realize the adaptation of the basic model in the drift of new environment data.

Images