An edge-oriented side data drift-based basic model neuron granularity adaptation method

Abstract

The application relates to a basic model neuron granularity adaptation method for edge side data drift, which comprises the following steps: step 1, after detecting that edge side input data drift occurs, an agent neuron extraction module takes a basic model and new environment data as input, quickly calculates the importance of neurons in the basic model, and extracts the most important neurons in the basic model; step 2, a neuron index construction module takes the basic model and the agent neurons output in step 1 as input, constructs a neuron index between each agent neuron and the basic model, and outputs; step 3, an agent neuron update module updates the agent neurons using new environment data; and the like. The superior technical effect of the application is that the most important neurons in the basic model are extracted for updating under limited resources at the edge side, the basic model does not need to be compressed, and adaptation can be carried out without losing the generalization of the basic model, so that the adaptation speed is accelerated and the adaptation accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a basic model neuron granularity adaptation method for edge-side data drift. Background Art

[0002] With the advancement of deep learning technology, the introduction of foundation models marks a major transformation in the field of artificial intelligence. Foundation models typically have hundreds of millions or even hundreds of billions of parameters, efficiently capture contextual information, and possess powerful representational and reasoning capabilities. Foundation models include large language models (such as ChatGPT), large vision models, and large multimodal models. They demonstrate complex capabilities and features not found in traditional non-foundation models in tasks such as human-computer dialogue, object detection, speech recognition, and image-text question-answering. They exhibit human-like thinking and intelligence, and are therefore considered the cornerstone of next-generation artificial intelligence. A key difference between foundation models and traditional non-foundation models is that they are typically pre-trained on cloud servers using massive amounts of data. However, these pre-trained models only store highly generalized knowledge and perform poorly on data from specific tasks and domains. Therefore, foundation models need to be "adapted" or "fine-tuned" on data from the target task and domain to better adapt to the data distribution of the target task and domain for proper operation.

[0003] The adaptation / fine-tuning here mainly refers to the process of updating all or part of the parameters in the pre-trained basic model on a specific dataset to improve its accuracy on the specific dataset. For example, the visual basic model Vision Transformer pre-trained on massive image data lacks accurate reasoning capabilities for specific fields (such as traffic scene pictures). At this time, it is necessary to use the traffic scene picture dataset to fine-tune all or part of the key parameters of the basic model. The common term "training" involves a wider range of fields. In addition to adaptation / fine-tuning, it can also refer to pre-training processes, etc. The present invention mainly focuses on the field of adaptation / fine-tuning of the basic model.

[0004] Most calls to the basic model come from users or devices on the edge (such as mobile phones and tablets). In order to quickly respond to requests, avoid network transmission delays, and protect data privacy, the basic model can be deployed on the edge after initial adaptation / fine-tuning. However, the environment on the edge may continue to change dynamically, causing the distribution of model input data to continue to evolve, which is called the "data drift" phenomenon on the edge, which will significantly reduce the accuracy of the basic model. For example, an edge-side unmanned vehicle may continue to move in different geographical locations (open roads, congested roads, and tunnels, etc.) and weather conditions (overcast, cloudy, and rainy, etc.). At this time, the recognition accuracy of the basic model in the unmanned vehicle may be greatly reduced, affecting the perception ability and driving safety of the unmanned vehicle. At this time, it is necessary to continuously use data from the new environment to further adapt / fine-tune the basic model to maintain the high accuracy of the basic model.

[0005] Basic model adaptation / fine-tuning for edge-side data drift includes the following three mainstream methods:

[0006] (1) Efficient adaptation / fine-tuning method of basic model parameters. This type of method adds a small number of trainable model parameters to each network layer in the basic model, freezes other parameters during fine-tuning, and only updates the newly added parameters, that is, training at the "parameter" granularity, so as to achieve the purpose of speeding up fine-tuning and reducing resource usage. For example, add a low-rank matrix to each layer, or add a magnitude vector and direction matrix, and then only train these newly added matrices and vectors. This type of method has a key problem when fine-tuning the basic model under edge-side data drift: because the number of basic model parameters is in the hundreds of millions and is huge, it is impossible to quickly calculate the importance of each parameter based on the newly arrived data under limited resources, and select the most important parameters to update. This may cause parameters with very low accuracy to occupy the computing resources used for adaptation, reducing the adaptation efficiency of the basic model under limited resources on the edge side, and low adaptation accuracy.

[0007] (2) Static small model generation and adaptation / fine-tuning methods. This type of method first compresses the base model into a small model with a fixed structure and size before deployment, and then uses the new environment data to train all the parameters of the small model, thereby speeding up training and reducing resource usage. For example, in the new environment, the knowledge of the base model is distilled to the small model, and the drift of the label distribution can be considered to optimize the adaptation algorithm. In addition, the small model that has been adapted in the new environment in the past can be stored and reused in the new environment in the future, reducing unnecessary adaptation costs.

[0008] (3) Dynamic small model generation and adaptation / fine-tuning methods. This type of method uses the dynamic scaling method of the model architecture in the inference phase to generate a small model from the base model before fine-tuning begins, and then uses the new environment data to train all the parameters of the small model. The dynamic scaling method of the model architecture in the inference phase refers to dynamically adjusting the model size according to the available resources or input data during inference, thereby adjusting the trade-off between model accuracy and resource usage under limited resources. For example, by allowing model inference to exit from the middle layer of the model early, generating a series of candidate models of different sizes in advance, searching for the best model structure in advance according to the hardware architecture and operation constraints, and dynamically increasing or decreasing the filters involved in the calculation in the model during runtime. The latest paper of this type of method, "ElasticDNN: On-Device Neural Network Remodeling for Adapting Evolving Vision Domains at Edge", describes a method that first reduces the original base model to a small model when the environment changes, then trains the small model using new environment data, and finally feeds the knowledge learned by the small model back to the large model.

[0009] The above static or dynamic small model generation and adaptation / fine-tuning methods all scale and train the base model at the "model granularity"—that is, compressing the entire base model into a small model and then training the small model. However, compressing the base model compromises its strong generalization performance, resulting in slow learning speed and limited learning capacity of the compressed small model during training, ultimately leading to low accuracy in base model adaptation / fine-tuning.

[0010] In summary, existing basic model adaptation / fine-tuning methods for edge-side data drift operate at a "parameter" or "model" granularity. The fine-grained parameters make it expensive to find the most important parameters, making it difficult to quickly perceive newly arriving data and make targeted adaptations to the most important parameters. The coarse-grained models, however, cannot effectively preserve the generalization performance of the basic model, resulting in limited learning capacity.

[0011] For example, the patent application document with Chinese invention patent application number CN202411141783.5 discloses a large model fine-tuning method, apparatus, equipment, medium and program product. For each self-attention layer in the large model, a reparameterization factor is constructed. Based on the reparameterization factor, a static multi-head self-attention weight is constructed. Based on the static multi-head self-attention weight and the initial self-attention map, a first fine-tuning module is determined. Based on the target feedforward neural network layer and the low-rank adapter parameters, a second fine-tuning module is determined. Based on the first fine-tuning module and the second fine-tuning module, some parameters of the large model are adjusted on the training data. This method is an efficient adaptation / fine-tuning method for basic model parameters at parameter granularity. It cannot quickly perceive the dynamic input data on the edge side to make targeted fine-tuning of the most important parameters. The efficiency of learning new data is limited, and the adaptation / fine-tuning accuracy is low under limited resources on the edge side.

[0012] For another example, the patent application document with Chinese invention patent application number CN202410480665.0 discloses a training method and device for a large language model in the vertical field of e-commerce. According to the data ratio and pre-training data set, the LoRA method is used to continue pre-training the existing open source large language model to obtain a basic model, and the fine-tuning data set is used to fine-tune the basic model with multi-task instructions to obtain the fine-tuned basic model, i.e., the large language model in the vertical field of e-commerce. This method is an efficient adaptation / fine-tuning method for basic model parameters at parameter granularity. It cannot quickly perceive the dynamic input data on the edge side to make targeted fine-tuning of the most important parameters. The efficiency of learning new data is limited, and the adaptation / fine-tuning accuracy is low under limited resources on the edge side.

[0013] For another example, the Chinese invention patent application document with the application number CN202411120027.4 discloses a multimodal large model training optimization method and device for the power vertical field. After the cloud server compresses the basic model, the compressed small model is updated and tuned using a low-rank fine-tuning method. This method belongs to the static small model generation and adaptation / fine-tuning method at the model granularity. The compression of the basic model on the cloud server impairs its generalization performance for dynamic input data on the edge side, and the small model has limited learning capacity. Therefore, the adaptation / fine-tuning accuracy is low under the limited resources on the edge side. Summary of the Invention

[0014] In view of the defects of the existing technology in this technical field, the present invention proposes a basic model neuron granularity adaptation method for edge-side data drift, and proposes a "neuron granularity" training method. The neuron granularity is between the parameter granularity and the model granularity, achieving a better balance between adaptation / fine-tuning accuracy and overhead. Neurons are the most basic computing units for feature extraction in the basic model. A neuron contains multiple model parameters, and multiple neurons constitute a basic model. The model parameters in a neuron have similar important characteristics.

[0015] The basic model neuron granularity adaptation method for edge-side data drift described in the present invention includes the following two key data structures:

[0016] Proxy neurons: A small number of neurons in the base model that are most important for accuracy in new environments. Training these neurons has the greatest benefit for the accuracy of the base model. They are identified and extracted for separate training.

[0017] Neuron index: Built between the base model and the proxy neurons, it is used to feed back the update amount of the proxy neurons to the base model to update the relevant neurons of the base model, thereby achieving rapid adaptation / fine-tuning of the base model without compromising the generalization of the base model.

[0018] The present invention proposes a neuron granularity adaptation method for the basic model facing edge-side data drift, and proposes a neuron granularity training method, which includes the following functional modules:

[0019] Proxy neuron extraction module: Based on the data distribution in the new environment, the most representative data sample is first selected. Then, based on this sample, the importance of each neuron in the base model in the new environment is quickly calculated. The higher the importance, the greater its contribution to accuracy and the more worthy of updating during fine-tuning. Then, the most important few neurons in the base model are extracted as proxy neurons.

[0020] Neuron index construction module: Builds a neuron index between the base model and each proxy neuron. Each neuron index is a one-to-many index that connects a proxy neuron to all neurons in the same layer of the base model. It also defines how the model parameters in the base model neurons are mapped to the model parameters in the corresponding proxy neurons through linear combination operations.

[0021] Agent neuron update module: Based on the contrastive learning loss function, each agent neuron is individually updated with new environment data;

[0022] Basic model adaptation module: Through neuron indexing, the update amount of the proxy neuron is quickly fed back to the basic model to update the relevant neurons of the basic model, thereby achieving rapid fine-tuning of the basic model without compromising the generalization of the basic model.

[0023] The basic model neuron granularity adaptation method for edge-side data drift described in the present invention comprises the following steps:

[0024] Step 1: When edge-side input data drift is detected, the proxy neuron extraction module takes the base model and new environment data as input, quickly calculates the importance of neurons in the base model, and extracts the most important neurons in the base model as the output proxy neurons;

[0025] Step 2: The neuron index construction module takes the base model and the proxy neurons output in step 1 as input, constructs the neuron index between each proxy neuron and the base model, and outputs it.

[0026] Step 3: The agent neuron update module updates the agent neuron using the new environment data;

[0027] In step 4, the basic model adaptation module uses the knowledge learned by the proxy neurons in step 3 in the new environment to update the corresponding neurons in the basic model based on the neuron index, thereby achieving the effect of basic model adaptation.

[0028] Furthermore, the proxy neuron extraction module in step 1 extracts the proxy neuron, specifically including:

[0029] Step 1.1, given v data samples in the new environment to Calculate the i-th sample Information entropy

[0030]

[0031] In the above formula, represent The probability of being predicted as category c. The information entropy of a data sample represents the amount of information it contains. The higher the information entropy, the more information the sample contains in the new environment and the more representative it is of the new environment. After calculating the information entropy of all v data samples, this step selects the sample with the highest information entropy as the most representative sample.

[0032] Step 1.2: The most representative sample selected in step 1.1 Input the basic model for forward propagation, during which the jth neuron in the i-th layer of the basic model is calculated Importance As shown in the following formula:

[0033]

[0034] In the above formula, Fi represents the input of the i-th layer of the basic model in the forward propagation, and FBS (Feature Boosting and Suppression) represents the feature boosting and compression module. The importance score of each neuron in each layer is calculated according to the input of each layer. In this step, the FBS module is based on F i As input, after processing by the average pooling layer AvgPool, the linear layer Linear and the activation function layer ReLU, the final output is a vector The jth element in That is, the jth neuron in the i-th layer of the basic model Importance score Apply this step to all neurons in the base model and calculate the importance scores of all neurons in the base model;

[0035] Step 1.3: Extract the neurons with the importance scores in the top ρ% from each layer of the base model as the proxy neurons of the base model. is extracted as the proxy neuron, then the proxy neuron is represented as

[0036] Furthermore, the neuron index construction module in step 2 constructs a neuron index between each proxy neuron and the basic model, specifically including:

[0037] Step 2.1, for a certain agent neuron and all neurons in layer i of the base model (w is the number of neurons in the i-th layer of the basic model), a neuron index is established between them, and the neuron index stores a set of weight coefficients {Γ i,j,1 ,Γ i,j,2 ,…,Γ i,j,w}, where Γ i,j,k represent and The correlation between these two sets of weight coefficients defines the basic model neurons To the agent neuron The mapping relationship is shown as follows:

[0038]

[0039] Based on the above mapping relationship, the basic model adaptation module can quickly feed back the proxy neuron update amount to the basic model neuron. This step is applied to all proxy neurons to initialize the neuron index of all proxy neurons.

[0040] Step 2.2, use the loss function and gradient descent algorithm to solve the weight coefficient in the neuron index:

[0041]

[0042] Where L is the number of network layers in the base model, and step 2.2 is applied to all neurons.

[0043] Furthermore, in step 3, the proxy neuron update module uses the new environment data to train the proxy neuron for several iterations, each of which specifically includes:

[0044] Step 3.1: Substitute the v data samples from step 1.1 to Perform data enhancement and input the basic model for forward propagation to obtain the input of the i-th layer of the basic model in this process, which is recorded as

[0045] Step 3.2, for the agent neuron The F obtained in step 1.2 i and the one obtained in step 3.1 Input the neuron respectively and get the neuron For F i and Output O i,j and

[0046] Step 3.3, calculate O in step 3.2 i,j and The difference:

[0047]

[0048] The smaller the difference, the better the proxy neuron The more similar features can be extracted from the same category of data in different environments, for example, similar features can be extracted from pictures of the same car in sunny and rainy days, the stronger the generalization of the neuron. In addition, the deeper the proxy neuron, the stronger its generalization and the more robust it should be to the input data. Therefore, if the externally given hyperparameter learning rate is l max , the learning rate l of the agent neuron i Adjust according to the following formula:

[0049] l i =l max *i / L, that is, the deeper the proxy neuron is, the more similar features should be extracted for the input data of different environments, and the above difference is used as the proxy neuron The loss function is back-propagated with a learning rate of l iPerform gradient descent to update the weight of the proxy neuron, apply steps 3.2 and 3.3 to all proxy neurons, and complete an iterative update of all proxy neurons.

[0050] Furthermore, in step 4, the basic model adaptation module uses the knowledge learned by the proxy neurons in the new environment in step 3 to update the corresponding neurons in the basic model based on the neuron index, including:

[0051] make Proxy neurons The update amount in step 3 (i.e. the knowledge learned) is used to update the corresponding neurons of the basic model through the neuron index

[0052]

[0053] Among them, ← is an assignment operation, and the same operation is performed on all proxy neurons to complete the update of the basic model neurons and realize the adaptation of the basic model to the drift of new environment data.

[0054] The basic model neuron granularity adaptation method for edge-side data drift described in the present invention has the following superior technical effects:

[0055] 1. The basic model neuron granularity adaptation method for edge-side data drift described in the present invention is the first neuron-granularity basic model adaptation method in the edge-side data drift scenario. It extracts the most important neurons in the basic model for updating under limited resources on the edge side, without compressing the basic model. It can adapt without losing the generalization of the basic model, and utilizes the high generalization of the basic model to accelerate the adaptation speed and improve the adaptation accuracy. However, the existing model granularity adaptation method requires compressing the basic model before adaptation / fine-tuning under limited resources on the edge side, which loses the powerful generalization and learning capacity of the basic model, resulting in slow adaptation convergence and low accuracy.

[0056] 2. The basic model neuron granularity adaptation method for edge-side data drift described in the present invention performs model adaptation at the neuron granularity, which is between the parameter granularity and the model granularity, achieving a better balance between adaptation / fine-tuning accuracy and overhead; on the other hand, the neuron granularity is tens of thousands of times finer than the model granularity, so that the adaptation method can accurately perceive new environmental data and select the neurons that are most important for accuracy for targeted adaptation; on the other hand, since neurons are the most basic computing units for feature extraction in the basic model, and the multiple parameters contained in each neuron have similar importance, a lot of redundant calculations are saved when calculating the importance. Compared with the parameter granularity adaptation method, while ensuring the adaptation accuracy, it can significantly speed up the process of finding the most important part of the basic model. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a schematic diagram of the basic model neuron granularity fine-tuning method for edge side data drift described in the embodiments of the present application;

[0058] Figure 2 is a flowchart of step 1 of the method described in the embodiments of the present application;

[0059] Figure 3 is a flowchart of step 2 of the method described in the embodiments of the present application;

[0060] Figure 4 is a flowchart of step 3 of the method described in the embodiments of the present application;

[0061] Figure 5 is a flowchart of step 4 of the method described in the embodiments of the present application;

[0062] Figure 6 is a flowchart of steps 1 to 4 in the embodiments of the present application. DETAILED DESCRIPTION

[0063] In order to enable a clearer understanding of the above-mentioned purposes, features and advantages of the present application, the present application will be further described in detail below in conjunction with the specific embodiments of the description. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0064] EMBODIMENT

[0065] As shown in Figure 1 , the four steps of the basic model neuron granularity fine-tuning method for edge side data drift run on the edge side, for example, a smart phone, and specifically include:

[0066] Step 1, when the drift of the input data on the edge side is detected, the proxy neuron extraction module takes the basic model and the new environment data as input, quickly calculates the importance of the neurons in the basic model, extracts the most important neurons in the basic model as the output proxy neurons, as shown in Figure 2 .

[0067] Step 1.1, given v data samples in the new environment to Calculate the information entropy of the i-th sample .

[0068]

[0069] wherein, represents The probability of being predicted as category c. The information entropy of a data sample represents the amount of information it contains. The higher the information entropy, the more information the sample contains in the new environment and the more representative it is of the new environment. After calculating the information entropy of all v data samples, this step selects the sample with the highest information entropy as the most representative sample.

[0070] Step 1.2: The most representative sample selected in step 1.1 Input the basic model for forward propagation, during which the jth neuron in the i-th layer of the basic model is calculated Importance

[0071]

[0072] Among them, F i represents the input of the i-th layer of the basic model in the forward propagation, and FBS represents the feature excitation compression module, which can calculate the importance score of each neuron in the layer according to the input of each layer. In this step, the FBS module is based on F i As input, after processing by the average pooling layer AvgPool, the linear layer Linear and the activation function layer ReLU, the final output is a vector The jth element in That is, the jth neuron in the i-th layer of the basic model Importance score Apply this step to all neurons in the base model and calculate the importance scores of all neurons in the base model;

[0073] Step 1.3: Extract the neurons with the importance scores in the top ρ% from each layer of the base model as the proxy neurons of the base model. is extracted as the proxy neuron, then the proxy neuron is represented as

[0074] Step 2: The neuron index construction module takes the basic model and the proxy neurons output in step 1 as input, constructs the neuron index between each proxy neuron and the basic model and outputs it, such as Figure 3 As shown:

[0075] Step 2.1, for a certain agent neuron and all neurons in layer i of the base model Here w is the number of neurons in the i-th layer of the basic model, and a neuron index is established between them. The neuron index stores a set of weight coefficients {Γ i,j,1 ,Γ i,j,2 ,…,Γ i,j,wwherein, Γ i,j,k represents and the correlation between the weight coefficients, the set of weight coefficients defines the mapping relationship of the base model neurons to the agent neurons , as follows:

[0076]

[0077] Based on the mapping relationship as above, the base model adaptation module can quickly feed back the agent neuron update amount to the base model neuron, apply this step to all agent neurons, and initialize the neuron index of all agent neurons.

[0078] Step 2.2, using the following loss function and gradient descent algorithm, solve the weight coefficients in the neuron index:

[0079]

[0080] wherein, L is the number of network layers in the base model, apply this step to all neurons.

[0081] Step 3, the agent neuron update module updates the agent neurons using new environment data, as shown in Figure 4 .

[0082] Step 3.1, data augmentation is performed on the v data samples in step 1.1 to , input the base model for forward propagation, and get the input of the i-th layer of the base model in this process, denoted as

[0083] Step 3.2, for agent neuron , input F i obtained in step 1.2 and obtained in step 3.1 into the agent neuron, respectively, to obtain the agent neuron For the output O i,j and of F i and

[0084] Step 3.3, calculate the difference between O i,j and in step 3.2, as follows:

[0085]

[0086] In the formula, the smaller the difference, the better the agent neuron The more similar features can be extracted from the same category of data in different environments, for example, similar features can be extracted from pictures of the same car on sunny days and rainy days, the stronger the generalization of the proxy neuron. In addition, the deeper the proxy neuron, the stronger its generalization should be and the more robust it should be to the input data. Therefore, if the externally given hyperparameter learning rate is l max , the learning rate l of the agent neuron i Adjust according to the following formula:

[0087] l i =l max *i / L,

[0088] That is, the deeper the proxy neurons are, the more similar features should be extracted for the input data of different environments, and the above difference is used as the proxy neuron The loss function is back-propagated with a learning rate of l i Perform gradient descent to update the weights of the proxy neurons, apply steps 3.2 and 3.3 to all proxy neurons, and complete an iterative update of all proxy neurons;

[0089] Step 4: The basic model adaptation module uses the knowledge learned by the proxy neurons in step 3 in the new environment to update the corresponding neurons in the basic model based on the neuron index, so as to achieve the effect of basic model adaptation. Figure 5 As shown:

[0090] make Proxy neurons The update amount in step 3, that is, the knowledge learned, is used to update the corresponding neurons of the basic model through the neuron index

[0091]

[0092] Among them, ← is an assignment operation, which is performed on all proxy neurons in the same way to complete the update of the basic model neurons and realize the adaptation of the basic model to the drift of new environment data.

[0093] Figure 6 The flowchart of steps 1 to 4 of the above embodiment is shown.

[0094] In the above embodiment, the edge side refers to portable terminal devices close to the data source and the user, such as smartphones and tablets; data drift means that the location of the edge device changes, resulting in changes in the distribution of the data it collects. For example, the lighting and traffic density of the images collected by the unmanned vehicle camera change, and the accent and volume of the voice collected by the smartphone change. The original model cannot handle it well, resulting in a decrease in accuracy; adaptation / fine-tuning means that after the environment changes, the basic model needs to use data from the new environment for further training to improve its own accuracy.

[0095] In the above embodiment, the popular visual basic model Vision Transformer is adopted, which has 110 million parameters and can be used for edge image classification visual analysis. It includes 12 Transformer layers. In the specific implementation, the image dataset GTA5 is used for pre-training. A Volta GPU chip with a processor of 384 cores and an edge device NVIDIA Xavier NX with 16GB of available memory is used as the target edge device, and the Vision Transformer and the method described in the present invention are deployed therein. The image dataset COCO is used as a new environment dataset, representing a computer vision analysis application running on the edge side.

[0096] In step 1 of the above embodiment, when the model inputs the dataset COCO representing the new environment, the proxy neuron extraction module first selects a batch of data from the dataset, calculates the information entropy of these samples, selects the sample with the largest information entropy and inputs it into the basic model, and obtains the output F of each layer in the basic model. i , calculate the importance of all neurons in the basic model, and finally extract a small part, such as ρ = 20%, with the most important neurons as proxy neurons. The total volume is about 40MB. The time consumed by this step is approximately equal to the time of one forward propagation of the basic model, which is less than 1 second.

[0097] In step 2 of the above embodiment, for example, the neurons in the first layer of the basic model are represented as An agent neuron is represented as Then the neuron index established between them is expressed as {Γ 1,3,1 ,Γ 1,3,2 ,…,Γ 1,3,10}, the defined mapping relationship is This step is performed for all agent neurons and takes less than 2 seconds.

[0098] In step 3 of the above embodiment, the proxy neuron update module first generates enhanced data using data enhancement methods such as rotation, scaling, adding noise, blurring, and hue change on the batch of data in step 1, and then inputs it into the basic model to obtain the output of each layer. F i and Input each agent neuron and get two outputs respectively. Assume that for the agent neuron Output O after input F1 1,3 for:

[0099] O 1,3 =[0.1,3.4,5.7,2.1,3.6,9.1,1.1,2.3,4.5,0.2],

[0100] enter Output after for:

[0101]

[0102] The difference between these two inputs is 2.48, which is the value of the loss function. After back propagation and gradient descent, the agent neuron is updated. The smaller the sum, the stronger the robustness of the proxy neuron to data drift, the more it can enhance the adaptability of the basic model to data drift and improve the accuracy of the basic model. max The update time is 1e-4, and 500 iterations are performed, that is, the proxy neuron is updated 500 times. This step is applied to all proxy neurons, and the total time taken to update all proxy neurons 500 times is 287 seconds.

[0103] In step 4 of the above embodiment, the basic model adaptation module uses the knowledge learned by the proxy neuron in the new environment to update the basic model based on the neuron index to achieve basic model adaptation / fine-tuning, for example, the proxy neuron The knowledge learned in step 3 is represented as To use the knowledge of the proxy neurons to update the neurons in the first layer of the base model

[0104]

[0105] Performing the above operations on all proxy neurons takes about 3 seconds to complete the adaptation / fine-tuning of the base model, and the accuracy of the base model is improved from 0.13 to 0.55.

[0106] Through tests on a variety of basic models with billions of parameters and edge devices, the basic model neuron granularity fine-tuning method for edge-side data drift performs excellently in the basic model tuning scenario for edge-side data drift. Compared with the most advanced parameter granularity or model granularity adaptation algorithm of predecessors, the accuracy is improved by an average of 21.88% and a maximum of 35.80%, the training memory usage is reduced by an average of 27.14%, and the energy consumption is reduced by an average of 65.65%. This shows that the specific method of neuron granularity training of the basic model neuron granularity fine-tuning method for edge-side data drift described in the present invention can better utilize the available resources on the edge to complete the basic model fine-tuning and improve the accuracy of the basic model, and has higher availability on the edge with limited resources.

[0107] The present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustration of the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements. These changes and improvements shall fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the appended claims.

Claims

1. A method for adapting the neuron granularity of a basic model for edge-side image data drift, comprising the following steps: Step 1: When drift of edge-side input data is detected, the proxy neuron extraction module takes the basic model and new environment data as input, quickly calculates the importance of neurons in the basic model, and extracts the most important neurons in the basic model as the output proxy neurons. Both the input data and the new environment data are image data. Step 2: The neuron index construction module takes the base model and the proxy neurons output in step 1 as input, constructs the neuron index between each proxy neuron and the base model, and outputs it. Step 3: The agent neuron update module updates the agent neuron using the new environment data; In step 4, the basic model adaptation module uses the knowledge learned by the proxy neurons in step 3 in the new environment to update the corresponding neurons in the basic model based on the neuron index, thereby achieving the effect of basic model adaptation.

2. According to the basic model neuron granularity adaptation method for edge-side image data drift according to claim 1, in step 1, the proxy neuron extraction module extracts the proxy neuron, specifically comprising: Step 1.1, given v data samples in the new environment to Calculate the i-th sample Information entropy In the above formula, represent The probability of being predicted as category c. The information entropy of a data sample represents the amount of information it contains. The higher the information entropy, the more information the sample contains in the new environment and the more representative it is of the new environment. After calculating the information entropy of all v data samples, this step selects the sample with the highest information entropy as the most representative sample. Step 1.2: The most representative sample selected in step 1.1 Input the basic model for forward propagation, during which the jth neuron in the i-th layer of the basic model is calculated Importance As shown in the following formula: In the above formula, F i represents the input of the i-th layer of the basic model in the forward propagation, and FBS represents the feature excitation compression module. The importance score of each neuron in each layer is calculated based on the input of each layer. In this step, the FBS module is based on F i As input, after processing by the average pooling layer AvgPool, the linear layer Linear and the activation function layer ReLU, the final output is a vector The jth element in That is, the jth neuron in the i-th layer of the basic model Importance score Apply this step to all neurons in the base model and calculate the importance scores of all neurons in the base model; Step 1.3: Extract the neurons with the importance scores in the top ρ% from each layer of the base model as the proxy neurons of the base model. is extracted as the proxy neuron, then the proxy neuron is represented as 3. According to the method for adapting the neuron granularity of the basic model for edge-side image data drift according to claim 1, in step 2, the neuron index construction module constructs a neuron index between each proxy neuron and the basic model, specifically comprising: Step 2.1, for a certain agent neuron and all neurons in layer i of the base model (w is the number of neurons in the i-th layer of the basic model), a neuron index is established between them, and the neuron index stores a set of weight coefficients {Γ i,j,1 ,Γ i,j,2 ,…,Γ i,j,w }, where Γ i,j,k represent and The correlation between these two sets of weight coefficients defines the basic model neurons To the agent neuron The mapping relationship is shown as follows: Based on the above mapping relationship, the basic model adaptation module can quickly feed back the proxy neuron update amount to the basic model neuron. This step is applied to all proxy neurons to initialize the neuron index of all proxy neurons. Step 2.2, use the loss function and gradient descent algorithm to solve the weight coefficient in the neuron index: Where L is the number of network layers in the base model, and this step is applied to all neurons.

4. According to the basic model neuron granularity adaptation method for edge-side image data drift according to claim 1, in step 3, the proxy neuron update module uses the new environment data to train the proxy neuron for several iterations, each iteration comprising the following specific steps: Step 3.1: Substitute the v data samples from step 1.1 to Perform data enhancement and input the basic model for forward propagation to obtain the input of the i-th layer of the basic model in this process, which is recorded as Step 3.2, for the agent neuron The F obtained in step 1.2 i and the one obtained in step 3.1 Input the neuron respectively and get the neuron For F i and Output O i,j and Step 3.3, calculate O in step 3.2 i,j and The difference: As shown in the above formula, the smaller the difference, the more likely the agent neuron is. The more similar features can be extracted from the same category of data in different environments, the stronger the generalization of the proxy neuron. The deeper the proxy neuron, the stronger its generalization should be and the more robust it should be to the input data. If the externally given hyperparameter learning rate is l max , then the learning rate of the agent neuron l i Adjust according to the following formula: l i =l max *i / L, That is, the deeper the proxy neuron is, the more similar features should be extracted for the input data of different environments, and the above difference is used as the proxy neuron The loss function is back-propagated with a learning rate of l i Perform gradient descent to update the weights of the proxy neurons, apply steps 3.2 and 3.3 to all proxy neurons, and complete an iterative update of all proxy neurons.

5. According to the method for adapting the neuron granularity of the base model for edge-side image data drift according to claim 3, in step 4, the base model adaptation module uses the knowledge learned by the proxy neurons in step 3 in the new environment to update the corresponding neurons in the base model based on the neuron index, including: make Proxy neurons The knowledge learned from the update amount in step 3 is used to update the corresponding neurons of the basic model through the neuron index Among them, ← is an assignment operation, and the same operation is performed on all proxy neurons to complete the update of the basic model neurons and realize the adaptation of the basic model to the drift of new environment data.

Images