End-side neural network efficient fine tuning method and system for microcomputer

Abstract

The invention discloses a microcomputer-oriented end-side neural network efficient fine tuning method and system, and belongs to the technical field of end-side neural network fine tuning. Comprises: acquiring an image; processing the image through a trained neural network to generate an image recognition result; wherein when the neural network is trained, on the basis of a pre-training model deployed on end-side equipment, the stability of parameter iteration is utilized, the relative change of parameters in a training period is determined, the importance of momentum factors and generated parameters is combined, and updated parameters are dynamically selected; meanwhile, the back propagation process is optimized through a gradient compression method. According to the method, unnecessary calculation overhead can be reduced, the training efficiency is improved, the requirements for calculation resources and storage resources are reduced, and meanwhile protection of data privacy is ensured; the problem that the training precision and speed of an existing end side model are difficult to balance is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of terminal-side neural network fine-tuning, and in particular to an efficient terminal-side neural network fine-tuning method and system for a microcomputer. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of hardware technology, the computing resources owned by edge devices are gradually increasing, and the application demand for edge computing is growing. Edge devices generate a large amount of data with great utilization value in life and production, which provides rich resources for the training of intelligent control models. General artificial intelligence technology mainly uses remote computing resources for centralized training. However, for specific simple downstream tasks, such as image recognition, uploading data to the cloud will not only waste a lot of network bandwidth, but also face the privacy risk of data leakage. In order to solve this problem, end-side training technology came into being, aiming to train and update models on local devices, so as to make full use of local computing resources and protect data privacy and security.

[0004] Despite the continuous development of hardware technology, on-device training still faces challenges. Embedded devices have poor performance and very limited resources, which means that mainstream training frameworks cannot fully perform the training process on the device side. Most of them only complete the inference process, resulting in an inability to effectively balance the inference speed and inference performance of on-device image recognition.

[0005] At present, there are three main methods for end-side training: partial update, layer freeze or channel freeze, and quantization. Partial update allows the model to update only some parameters in each training, reducing the computational burden and memory usage, and is suitable for resource-constrained environments; however, this may lead to a decrease in the model's learning ability, and the unupdated parameters cannot adapt to the new data distribution, affecting the overall performance and resulting in poor robustness of image recognition; in addition, selecting update parameters requires additional strategies, which increases implementation complexity. In the freezing method, some layers or channels of the model remain unchanged during the training process, and only the parameters of other layers are updated; although this method reduces the amount of calculation, it limits the model's learning ability, resulting in the model's adaptability and flexibility being affected in a dynamically changing environment, and it cannot adapt to the processing of different types of images. Quantization technology reduces storage requirements and computational complexity by converting model parameters from high precision to low precision; this method can increase inference speed and reduce memory usage, but may cause model accuracy loss, affecting the accuracy of image recognition. Summary of the invention

[0006] In order to address the deficiencies in the prior art, the present invention provides an efficient fine-tuning method, system, electronic device, computer-readable storage medium and computer program product for an end-side neural network for a microcomputer, designs a smart online parameter selection method and an efficient gradient compression calculation method to ensure high accuracy while reducing the resources occupied by the model, thereby ensuring the accuracy and real-time performance of image recognition.

[0007] In a first aspect, the present invention provides an efficient fine-tuning method for a terminal-side neural network for a microcomputer;

[0008] An efficient fine-tuning method for a microcomputer-based end-side neural network, comprising:

[0009] Get the image;

[0010] Processing the image through a trained neural network to generate an image recognition result;

[0011] Among them, when training the neural network, based on the pre-trained model deployed on the terminal device, the stability of parameter iteration is used to determine the relative change of the parameters in the training cycle, and the momentum factor is combined to generate the importance of the parameters and dynamically select the update parameters; at the same time, the back propagation process is optimized through the gradient compression method.

[0012] In some embodiments, the use of the stability of parameter iteration to determine the relative change of the parameter during the training cycle and combining the momentum factor to generate the importance of the parameter includes:

[0013] Using the output feature graphs generated by successive iterations in the training cycle, the similarity of parameters between iterations is calculated; based on the similarity of parameters between consecutive training cycles, the relative change information of the parameters is determined;

[0014] The relative change information of the parameter is combined with the momentum factor and historical change information to generate the importance of the parameter.

[0015] In some embodiments, the importance of the parameters is expressed as:

[0016]

[0017] In the formula, represents the relative change information of the parameters, γ represents the momentum factor, Indicates historical change information.

[0018] In some implementations, dynamically selecting the update parameters according to the importance of the parameters specifically includes: comparing the importance of the parameters with a preset parameter freezing threshold, and screening the update parameters based on the comparison result.

[0019] In some embodiments, the optimization of the back propagation process by the gradient compression method is specifically: average summing and compressing the corresponding areas of the intermediate values ​​in the forward and reverse stages of training, and using the lightweight activation values ​​and intermediate gradients to update the parameter gradients and input gradients.

[0020] In some embodiments, when training the neural network, the change in activation value is determined during the forward propagation process by calculating the sum of corresponding areas during the convolution process.

[0021] In a second aspect, the present invention provides an efficient fine-tuning system for a terminal-side neural network for a microcomputer;

[0022] An efficient fine-tuning system for a microcomputer-oriented end-side neural network, comprising:

[0023] The acquisition module is configured to: acquire an image;

[0024] The image recognition module is configured to: process the image through a trained neural network to generate an image recognition result;

[0025] Among them, when training the neural network, based on the pre-trained model deployed on the terminal device, the stability of parameter iteration is used to determine the relative change of the parameters in the training cycle, and the momentum factor is combined to generate the importance of the parameters and dynamically select the update parameters; at the same time, the back propagation process is optimized through the gradient compression method.

[0026] In a third aspect, the present invention provides an electronic device;

[0027] An electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned end-side neural network efficient fine-tuning method for microcomputers.

[0028] In a fourth aspect, the present invention provides a computer-readable storage medium;

[0029] A computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for efficient fine-tuning of a terminal-side neural network for a microcomputer.

[0030] In a fifth aspect, the present invention provides a computer program product;

[0031] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned end-side neural network efficient fine-tuning method for a microcomputer.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. The technical solution provided by the present invention trains the neural network on the terminal device through an online selection strategy for alternating partial updates of sensitive parameters and an efficient back-propagation gradient compression calculation method, thereby ensuring high accuracy while reducing the resources occupied by the model. It provides a new design pattern and application paradigm for efficient fine-tuning on the terminal side of microcomputers, helps to achieve efficient model training on resource-constrained edge devices, while ensuring the protection of data privacy and ensuring a balance between accuracy and efficiency in image recognition tasks.

[0034] 2. The technical solution provided by the present invention, the online selection technology for alternating partial update of parameters can dynamically select the parameters that need to be updated according to the current training state, thereby reducing unnecessary computing overhead and improving training efficiency.

[0035] 3. The technical solution provided by the present invention is an efficient gradient compression calculation method, which reduces the demand for computing resources and storage resources by optimizing the gradient back-propagation calculation process, thereby improving image recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0037] Figure 1 A schematic diagram of a flow chart of an online parameter selection strategy for alternating partial updates provided by an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of a flow chart of a gradient compression calculation method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0040] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0042] Embodiment 1

[0043] Due to the limited resources of the terminal devices, it is difficult to balance the training accuracy and efficiency of the existing neural network deployed on the terminal devices, affecting the accuracy and real-time performance of image recognition. Therefore, the present invention provides an efficient fine-tuning method for terminal neural networks for microcomputers. Based on the implementation concept of algorithm-system collaborative optimization, a smart online parameter selection method and an efficient gradient compression calculation method are designed on terminal devices with extremely limited resources to ensure high accuracy of image recognition while reducing the resources occupied by the model.

[0044] Next, combine Figure 1-Figure 2 , a method for efficiently fine-tuning a neural network on a microcomputer disclosed in this embodiment is described in detail. The method for efficiently fine-tuning a neural network on a microcomputer comprises the following steps:

[0045] S1. Acquire an image.

[0046] S2. Process the image through the trained neural network to generate image recognition results.

[0047] Furthermore, the neural network is deployed on a microcomputer, which takes the image as input, processes the image through the trained neural network, and outputs the image recognition result, that is, the image of the recognition target is selected.

[0048] The neural network can be a convolutional neural network (CNN), a recurrent neural network (RNN), and a neural network based on an attention mechanism (such as Transformer) and other network architectures. In this embodiment, the network architecture of the neural network is not improved and will not be described in detail.

[0049] Next, taking the convolutional neural network CNN as an example, the training process of the neural network is further explained.

[0050] As an implementation method, the specific process of training a neural network is as follows:

[0051] Step 1: Based on the pre-trained model, deploy it on a microcomputer; obtain image information collected by users in real scenes and label them, and build training sets and verification sets.

[0052] Here, the pre-trained model refers to the convolutional neural network that has been pre-trained using image datasets on other devices.

[0053] Step 2: Input the training set into the pre-trained model for iterative training until the termination condition is reached and the training is completed.

[0054] In order to further reduce the memory and computing resources occupied by end-side training, during the training process of the pre-trained model, the stability of parameter iteration is used to determine the relative change of the parameters in the training cycle, and the momentum factor is combined to generate the importance of the parameters and dynamically select the update parameters; specifically, the following are included:

[0055] (1) Using the output feature map generated by continuous iterations in the training cycle, at the beginning of each iteration, the mean and variance of each layer of data with respect to the parameters are counted, and its similarity with the previous iteration is calculated, which is expressed as:

[0056]

[0057] Among them, r represents the convolution layer, c represents the output channel, t represents the training cycle, and D ip represents a small amount of input data that is different from the training set and the validation set, represents the parameter of the second layer output channel c, μ represents the mean of the output feature map, σ represents the variance of the output feature map, represents the mean value of the output feature map y of the parameter θ of the output channel c of the rth layer in the tth training cycle, represents the mean value of the output feature map y of the parameter θ of the output channel c of the r-th layer in the t-1th training cycle, represents the variance of the output feature y of the parameter θ of the output channel c of the rth layer in the tth training cycle, represents the variance of the output feature map y of the parameter θ of the output channel c of the r-th layer in the t-1th training cycle, Represents the parameters of the tth training cycle stability.

[0058] Based on this, the different performances of the same parameters on different data at different training stages are used to measure their contribution and potential for improvement to the fine-tuning process.

[0059] (2) According to the similarity change of parameters between two consecutive training cycles, the relative change information of the parameters is determined, which is expressed as:

[0060]

[0061] In the formula, Indicates the parameters at the t-1th iteration stability.

[0062] (3) The relative change information of the parameters is combined with the momentum factor and the historical change information to generate the importance of the parameters at the t-th training cycle, expressed as:

[0063]

[0064] In the formula, Represents the parameters of the tth training cycle The relative change information of Y represents the momentum factor, which makes it possible to take into account historical information; Represents historical change information, i.e., the parameters of the t-1th training cycle relative change information.

[0065] (4) Compare the importance of the parameter with the preset parameter freezing threshold. If the importance of the parameter is greater than the preset parameter freezing threshold, it will be used as the updated parameter to continue training in the next iteration.

[0066] The higher the value, the higher the current iteration. Training is still required if the saturation value has not reached a stable state. On the contrary, the lower the value is, the corresponding parameter has reached a stable state. Further training will not improve the benefits but will bring unnecessary calculations. Therefore, parameters below a certain threshold are frozen to increase the training speed.

[0067] Specifically, the freezing process is as follows:

[0068]

[0069] Among them, T a represents the parameter freezing threshold so that α% of the parameters have higher importance scores.

[0070] Here, the parameter freezing threshold is set to ensure that α% of the parameters can be selected. Once the value calculation is complete, the parameter freezing threshold can be determined.

[0071] (5) After selecting the parameters that need to be trained, the gradient compression method is applied to update the forward propagation process and the subsequent propagation process.

[0072] After selecting the parameters to be trained, the gradient compression technique is applied to change the original activation value to the sum of the corresponding area in the convolution process during the forward propagation process, and correspondingly to the original activation value in the back propagation process. The change is to average the corresponding area during the convolution process. The specific process is as follows:

[0073] (501) The input activation value x of the forward propagation pod [c,s H ·h+i,s W w+j] is summed according to the corresponding area of ​​the convolution to obtain the compressed input activation value

[0074] For example, in the forward propagation, the original activation value is compressed by summing the corresponding area of ​​the convolution as follows:

[0075]

[0076] Among them, c represents the number of channels of the activation value, h and w represent the size of the activation value, and s H and W represents the step length, x pad Represents the input activation value, that is, the padded feature map.

[0077] (502) According to the intermediate gradient obtained during the back propagation process Taking the average value of the corresponding area in its convolution And fill and sum to get

[0078] Specifically, There are two calculation processes involved in back propagation, namely, calculating parameter gradients and calculating input gradients. Different compression methods are used in these two calculation processes. Perform average compression, It is expressed as:

[0079]

[0080] When calculating the input gradient, Perform sum compression, It is expressed as:

[0081]

[0082] Where H' and W' represent the output size calculated based on the step size and padding, K H With K W Represents the size of the convolution kernel.

[0083] (503) The corresponding layer parameters in the back propagation process are averaged to obtain lightweight parameters

[0084] Specifically, when calculating the gradient of the parameters, θ needs to be compressed Make the corresponding operation and average the corresponding area in the normal convolution process, expressed as:

[0085]

[0086] (504) In the process of back propagation, using Update the parameter gradient using and Update the input gradients.

[0087] Exemplarily, the parameter gradient is expressed as:

[0088]

[0089] The input gradient is expressed as:

[0090]

[0091] Step 3: Input the verification set into the trained neural network to verify the effect.

[0092] In order to verify the effect of the efficient fine-tuning method of the end-side neural network for microcomputers described in this embodiment, it was verified on a resource-constrained embedded device (OpenMV Cam H7 development board equipped with an ARM Cortex-M7 processor, memory <1MB, external memory <2MB), and successfully implemented the training of various convolutional neural network models such as MCUNet, MobileNetV2, and ResNet. Compared with the prior art, this solution improves the model accuracy by 30.3%, achieves a 2.4-fold acceleration in training latency, and reduces the memory usage during training by 80%, significantly improving the efficiency of deep learning training in resource-constrained environments.

[0093] In summary, the efficient fine-tuning method of the end-side neural network for microcomputers described in this embodiment realizes the organic coupling of alternating partial updates and gradient compression technology based on the concept of algorithm-system co-design; in the application of end-side microcomputers with extremely limited resources, efficient fine-tuning of the neural network is realized, so that the sensitive data of the end-side device can be fully utilized and the environmental changes can be responded to in a timely manner so that the neural network model deployed on the end-side has the learning ability to adapt to environmental changes. In addition, this embodiment focuses on the model accuracy and the convergence speed of the end-side training. Compared with other methods, it can not only maintain the ability of fast convergence, but also ensure a certain accuracy of the model in specific tasks.

[0094] High accuracy of local device image recognition can be achieved without relying on complex end-cloud communications. After the corresponding CNN model is deployed to the end-side device, the device can specify updates to the model according to instructions, avoiding additional bandwidth waste. Thanks to the advantages of efficient computing and memory saving in the back-propagation process of this technology, the computing potential of the end-side device can be maximized without sacrificing latency, and even devices with extremely limited resources can give full play to their performance.

[0095] Embodiment 2

[0096] This embodiment discloses a microcomputer-oriented end-side neural network efficient fine-tuning system, including:

[0097] The acquisition module is configured to: acquire an image;

[0098] The image recognition module is configured to: process the image through a trained neural network to generate an image recognition result;

[0099] Among them, when training the neural network, based on the pre-trained model deployed on the terminal device, the stability of parameter iteration is used to determine the relative change of the parameters in the training cycle, and the momentum factor is combined to generate the importance of the parameters and dynamically select the update parameters; at the same time, the back propagation process is optimized through the gradient compression method.

[0100] It should be noted that the acquisition module and the image recognition module correspond to the steps in Embodiment 1, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0101] Embodiment 3

[0102] Embodiment 3 of the present invention provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the above-mentioned efficient fine-tuning method of the end-side neural network for microcomputers are completed.

[0103] Embodiment 4

[0104] Embodiment 4 of the present invention provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned method for efficient fine-tuning of a terminal-side neural network for a microcomputer are completed.

[0105] Embodiment 5

[0106] Embodiment 5 of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned method for efficient fine-tuning of a terminal-side neural network for a microcomputer.

[0107] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0110] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0111] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An efficient fine-tuning method for a microcomputer-oriented end-side neural network, characterized in that: include: Get the image; Processing the image through a trained neural network to generate an image recognition result; Among them, when training the neural network, based on the pre-trained model deployed on the terminal device, the stability of parameter iteration is used to determine the relative change of the parameters in the training cycle, and the momentum factor is combined to generate the importance of the parameters and dynamically select the update parameters; at the same time, the back propagation process is optimized through the gradient compression method.

2. The method for efficiently fine-tuning a neural network on a microcomputer side according to claim 1, characterized in that: The stability of parameter iteration is used to determine the relative change of parameters in the training cycle. Combined with the momentum factor, the importance of generating parameters includes: Using the output feature graphs generated by successive iterations in the training cycle, the similarity of parameters between iterations is calculated; based on the similarity of parameters between consecutive training cycles, the relative change information of the parameters is determined; The relative change information of the parameter is combined with the momentum factor and historical change information to generate the importance of the parameter.

3. The method for efficiently fine-tuning a neural network on a microcomputer side according to claim 2, characterized in that: The importance of the parameters is expressed as: In the formula, represents the relative change information of the parameters, γ represents the momentum factor, Indicates historical change information.

4. The method for efficiently fine-tuning a neural network on a microcomputer side according to claim 1, characterized in that: Dynamically selecting update parameters according to the importance of the parameters specifically includes: comparing the importance of the parameters with a preset parameter freezing threshold, and screening the update parameters based on the comparison result.

5. The method for efficiently fine-tuning a neural network on a microcomputer side according to claim 1, characterized in that: The optimization of the back propagation process by the gradient compression method is specifically as follows: updating the parameter gradient and the input gradient by averaging and summing the corresponding areas in the convolution process.

6. The method for efficient fine-tuning of a microcomputer-oriented end-side neural network according to claim 1, characterized in that: When training the neural network, the activation value change is determined during the forward propagation process by calculating the sum of the corresponding areas during the convolution process.

7. An efficient fine-tuning system for end-side neural networks for microcomputers, characterized in that: include: The acquisition module is configured to: acquire an image; The image recognition module is configured to: process the image through a trained neural network to generate an image recognition result; Among them, when training the neural network, based on the pre-trained model deployed on the terminal device, the stability of parameter iteration is used to determine the relative change of the parameters in the training cycle, and the momentum factor is combined to generate the importance of the parameters and dynamically select the update parameters; at the same time, the back propagation process is optimized through the gradient compression method.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the efficient fine-tuning method for a microcomputer-oriented end-side neural network as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the efficient fine-tuning method for a microcomputer-oriented end-side neural network as described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the efficient fine-tuning method for a microcomputer-oriented end-side neural network as described in any one of claims 1 to 6 are implemented.

Images