Retraining method and system for replacing activation function in computer vision neural network

Abstract

The present invention provides a retraining method and system for replacing activation functions in a computer vision neural network, relating to the technical field of computer vision neural networks. The method comprises: a single-stage replacement training method paradigm or a two-stage replacement training method paradigm; the single-stage replacement training method paradigm or the two-stage replacement training method paradigm includes: starting a training program after specifying training scheme parameters, and then providing a model weight file after replacing the activation function. The present invention has good applicability and stable performance, and can quickly and stably retrain the accuracy to a level close to that of the original model.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision neural networks, and in particular to a retraining method and system for replacing activation functions in a computer vision neural network. Background Art

[0002] Computer Vision Neural Networks (CVNs) are a representative technology in artificial intelligence. They receive input visual data such as 2D images, 3D point cloud frames, 2D video frame streams, 3D point cloud frame streams, and binocular camera input streams. Through a series of linear and nonlinear operations, they output prediction results for user-specified tasks. These tasks include, but are not limited to, image classification, image object detection, image semantic segmentation, 3D object detection, 3D semantic segmentation, monocular motion detection, monocular depth prediction, and image style transfer.

[0003] Typically, computer vision neural networks are constructed by stacking a series of feature extraction modules and adding a prediction head, or by using an "encoder-decoder" approach. Regardless of the method, the core component is always the convolution module. The input of the convolution module is the "input feature map," and the output is the "output feature map." After the input feature map passes through the convolution module, the image features are extracted by the convolution kernel and deeply abstracted to the output feature map. After multiple such convolution modules are stacked to extract features and fed into the prediction head, the prediction head can make predictions based on the abstract semantic information.

[0004] The convolutional module described above typically consists of three parts: ① a linear operation; ② a feature map normalization; and ③ a nonlinear operation. The scope of this discussion is also limited to convolutional modules constructed in this paradigm. Each part of this three-part convolutional module may be composed of the following specific "operators, layers, and nodes."

[0005] 1. Linear operation part:

[0006] a) Matrix multiplication operation (Matrix Multiply);

[0007] b) Full Connect operation;

[0008] c) Convolution operation (Convolution), including various variants such as ordinary convolution, void convolution, and transposed convolution;

[0009] 2. Feature map normalization part:

[0010] a) Batch Normalization;

[0011] b) Layer Normalization;

[0012] c) Instance Normalization;

[0013] d) Group Normalization;

[0014] 3. Nonlinear operation part:

[0015] a) Rectified Linear Unit (ReLU);

[0016] b) Sigmoid activation function;

[0017] c) Swish activation function;

[0018] d) Other common activation functions, etc.

[0019] A typical real-world example is ResNet-50, where the composition of each convolutional module can be enumerated into the following two combinations:

[0020] 1. Convolution operation, batch normalization, linear rectification function (denoted as Conv-BN-ReLU);

[0021] 2. Fully connected operation, batch normalization, linear rectification function (denoted as FC-BN-ReLU).

[0022] The entire ResNet is composed of stacked and connected modules like these. By slightly varying the convolution parameters within each module, the network can extract high-level, abstract semantics from an image and perform category prediction. Other computer vision neural networks operate similarly, combining different linear, normalization, and nonlinear operations, modifying the convolution kernel parameters, and employing different stacking and connection methods to create a variety of network structures.

[0023] Computer vision neural networks require "training" to be capable of completing assigned tasks. Generally speaking, the most common method for training computer vision neural networks is "supervised training," whereby a neural network receives an input-label pair and, after internal calculations, generates a "predicted output" from the network. This "predicted output" is then compared with the label, and an "error" is calculated using a specific method. Finally, the error is propagated back through "backpropagation" to update each parameter or weight in the network.

[0024] The computational cost of training a neural network varies depending on the complexity of the task, the complexity of the network, and the complexity of the dataset itself, but in general, the training process described above needs to be repeated many times before the weights reach a "convergence" state. After the final training is completed, we can call the "weights" of the neural network "trained weights", or in other words, the weights have "converged". We can directly load such weights intact into the corresponding neural network structure and train other tasks or datasets. At this time, the loaded weights can be called "pre-trained weights", which specifically refers to the weights that have been trained to a convergence state under a certain dataset.

[0025] The process of training the network is necessary at least in the following cases:

[0026] 1. Simplify, expand, modify, or perform other modifications to a certain part of the original model;

[0027] 2. Do not change any structure of the original model, but change the task domain of the trained original model, for example, switching from a "general image classification task" to a "subcategory classification task of a certain large category of objects, such as classifying dog breeds."

[0028] The "activation function replacement" involved in this invention satisfies the above-mentioned condition 1, that is, the structure of the neural network itself is modified. The purposes for modifying the model structure itself are varied, such as replacing the prediction head to complete different tasks, or streamlining some network layers to improve the model inference speed. However, in general, after modifying the structure of the model itself, the original weights of the neural network are no longer applicable and need to be trained again to reach a convergence state. The following text will discuss two traditional methods for training a neural network with a replaced activation function to convergence.

[0029] Technique 1: Train From Scratch:

[0030] Training from scratch is the most direct training method. Given a neural network, its weights are randomly initialized using a method with certain characteristics. Then, given a dataset, a training strategy (including optimizer selection, learning rate decay, weight decay, and several regularization techniques) is employed to achieve convergence of the neural network weights. Trainable parameters typically fall into three categories: ① Linear operation parameters, namely convolution kernel parameters, fully connected layer parameters, and weight matrix parameters in matrix multiplication; ② Normalization layer parameters, such as the mean and variance parameters in batch normalization; and ③ Other linear calculation parameters, such as the multiplication coefficient parameters used in regression tasks.

[0031] A pseudocode representation of the process is given below.

[0032]

[0033]

[0034] Through the above method, under the premise that the replacement activation function and the training scheme are reasonable, the network weights can be trained to a convergent state.

[0035] Technique 2: Loading pre-trained weights for full network training:

[0036] Loading pre-trained weights and then training the entire network is a more practical training paradigm. Generally speaking, application personnel do not necessarily build a completely new computer vision neural network. Instead, they use the "backbone structure" of an existing computer vision neural network with the prediction head required for their own tasks, or they partially modify the backbone structure of the original network to achieve a better effect between "accuracy" and "performance". For the computer vision neural network constructed in this way, training from scratch is obviously not a wise choice. The more widely used training scheme is: for the unmodified part, the previously trained pre-trained weights are loaded, and the modified part is initialized with some random initialization method. Then, the weights of the unmodified part can be optionally fixed or not fixed, and the network is trained to eventually reach a convergence state.

[0037] A pseudocode representation of the process is given below.

[0038]

[0039] Through the above method, under the premise that the replacement activation function and the training scheme are reasonable, the network weights can be trained to a convergence state faster and easier than training from scratch.

[0040] The above two technologies have the following disadvantages:

[0041] 1. Training from scratch consumes too much computing resources. For several popular networks at this stage, if you want to train them completely from scratch, you will need to consume a lot of computing resources to train the network to a convergence state. In addition, since the training effect is seriously coupled with the structure of the model itself and the related hyperparameter schemes selected during training (optimizer, learning rate strategy, regularization strategy, input data pre-processing strategy, output prediction post-processing strategy, etc.), it is often necessary to go through multiple complete trainings to obtain the weights that achieve the expected effect. For application personnel, if they want to simply replace the activation function to obtain accuracy or performance improvement, it is obviously too costly;

[0042] 2. The effect of training with pre-trained weights is still strongly coupled to the training method and the model itself, and it is possible that the weights cannot be trained to converge. Although training a network with a replaced activation function using pre-trained weights is easier and less time-consuming, the training method, training results, and the model structure are still strongly coupled. For example, two different networks (such as ResNet-50 and VGG-16) can achieve completely different results when performing the same task and using the same activation function replacement strategy and training method. This means that each time an application developer wants to replace the activation function of a network, they need to design a separate training method for that network. In addition, in some cases, if training with pre-trained weights is performed and all activation functions of the network are replaced, it may not be possible to train to convergence. For example, in EfficientDet-3, after replacing all Swish activation functions with ReLU and training using the pre-trained weights as the starting point, several conventional training methods failed to train the replaced model back to the accuracy within -5mAP of the original model structure.

[0043] The invention patent with publication number CN107644252A discloses a multi-mechanism hybrid recursive neural network model compression method, which includes: A. Circular matrix constraint: restricting some parameter matrices in the recursive neural network to circulant matrices, and updating the backward gradient propagation algorithm so that the network can perform batch training of circulant matrices; B. Forward activation function approximation: replacing the nonlinear activation function with a hardware-friendly linear function during forward operation, and the backward gradient update process remains unchanged; C. Hybrid quantization: adopting different quantization mechanisms for different parameters based on the difference in error tolerance of different parameters in the recursive neural network; D. Secondary training mechanism: dividing the training of the network model into two stages: initial training and retraining. Each stage focuses on different model compression methods, which effectively avoids the mutual influence between different model compression methods and minimizes the accuracy loss caused by the model compression method. Summary of the Invention

[0044] In response to the deficiencies in the prior art, the present invention provides a retraining method and system for replacing activation functions in a computer vision neural network.

[0045] According to a retraining method and system for replacing activation functions in a computer vision neural network provided by the present invention, the scheme is as follows:

[0046] In a first aspect, a retraining method for replacing an activation function in a computer vision neural network is provided, the method comprising:

[0047] a single-stage replacement training method paradigm or a two-stage replacement training method paradigm;

[0048] The single-stage replacement training method paradigm or the two-stage replacement training method paradigm includes: starting the training program after specifying the training scheme parameters, and then providing the model weight file after replacing the activation function.

[0049] Preferably, the single-stage replacement training method paradigm is as follows: after specifying the training scheme parameters, the training program is started, and after the program ends, a weight file with the activation function replaced and the accuracy close to the original model is directly given.

[0050] Preferably, the single-stage replacement training method paradigm specifically includes:

[0051] Step S1.1: Determine the optimizer. The choice of the optimizer can be directly based on the optimizer used in the original model, or a general SGD optimizer or AdamW optimizer.

[0052] Step S1.2: Calibrate the optimizer hyperparameters. Before replacing the activation function, use the original model as the trained model and explore a stable learning rate optimizer hyperparameter so that the model can maintain accuracy across training iterations.

[0053] Step S1.3: Separate the normalization layer trainable parameters from the rest of the model trainable parameters;

[0054] Step S1.4: Set two different learning rate scheduling strategies for the two types of weights;

[0055] Step S1.5: Start training until the training reaches the predetermined loss value\accuracy\epoch.

[0056] Preferably, the specific method of step S1.2 includes:

[0057] Step S1.2.1: After selecting the optimizer, train with zero learning rate and zero weight decay, and record the average loss value after the first few iterations;

[0058] Step S1.2.2: Follow the same data input order as in step S1.2.1, perform random data preprocessing, set a non-zero learning rate, weight decay parameter, and other optimizer-related hyperparameters, and perform training. Record the mean loss value after the same number of iterations.

[0059] Step S1.2.3: If the difference between the mean loss value of step S1.2.2 and the mean loss value of step S1.2.1 reaches a certain threshold, then terminate this step and calibrate the current optimizer hyperparameters to the optimizer hyperparameter set A;

[0060] If the threshold is not reached, then choose whether to adjust based on the current optimizer hyperparameters themselves, and then repeat step S1.2.2 until the learning rate in the optimizer hyperparameters has reached a larger value or the loss value in step S1.2.2 exceeds the threshold.

[0061] Preferably, step S1.3 includes: by writing a model weight filter, placing a reference to the trainable parameters of the normalization layer in the model into a certain data structure, and placing other trainable parameters into another data structure of the same type, so that the optimizer updates these two parts of the weights with two update strategies.

[0062] Preferably, in step S1.4, two different learning rate scheduling strategies are set for the two types of weights, and the setting principles include:

[0063] Step S1.4.1: During most of the training period, the learning rate of the normalization layer's trainable parameters must be greater than the learning rates of other trainable parameters, and this learning rate must be maintained throughout the training. The maximum learning rate should be several times the learning rate of the optimizer's parameter set A.

[0064] Step S1.4.2: At the end of the training period, the learning rate of the normalization layer trainable parameters begins to decrease until it is aligned with the learning rates of other trainable parameters;

[0065] Step S1.4.3: The learning rates of other trainable parameters are set to 0 in the initial to mid-stages of training, and then gradually adjusted to the learning rates in the optimizer parameter set A at the end. The learning rate can also be kept at zero until the end of training.

[0066] Preferably, the two-stage replacement training method paradigm is as follows: the entire training process is completed in two steps, firstly, after specifying the first training program parameters, the training program is started, and after the program is completed, the model weight file after the initial replacement of the activation function is provided;

[0067] Then the second stage of training begins, loading the weight file obtained in the previous step, specifying new training scheme parameters, starting the training program, and giving the final model weight file after replacing the activation function after the program ends.

[0068] Preferably, the two-stage replacement training method paradigm specifically includes:

[0069] Step S2.1: Determine the optimizer. The choice of the optimizer can be directly based on the optimizer used in the original model, or a general SGD optimizer or AdamW optimizer;

[0070] Step S2.2: Calibrate the optimizer hyperparameters. Before replacing the activation function, use the original model as the trained model and explore a stable learning rate optimizer hyperparameter so that the model can maintain accuracy across training iterations.

[0071] Step S2.3: Separate the normalization layer trainable parameters from the rest of the model trainable parameters;

[0072] Step S2.4: Set two different learning rate scheduling strategies for the two types of weights;

[0073] Step S2.5: Start the first phase of training until the training reaches the predetermined loss value\accuracy\epoch, and save the weights of the first phase;

[0074] Step S2.6: Reinitialize the model and optimizer, load the weights obtained in step S2.5, set the optimizer parameter set A obtained in step S2.2 for the optimizer, and apply it to all trainable parameters;

[0075] Step S2.7: Start training until the training reaches the predetermined loss value\accuracy\epoch.

[0076] Preferably, in step S2.4, two different learning rate scheduling strategies are set for the two types of weights, and the setting principles include:

[0077] Step S2.4.1: For the learning rate of the trainable parameters of the normalization layer, train at a multiple of the learning rate in the optimizer parameter set A. Perform a warm-up at the beginning of training, and then maintain the maximum learning rate for training. Alternatively, adjust the learning rate by step-by-step learning rate. In principle, maintain the maximum learning rate for more than 70% of the total training steps in the first stage of training.

[0078] Step S2.4.2: Set the learning rates of other trainable parameters to zero.

[0079] In a second aspect, a retraining system for replacing activation functions in a computer vision neural network is provided, the system comprising:

[0080] a single-stage replacement training method paradigm or a two-stage replacement training method paradigm;

[0081] The single-stage replacement training method paradigm or the two-stage replacement training method paradigm includes: starting the training program after specifying the training scheme parameters, and then providing the model weight file after replacing the activation function.

[0082] Compared with the prior art, the present invention has the following beneficial effects:

[0083] 1. By proposing the core concept of separating the training parameters of the normalization layer from those of other layers, the exploration and practice of replacing network activation functions becomes a low-cost activity, which helps researchers and application personnel to conduct model design exploration and model deployment testing more quickly;

[0084] 2. By proposing a dedicated activation function replacement and retraining paradigm, researchers and deployers have clear training strategy guidance when conducting such retraining. Large-scale parameter adjustment experiments are no longer necessary. According to the solution paradigm provided by this invention, only simple parameter adjustments are required within a small range, which greatly reduces the uncertainty and training difficulties caused by the large space of optional hyperparameters when (re)training the network under traditional methods.

[0085] 3. More specifically, through the training scheme paradigm proposed in the present invention, when some neural networks with more complex activation functions are deployed to edge devices, or in order to achieve extreme performance during deployment, the activation function replacement training strategy proposed in the present invention can stably replace the original activation function with the ReLU activation function that has the least hardware burden at a lower cost in time and hardware resources, as well as a slight loss in accuracy, thereby improving the network's operating speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0087] Figure 1 Response graph of ReLU function and (standard) Swish function;

[0088] Figure 2 Schematic diagram summarizing the process of the method. DETAILED DESCRIPTION

[0089] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0090] Embodiments of the present invention provide a retraining method for replacing activation functions in a computer vision neural network. In computer vision neural networks, semantic extraction and abstraction of input data are often accomplished through convolutional kernels. Before the rise of deep neural networks, neural networks such as AlexNet did not include a "feature map normalization" layer in their architecture. The most popular "feature map normalization layer," or Batch Normalization (BN), was only proposed in 2015. Since then, neural network construction has adopted "feature map normalization" as a standard construction paradigm. This demonstrates that a neural network's ability to extract semantic meaning from input data is maximized by the convolutional kernels within the network. In other words, feature normalization layers, whether BN, GN, LN, or InstanceNormalization, have no substantial impact on feature extraction, or their impact is far less than that of the convolutional kernels themselves. The transformations performed by various normalization layers on feature maps essentially adjust the data distribution within the feature maps, ensuring that different data have a more regular and uniform numerical distribution after passing through the normalization layer.

[0091] Therefore, when the activation function of the network is replaced, the output of the activation function before and after the feature map can be regarded as two outputs with different numerical distributions, especially when the replaced activation function pair has a similar response curve. Figure 1 As shown in the figure, the response curve of the ReLU activation function and the response curve of the standard Swish activation function (y = x * sigmoid (x)) are given. We can intuitively feel that after the feature map passes through the two activation functions, the biggest difference between the two will be the change in the numerical distribution.

[0092] After the feature map passes through the Swish function, it will have a lower mean than after passing through the ReLU function, and the variance will change, but both achieve great suppression of the negative part and no suppression or very little suppression of the positive part. The idea of ​​the present invention is to "first train only the normalization layer, and then train the entire model". After being replaced by a similar activation function, the biggest change in the feature map is the change in its numerical distribution, and the numerical distribution is affected by the previous normalization layer, or in other words, the previous normalization layer determines the numerical distribution of the feature map before the activation function. Therefore, in essence, only the normalization layer needs to be retrained so that its parameters are adjusted to compensate for the change in numerical distribution before and after the activation function is replaced, so that after the activation function is replaced, the numerical distribution of the output feature map still does not change much, so that the features can be effectively extracted by the convolution kernel of the subsequent layer.

[0093] A retraining method for replacing activation functions in computer vision neural networks is designed to solve the problem that existing training schemes that load pre-trained weights cannot have a stable training scheme paradigm for activation function replacement and cannot guarantee the retraining effect. The present invention has good applicability and stable performance, and can be widely used in retraining after replacing activation functions when neural networks are deployed to various devices by application personnel. Figure 2 As shown in the figure, the core of this method is to "separately train the trainable weights of the convolution kernel and the trainable weights of the normalization layer", which specifically includes a single-stage replacement training method paradigm and a two-stage replacement training method paradigm.

[0094] Among them, the single-stage replacement training method paradigm: the entire training process is completed in a single time, the training program is started after the training scheme parameters are specified, and after the program ends, a weight file with the accuracy close to the original model after replacing the activation function is directly given.

[0095] The single-stage replacement training method paradigm specifically includes:

[0096] Step S1.1: Determine the optimizer. The optimizer can be selected directly based on the optimizer used in the original model, or a general SGD optimizer or AdamW optimizer.

[0097] Step S1.2: Calibrate the optimizer (training) hyperparameters. Before replacing the activation function of the model, use the original model as the trained model and explore a stable learning rate optimizer (training) hyperparameter so that the model can maintain accuracy during training iterations. The specific methods are as follows:

[0098] Step S1.2.1: After selecting the optimizer, train with zero learning rate and zero weight decay (if any), and record the average loss value after the first several iterations (for example, one thousand, depending on the size of the dataset itself).

[0099] Step S1.2.2: Follow the same data input order as step S1.2.1, randomly preprocess the data, set a non-zero but small learning rate and weight decay parameter and other optimizer-related hyperparameters (if any), perform training, and record the average loss value after the same number of iterations.

[0100] Step S1.2.3: If the difference between the mean loss value of step S1.2.2 and the mean loss value of step S1.2.1 reaches a certain threshold (for example, but not limited to 1%), then terminate this step and calibrate the current optimizer hyperparameters as "optimizer hyperparameter set A".

[0101] If the threshold is not reached, then choose whether to adjust it based on the current optimizer hyperparameters. For example, you can choose to continue increasing the learning rate, and then repeat step S1.2.2 until the learning rate in the optimizer hyperparameters has reached a larger value or the loss value in step S1.2.2 exceeds the threshold.

[0102] Step S1.3: Separate the normalization layer trainable parameters from the rest of the model's trainable parameters. By writing a simple model weight filter, references to the normalization layer's trainable parameters are placed into a data structure (e.g., but not limited to, a Python filter), while the rest of the trainable parameters are placed into a separate data structure of the same type. This allows the optimizer to update these two sets of weights using two different update strategies.

[0103] Step S1.4: Set two different learning rate scheduling strategies (learning ratescheduler) for the two types of weights. The setting principles are as follows:

[0104] Step S1.4.1: During most of the training period, the learning rate of the normalization layer's trainable parameters must be greater than the learning rates of other trainable parameters, and this learning rate must be maintained during training. The maximum learning rate should be several times the learning rate in "Optimizer Parameter Set A" (for example, but not limited to, 10 times, 100 times, 1000 times, etc.).

[0105] Step S1.4.2: At the end of the training period, the learning rate of the normalization layer trainable parameters begins to decrease until it is aligned with the learning rates of other trainable parameters;

[0106] Step S1.4.3: The learning rates of other trainable parameters can be set to 0 in the initial to mid-stages of training, and then gradually adjusted to the learning rates in the "Optimizer Parameter Set A" until the end of the training. Alternatively, the learning rate can be kept at zero until the end of training.

[0107] Step S1.5: Start training until the training reaches the predetermined loss value\accuracy\epoch.

[0108] Here is a specific and feasible design solution.

[0109] For EfficientDet-3, replace all Swishes with ReLU, select the SGD optimizer, use the Warm-Up scheduling strategy for the learning rate of the normalization layer trainable parameters, set the maximum learning rate to 1e-4, and set the warm-up (Warm) to 1 epoch; set the learning rate of other trainable parameters to zero, and go through a total of 7 epochs. The accuracy of the replaced model can be trained to within 3mAP of the original accuracy.

[0110] Secondly, the two-stage replacement training method paradigm: the entire training process is completed in two steps. First, after specifying the first training scheme parameters, the training program is started. After the program ends, the model weight file after the initial replacement of the activation function is given; then the second stage of training begins, the weight file obtained in the previous step is loaded, the new training scheme parameters are specified, the training program is started, and after the program ends, the final model weight file after the activation function is replaced is given.

[0111] The two-stage replacement training paradigm has exactly the same steps as the single-stage replacement training paradigm in the first three steps. The only difference is from the fourth step onwards, including:

[0112] Step S2.1: Determine the optimizer. The choice of the optimizer can be directly based on the optimizer used in the original model, or a general SGD optimizer or AdamW optimizer;

[0113] Step S2.2: Calibrate the optimizer hyperparameters. Before replacing the activation function, use the original model as the trained model and explore a stable learning rate optimizer hyperparameter so that the model can maintain accuracy across training iterations.

[0114] Step S2.3: Separate the normalization layer trainable parameters from the rest of the model trainable parameters;

[0115] Step S2.4: Set two different learning rate scheduling strategies (learning ratescheduler) for the two types of weights; the setting principles include:

[0116] Step S2.4.1: For the learning rate of the trainable parameters of the normalization layer, train at a multiple of the learning rate in the optimizer parameter set A. Perform a warm-up at the beginning of training, and then maintain the maximum learning rate for training. Alternatively, adjust the learning rate by step-by-step learning rate. In principle, maintain the maximum learning rate for more than 70% of the total training steps in the first stage of training.

[0117] Step S2.4.2: Set the learning rates of other trainable parameters to zero.

[0118] Step S2.5: Start the first phase of training until the training reaches the predetermined loss value\accuracy\epoch, and save the weights of the first phase;

[0119] Step S2.6: Reinitialize the model and optimizer, load the weights obtained in step S2.5, set the optimizer's "optimizer parameter set A" obtained in step S2.2, and apply it to all trainable parameters;

[0120] Step S2.7: Start training until the training reaches the predetermined loss value\accuracy\epoch.

[0121] Here is a specific and feasible design solution.

[0122] For EfficientDet-3, all Swishes were replaced with ReLUs, the SGD optimizer was selected, and the learning rates of the trainable parameters of the normalization layer were scheduled using a warm-up strategy, with a maximum learning rate of 1e-4 and a warmup period of one epoch. All other trainable parameters were set to zero learning rate, and a total of seven epochs were run. The weights from the first phase were then saved. In the second phase, the previous weights were loaded, and the learning rate for all trainable parameters was set to 1e-5. A warmup period of one epoch was set. After five epochs, the replaced model achieved an accuracy within 2 mAP of the original model.

[0123] The method is applicable to:

[0124] 1. The applicable subject is "computer vision neural network model";

[0125] 2. Modifications to the subject matter are limited to "replacing some or all activation functions in the model. Activation function replacement can be unidirectional or combined." This means that replacing the activation function of a convolutional module anywhere in the model with any other activation function is within the scope of the claimed invention. For example, the following network structure: "Conv-BN-ReLU①-Convㄱ-BN–ReLU②" falls within the scope of the invention under the following replacement methods.

[0126] a) Replace all ① and ② with another activation function, for example, both are replaced with Swish to pursue higher accuracy;

[0127] b) ①② are all replaced with multiple different activation functions, for example, ① is replaced by Swish, and ② is replaced by Leaky ReLU;

[0128] c) Part ① and ② are replaced with another activation function, for example, only ② is replaced with Swish;

[0129] d) Some activation functions in the model are replaced with multiple different activation functions.

[0130] In summary, the embodiments of the present invention provide a retraining method and system for replacing activation functions in a computer vision neural network. The advantage of the two-stage training method paradigm is that, because the training is completed in two steps, the two stages can customize more diverse training strategies. Furthermore, after the first stage is completed, the training has a "save point," and the second stage of training can continuously trial and error on the "save point," ultimately achieving better results. The advantage of the single-stage replacement training method paradigm is that it consumes less time and computing resources.

[0131] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0132] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A retraining method for replacing activation functions in a computer vision neural network, characterized in that: The training method is used for at least one of image classification, image object detection, image semantic segmentation, 3D object detection, 3D semantic segmentation, monocular dynamic detection, monocular depth prediction, and image style transfer, including: a single-stage replacement training method paradigm or a two-stage replacement training method paradigm; The single-stage replacement training method paradigm or the two-stage replacement training method paradigm includes: starting a training program after specifying training scheme parameters, and then providing a model weight file after replacing the activation function; The single-stage replacement training method paradigm is as follows: after specifying the training scheme parameters, the training program is started, and after the program is completed, a weight file with the activation function replaced and the accuracy close to the original model is directly given; The single-stage replacement training method paradigm specifically includes: Step S1.1: Determine the optimizer. The choice of the optimizer can be directly based on the optimizer used in the original model, or a general SGD optimizer or AdamW optimizer; Step S1.2: Calibrate the optimizer hyperparameters. Before replacing the activation function, use the original model as the trained model and explore a stable learning rate optimizer hyperparameter so that the model can maintain accuracy across training iterations. Step S1.3: Separate the normalization layer trainable parameters from the rest of the model trainable parameters; Step S1.4: Set two different learning rate scheduling strategies for the two types of weights; Step S1.5: Start training until the training reaches the predetermined loss value\accuracy\epoch; The two-stage replacement training method paradigm: the entire training process is completed in two steps. First, the first training program parameters are specified and the training program is started. After the program is completed, the model weight file after the initial replacement of the activation function is generated; Then the second stage of training begins, loading the weight file obtained in the previous step, specifying new training program parameters, starting the training program, and giving the final model weight file after replacing the activation function after the program ends; The two-stage replacement training method paradigm specifically includes: Step S2.1: Determine the optimizer. The choice of the optimizer can be directly based on the optimizer used in the original model, or a general SGD optimizer or AdamW optimizer; Step S2.2: Calibrate the optimizer hyperparameters. Before replacing the activation function, use the original model as the trained model and explore a stable learning rate optimizer hyperparameter so that the model can maintain accuracy across training iterations. Step S2.3: Separate the normalization layer trainable parameters from the rest of the model trainable parameters; Step S2.4: Set two different learning rate scheduling strategies for the two types of weights; Step S2.5: Start the first phase of training until the training reaches the predetermined loss value\accuracy\epoch, and save the weights of the first phase; Step S2.6: Reinitialize the model and optimizer, load the weights obtained in step S2.5, set the optimizer parameter set A obtained in step S2.2 for the optimizer, and apply it to all trainable parameters; Step S2.7: Start training until the training reaches the predetermined loss value\accuracy\epoch.

2. The retraining method for replacing activation functions in a computer vision neural network according to claim 1, characterized in that The specific method of step S1.2 includes: Step S1.2.1: After selecting the optimizer, train with zero learning rate and zero weight decay, and record the average loss value after the first few iterations; Step S1.2.2: Follow the same data input order as in step S1.2.1, perform random data preprocessing, set a non-zero learning rate, weight decay parameter, and other optimizer-related hyperparameters, and perform training. Record the mean loss value after the same number of iterations. Step S1.2.3: If the difference between the mean loss value of step S1.2.2 and the mean loss value of step S1.2.1 reaches a certain threshold, then terminate this step and calibrate the current optimizer hyperparameters to the optimizer hyperparameter set A; If the threshold is not reached, then choose whether to adjust based on the current optimizer hyperparameters themselves, and then repeat step S1.2.2 until the learning rate in the optimizer hyperparameters has reached a larger value or the loss value in step S1.2.2 exceeds the threshold.

3. The retraining method for replacing activation functions in a computer vision neural network according to claim 1, characterized in that The step S1.3 includes: by writing a model weight filter, placing the reference of the trainable parameters of the normalization layer in the model into a certain data structure, and placing the other trainable parameters into another data structure of the same type, so that the optimizer updates these two parts of the weight using two update strategies.

4. The retraining method for replacing activation functions in a computer vision neural network according to claim 1, characterized in that In step S1.4, two different learning rate scheduling strategies are set for the two types of weights. The setting principles include: Step S1.4.1: During most of the training period, the learning rate of the normalization layer's trainable parameters must be greater than the learning rates of other trainable parameters, and this learning rate must be maintained throughout the training. The maximum learning rate should be several times the learning rate of the optimizer's parameter set A. Step S1.4.2: At the end of the training phase, the learning rate of the normalization layer trainable parameters begins to decrease until it matches the learning rate of other trainable parameters; Step S1.4.3: The learning rates of other trainable parameters are set to 0 in the initial to mid-stages of training, and then gradually adjusted to the learning rates in the optimizer parameter set A at the end. The learning rate can also be kept at zero until the end of training.

5. The retraining method for replacing activation functions in a computer vision neural network according to claim 1, characterized in that: In step S2.4, two different learning rate scheduling strategies are set for the two types of weights. The setting principles include: Step S2.4.1: For the training parameters of the normalization layer, the learning rate is trained at a multiple of the learning rate in the optimizer parameter set A. Warm-up is performed at the beginning of training, and then the maximum learning rate is maintained for training. Alternatively, the learning rate can be adjusted by step-by-step learning rate. In principle, the maximum learning rate is maintained for more than 70% of the total number of training steps in the first stage of training. Step S2.4.2: Set the learning rates of other trainable parameters to zero.

6. A retraining system for replacing activation functions in a computer vision neural network, characterized in that A retraining method for replacing an activation function in a computer vision neural network according to any one of claims 1 to 5, comprising: a single-stage replacement training method paradigm or a two-stage replacement training method paradigm; The single-stage replacement training method paradigm or the two-stage replacement training method paradigm includes: starting the training program after specifying the training scheme parameters, and then providing the model weight file after replacing the activation function.

Images