Parameter-free automatic adaptation method for pre-trained neural networks based on reparameterization

Abstract

The present invention provides a parameter-free automatic adaptive method for a pre-trained neural network based on re-parameterization. The method is based on a re-parameterization method, which enables the filter to achieve a soft balance between frozen pre-trained weights and fully fine-tuned weights, and can be equivalently converted into a single weight layer without introducing additional model parameters in the inference stage; analysis shows that the method can simulate fine-tuning each filter in the neural network layer with different learning rates. Therefore, the method realizes automatic adaptation of the pre-trained model without additional search costs.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and more specifically, to a parameter-free automatic self-adaptation method based on a re-parameterized pre-trained neural network. Background Art

[0002] The latest progress in vision pre-training has demonstrated the importance and advantages of migrating pre-trained models to target tasks. Different transfer learning methods are suitable for different networks and target tasks. Whether a certain layer should be fine-tuned or frozen, and the degree of fine-tuning require tedious repeated experiments and manual adjustments according to actual conditions. This paper proposes a parameter-free automatic model adaptation method for convolutional neural networks without additional search costs.

[0003] Transfer learning: Deep learning is highly data-driven. In order for the model to learn a specific task, in addition to collecting a large amount of task-related data, it is more difficult to require expensive labeling costs. The performance of the model depends on the amount of labeling of the training data. In order to reduce the cost of data collection and labeling and improve training efficiency, transfer learning first pre-trains the model on a large-scale dataset, and then continues to optimize the network weights of the pre-trained model on the target task dataset, and transfers the relevant knowledge of the source task in the source domain to the target task in the target domain. The source domain can be single or multiple, and can generally be reused for different target tasks. In 1986, Hinton introduced transfer learning in training neural networks. Since large-scale datasets such as ImageNet were used for a large number of computer vision tasks in 2012, transfer learning technology has attracted the interest of many researchers. Pre-training models on ImageNet is the key to achieving advanced performance in various tasks.

[0004] Transfer learning through model adaptation: Model adaptation methods modify the pre-trained model to achieve higher performance on the target task. Popular model adaptation methods include: (1) Linear detection methods, which directly learn the task-specific head on frozen features without changing the pre-trained network weights. Although they are fast and use less video memory during training, and fully retain the representation ability obtained by pre-training, the pre-trained representation does not necessarily help the target task and may lead to poor performance; (2) Fine-tuning methods, which are the most commonly used transfer learning methods and have achieved advanced performance on many computer vision and natural language processing tasks, including image recognition, object detection, semantic segmentation, text classification, and question answering. Fine-tuning can adjust the pre-trained representation to a distribution suitable for the target task, thereby better leveraging the pre-trained knowledge. However, it is easy to damage the representation ability obtained by pre-training that is effective for the target task, such as when the target task dataset has a small number of samples but a large number of model parameters, which can easily lead to overfitting; (3) Adapter tuning methods, which add some lightweight modules to the pre-trained network to adapt the model without changing its original parameters, at the cost of increasing the model size in both training and inference stages.

[0005] Automated transfer learning: There are some preliminary attempts at automated transfer learning, such as activating different parts of the network for each sample during training, dynamically routing between freezing weights or fine-tuning weights for each layer or even each filter for each input sample, using neural architecture search or pruning to adjust the network, and automatically adjusting the learning rate layer by layer. These methods either introduce policy networks, increase the model size, or introduce repeated training cycles, which increases the training cost or inference cost and complicates the entire process.

[0006] Reparameterization: Reparameterization is a method of converting the parameters of a network structure into parameters of another different network structure, so that the functions of the two network structures are equivalent. Recently, the method of introducing multiple mergeable branches in the training phase and using reparameterization to merge branches in the inference phase to keep the network structure unchanged has been widely used to enhance the performance of convolutional neural networks. Reparameterization is also used in neural structure search, such as using reparameterization to merge multiple candidate operations into one operation to improve search efficiency, weighting multiple convolution kernels and generating a new kernel to implement conditional parameterized convolution, etc.

[0007] The prior art discloses a patent for a method for searching for a re-parameterized neural network architecture for edge computing. The patent designs a linear operator and a multi-branch block structure; then, a super network is constructed by stacking the multi-branch block structure; then, the super network is trained by a gradient-based one-stage search algorithm; then, the redundant branches in the super network are deleted to construct the best sub-network; then, the multi-branch best sub-network is converted into a single-branch network; finally, the single-branch network is used to complete task reasoning; the patent is used to search for a neural network structure that can be re-parameterized, while ensuring the accuracy of reasoning, the real-time nature of reasoning and the high efficiency of model operations. However, the patent rarely involves technical solutions on how to achieve automatic adaptation of pre-trained models without additional search costs. Summary of the invention

[0008] The present invention provides a parameter-free automatic self-adaptation method of a pre-trained neural network based on re-parameterization, which realizes the automatic self-adaptation of the pre-trained model without requiring additional search costs.

[0009] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0010] A parameter-free automatic adaptive method for a pre-trained neural network based on re-parameterization, comprising the following steps:

[0011] S1: Each convolution-batch normalization layer of the network is expanded into a weighted sum of two branches, a frozen branch whose parameters come from the pre-trained model and a trainable branch for adapting to the target task;

[0012] S2: Perform reparameterized adaptive tuning on the target task, jointly optimize network parameters and branch scaling factors, and simulate fine-tuning at arbitrary learning rates for each filter;

[0013] S3: Reparameterization is used during the inference phase to equivalently transform the two branches into a single weight layer.

[0014] Furthermore, in step S1, each convolution-batch normalization layer of the network is expanded into a weighted sum of two branches, a frozen branch whose parameters come from the pre-trained model and a trainable branch for adapting the target task, that is, each convolution-batch normalization layer of the image recognition neural network is expanded into a weighted sum of two convolution-batch normalization branches, both branches are initialized with the same pre-trained network parameters, one branch is frozen, and the other branch is not frozen.

[0015] Furthermore, in step S2, re-parameterized adaptive tuning is performed on the target task, and the network parameters and branch scaling factors are jointly optimized to simulate fine-tuning at any learning rate for each filter. The network in step S1 is end-to-end trained on the target task, and no additional training iterations or search phases are introduced to simulate model fine-tuning in which each filter has an adaptive learning rate.

[0016] Furthermore, in step S3, reparameterization is used in the inference stage to convert the two branches equivalently into a single weight layer without introducing additional model parameters and computational costs. Each two convolution-batch normalization branches of the network trained in step S2 are merged into a single weight layer by reparameterization, so that the inference stage does not increase the model parameters and computational costs, that is, parameter-free model adaptation is achieved.

[0017] Furthermore, in step S1, ψ is used to represent a pre-trained model equipped with a randomly initialized task-specific head, given a target The target task, model adaptation generates the target model ψ by executing a specific migration strategy ζ ★ :

[0018]

[0019] In the field of computer vision, the transfer learning strategy of convolutional networks uses the same network structure as the pre-trained model on the target task, where the network parameters ω are inherited from the pre-trained ω0. These strategies can be roughly divided into two categories: fine-tuning strategies and linear detection strategies. The essential difference between these strategies lies in the learning rate of each network parameter. Without loss of generality, let η m is the reference learning rate, η∈R |i| represents the learning rate multiplier η of all |i| filters (i) ∈R where i represents the filter index and |i| represents the total number of filters. Given the hyperparameter η, the goal of these model adaptation strategies is:

[0020]

[0021] In this framework, the fine-tuning strategy is equivalent to setting a global η for all parameters, while the linear probing strategy is equivalent to setting η = 0 for all pre-trained parameters;

[0022] The soft balance between freezing filters and fine-tuning filters represents the intermediate state between them, that is, tuning filters with a smaller learning rate, while the optimal learning rate for each parameter is affected by data distribution and model structure factors, aiming to automatically learn a reasonable migration strategy ζ for the target task. The goal of automated model adaptation is expressed as:

[0023]

[0024] In order to achieve a soft balance between the frozen pre-trained weights and the fully fine-tuned weights, each convolution-batch normalization layer of the network is expanded into two branches, a frozen branch with parameters from the pre-trained model and a trainable branch for adapting the target task. Both branches are initialized with the same pre-trained network parameters; a scaling factor δ is explicitly introduced for each filter in both branches. Where d represents the filter number in a network layer, and the reparameterized adaptation is defined as:

[0025]

[0026] where ω0 and represents the parameters of the pre-trained network and the parameters of the fine-tuning branch, and diag(·) represents the conversion of the vector into a diagonal matrix.

[0027] Furthermore, in step S2, the process of re-parameterization adaptation to simulate fine-tuning at different learning rates for each training iteration is:

[0028] Given a pre-trained linear network layer ψ(ω0), use Indicates that this layer uses a learning rate η m Implement an adaptive tuning strategy, using Indicates that all d filters in this layer are respectively adjusted with learning rate Implement fine-tuning strategies;

[0029] If the network functions of the two strategies are the same at the tth training iteration, that is, Then there is always a certain scaling factor So that after one training iteration, the network functions of the two strategies are still the same, that is,

[0030] The network functions of these two strategies are the same at the tth training iteration, and

[0031] First, calculate the adaptive The weight change:

[0032]

[0033] set up The function of this layer after one adaptive tuning iteration is:

[0034]

[0035] For fine-tuning:

[0036]

[0037] Recall and Therefore, there exists:

[0038] Make

[0039] Furthermore, by setting the scaling factor The learning rate is η m The training reparameterization adaptation can express any learning rate for each training iteration The BN layer usually uses the statistics of the current batch to simulate the data distribution during the training phase, which leads to BN nonlinearity and makes the above analysis not strictly true. There is no obvious difference between using batch statistics and using frozen cumulative statistics.

[0040] Further, in step S2, re-parameterization adaptation is implemented for each layer of the pre-trained model, and the coefficients δ and ζ are made learnable; in the above analysis, δ and ζ are fixed and cannot be learned. In order to make them learnable, it is necessary to use separate training iterations to alternately optimize the scaling factors δ, ζ and the network parameter ω, that is, use one training iteration to fix δ and ζ to optimize ω, so that equations (6) and (8) are established, and then use one training iteration to fix ω to optimize δ and ζ to simulate the optimization of the learning rate, and then repeat these two training iterations;

[0041] To save training cost and simplify the process, the adaptive parameters are jointly optimized in each training iteration. and the factors of all layers Based on the above analysis, there are Therefore, the original goal of automated model adaptation in equation (3) is transformed into:

[0042]

[0043] Among them, the pre-trained ω0 and the reference learning rate η m are task-independent network parameters and hyperparameters. Since δ and ζ participate in the loss calculation, they can be simply are optimized together via back-propagation; thus, reparameterized adaptive tuning mimics model fine-tuning with adaptive learning rates for each filter without introducing additional training iterations or search phases.

[0044] Furthermore, the weights γ of the BN layers of the two branches are contained in ω0, In , they have the same shape and play a similar role as the scaling factors δ and ζ, which are directly used as scaling factors; both the frozen branch and the fine-tuning branch are initialized with the same pre-trained network parameters, that is, at the 0th training iteration Then there is Therefore, you need to add for each branch The coefficient of

[0045] Furthermore, in step S3, using the soft weights of the two branches for adaptation usually introduces twice the model size and computational complexity, resulting in higher inference costs; using a reparameterization strategy to avoid this dilemma, an adaptive network layer It consists of a frozen branch and a fine-tuning branch. Both branches consist of a convolutional Conv layer and a batch normalization BN layer. The weights and optional bias of the convolutional layer can be represented by w and b respectively. The BN layer performs channel-by-channel normalization and scaling, using μ and σ to represent the cumulative mean and variance of the training samples, and using γ and β as channel-by-channel scaling factors. A Conv-BN branch is merged into a single Conv layer as shown below:

[0046]

[0047] Where ★ represents the convolution operation, w ′ and b ′ are the weights and biases of the merged Conv layer; if the outputs of two or more Conv layers with the same configuration are added, merge them into one Conv; let w0, b0 be the weights and biases of the frozen Conv-BN branch, respectively. Fine-tune the weights and biases of the Conv-BN branch respectively, and the weighted sum of the two branches is converted to:

[0048]

[0049] Therefore, combined into a single convolutional layer:

[0050]

[0051] Thus, during the inference phase, the reparameterized adaptive Can be equivalently converted to a single weight layer This enables parameter-free model adaptation.

[0052] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0053] The method of the present invention is based on a reparameterization method, which enables the filter to achieve a soft balance between frozen pre-trained weights and fully fine-tuned weights, and can be equivalently converted into a single weight layer without introducing additional model parameters in the inference stage; analysis shows that this method can simulate fine-tuning each filter in the neural network layer with different learning rates. Therefore, this method realizes automatic adaptation of the pre-trained model without additional search costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of the method of the present invention;

[0055] Figure 2 This is a schematic diagram of reparameterized adaptation. DETAILED DESCRIPTION

[0056] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0057] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0058] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0059] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0060] Example 1

[0061] like Figure 1 As shown, a parameter-free automatic adaptive method based on a re-parameterized pre-trained neural network comprises the following steps:

[0062] S1: Each convolution-batch normalization layer of the network is expanded into a weighted sum of two branches, a frozen branch whose parameters come from the pre-trained model and a trainable branch for adapting to the target task;

[0063] S2: Perform reparameterized adaptive tuning on the target task, jointly optimize network parameters and branch scaling factors, and simulate fine-tuning at arbitrary learning rates for each filter;

[0064] S3: Reparameterization is used during the inference phase to equivalently transform the two branches into a single weight layer.

[0065] In step S1, each convolution-batch normalization layer of the network is expanded into a weighted sum of two branches, a frozen branch whose parameters come from the pre-trained model and a trainable branch for adapting the target task. Each convolution-batch normalization layer of the image recognition neural network is expanded into a weighted sum of two convolution-batch normalization branches, both branches are initialized with the same pre-trained network parameters, one of which is frozen and the other is not frozen.

[0066] In step S2, re-parameterized adaptive tuning is performed on the target task to jointly optimize the network parameters and branch scaling factors, and fine-tuning at any learning rate is simulated for each filter. The network in step S1 is trained end-to-end on the target task without introducing additional training iterations or search phases, and the model fine-tuning with adaptive learning rate for each filter can be simulated.

[0067] In step S3, reparameterization is used in the inference stage to convert the two branches equivalently into a single weight layer without introducing additional model parameters and computational costs. Each two convolution-batch normalization branches of the network trained in step S2 are merged into a single weight layer by reparameterization, so that the inference stage does not increase the model parameters and computational costs, that is, parameter-free model adaptation is achieved.

[0068] Example 2

[0069] like Figure 1 As shown, a parameter-free automatic adaptive method based on a re-parameterized pre-trained neural network comprises the following steps:

[0070] S1: Each convolution-batch normalization layer of the network is expanded into a weighted sum of two branches, a frozen branch whose parameters come from the pre-trained model and a trainable branch for adapting to the target task;

[0071] S2: Perform reparameterized adaptive tuning on the target task, jointly optimize network parameters and branch scaling factors, and simulate fine-tuning at arbitrary learning rates for each filter;

[0072] S3: Reparameterization is used during the inference phase to equivalently transform the two branches into a single weight layer.

[0073] like Figure 2 As shown, in step S1, ψ represents a pre-trained model equipped with a randomly initialized task-specific head, given a target The target task, model adaptation generates the target model ψ by executing a specific migration strategy ζ ★ :

[0074]

[0075] In the field of computer vision, the transfer learning strategy of convolutional networks uses the same network structure as the pre-trained model on the target task, where the network parameters ω are inherited from the pre-trained ω0. These strategies can be roughly divided into two categories: fine-tuning strategies and linear detection strategies. The essential difference between these strategies lies in the learning rate of each network parameter. Without loss of generality, let η m is the reference learning rate, η∈R |i| represents the learning rate multiplier η of all |i| filters (i) ∈R where i represents the filter index and |i| represents the total number of filters. Given the hyperparameter η, the goal of these model adaptation strategies is:

[0076]

[0077] In this framework, the fine-tuning strategy is equivalent to setting a global η for all parameters, while the linear probing strategy is equivalent to setting η = 0 for all pre-trained parameters;

[0078] The soft balance between freezing filters and fine-tuning filters represents the intermediate state between them, that is, tuning filters with a smaller learning rate, while the optimal learning rate for each parameter is affected by data distribution and model structure factors, aiming to automatically learn a reasonable migration strategy ζ for the target task. The goal of automated model adaptation is expressed as:

[0079]

[0080] In order to achieve a soft balance between the frozen pre-trained weights and the fully fine-tuned weights, each convolution-batch normalization layer of the network is expanded into two branches, a frozen branch with parameters from the pre-trained model and a trainable branch for adapting the target task. Both branches are initialized with the same pre-trained network parameters; a scaling factor δ is explicitly introduced for each filter in both branches. Where d represents the filter number in a network layer, and the reparameterized adaptation is defined as:

[0081]

[0082] where ω0 and represents the parameters of the pre-trained network and the parameters of the fine-tuning branch, and diag(·) represents the conversion of the vector into a diagonal matrix.

[0083] In step S2, the process of reparameterization adaptation to simulate fine-tuning at different learning rates for each training iteration is:

[0084] Given a pre-trained linear network layer ψ(ω0), use Indicates that this layer uses a learning rate η m Implement an adaptive tuning strategy, using Indicates that all d filters in this layer are respectively adjusted with learning rate Implement fine-tuning strategies;

[0085] If the network functions of the two strategies are the same at the tth training iteration, that is, Then there is always a certain scaling factor So that after one training iteration, the network functions of the two strategies are still the same, that is,

[0086] The network functions of these two strategies are the same at the tth training iteration, and

[0087] First, calculate the adaptive The weight change:

[0088]

[0089] set up The function of this layer after one adaptive tuning iteration is:

[0090]

[0091] For fine-tuning:

[0092]

[0093] Recall and Therefore, there exists:

[0094] Make

[0095] By setting the scaling factor The learning rate is η m The training reparameterization adaptation can express any learning rate for each training iteration The BN layer usually uses the statistics of the current batch to simulate the data distribution during the training phase, which leads to BN nonlinearity and makes the above analysis not strictly true. There is no obvious difference between using batch statistics and using frozen cumulative statistics.

[0096] In step S2, reparameterization adaptation is performed for each layer of the pre-trained model, and the coefficients δ and ζ are made learnable. In the above analysis, δ and ζ are fixed and cannot be learned. In order to make them learnable, it is necessary to use separate training iterations to alternately optimize the scaling factors δ, ζ and the network parameter ω, that is, use one training iteration to fix δ and ζ to optimize ω, so that equations (6) and (8) are established, and then use one training iteration to fix ω to optimize δ and ζ to simulate the optimization of the learning rate, and then repeat these two training iterations.

[0097] To save training cost and simplify the process, the adaptive parameters are jointly optimized in each training iteration. and the factors of all layers Based on the above analysis, there are η=diag(ζ 2 )η m , therefore, the original objective of automated model adaptation in equation (3) is transformed into:

[0098]

[0099] Among them, the pre-trained ω0 and the reference learning rate η m are task-independent network parameters and hyperparameters. Since δ and ζ participate in the loss calculation, they can be simply are optimized together via back-propagation; thus, reparameterized adaptive tuning mimics model fine-tuning with adaptive learning rates for each filter without introducing additional training iterations or search phases.

[0100] The weights γ of the BN layers of the two branches are contained in ω0, In , they have the same shape and play a similar role as the scaling factors δ and ζ, which are directly used as scaling factors; both the frozen branch and the fine-tuning branch are initialized with the same pre-trained network parameters, that is, at the 0th training iteration Then there is Therefore, you need to add for each branch The coefficient of

[0101] In step S3, using soft weights of two branches for adaptation usually introduces twice the model size and computation, resulting in higher inference cost; using a reparameterization strategy to avoid this dilemma, an adaptive network layer It consists of a frozen branch and a fine-tuning branch. Both branches consist of a convolutional Conv layer and a batch normalization BN layer. The weights and optional bias of the convolutional layer can be represented by w and b respectively. The BN layer performs channel-by-channel normalization and scaling, using μ and σ to represent the cumulative mean and variance of the training samples, and using γ and β as channel-by-channel scaling factors. A Conv-BN branch is merged into a single Conv layer as shown below:

[0102]

[0103] Where ★ represents the convolution operation, w′ and b′ are the weights and biases of the merged Conv layer; if the outputs of two or more Conv layers with the same configuration are added, they are merged into one Conv; let w0 and b0 be the weights and biases of the frozen Conv-BN branch respectively, Fine-tune the weights and biases of the Conv-BN branch respectively, and the weighted sum of the two branches is converted to:

[0104]

[0105] Therefore, combined into a single convolutional layer:

[0106]

[0107] Thus, during the inference phase, the reparameterized adaptive Can be equivalently converted to a single weight layer This enables parameter-free model adaptation.

[0108] Example 3

[0109] like Figure 1-2 As shown, a parameter-free automatic adaptive method based on a re-parameterized pre-trained neural network comprises the following steps:

[0110] S1: Each convolution-batch normalization layer of the network is expanded into a weighted sum of two branches, a frozen branch whose parameters come from the pre-trained model and a trainable branch for adapting to the target task;

[0111] S2: Perform reparameterized adaptive tuning on the target task, jointly optimize network parameters and branch scaling factors, and simulate fine-tuning at arbitrary learning rates for each filter;

[0112] S3: Reparameterization is used during the inference phase to equivalently transform the two branches into a single weight layer.

[0113] In step S1, each convolution-batch normalization layer of the network is expanded into a weighted sum of two branches, a frozen branch whose parameters come from the pre-trained model and a trainable branch for adapting the target task. Each convolution-batch normalization layer of the image recognition neural network is expanded into a weighted sum of two convolution-batch normalization branches, both branches are initialized with the same pre-trained network parameters, one of which is frozen and the other is not frozen.

[0114] In step S2, re-parameterized adaptive tuning is performed on the target task to jointly optimize the network parameters and branch scaling factors, and fine-tuning at any learning rate is simulated for each filter. The network in step S1 is trained end-to-end on the target task without introducing additional training iterations or search phases, and the model fine-tuning with adaptive learning rate for each filter can be simulated.

[0115] In step S3, reparameterization is used in the inference stage to convert the two branches equivalently into a single weight layer without introducing additional model parameters and computational costs. Each two convolution-batch normalization branches of the network trained in step S2 are merged into a single weight layer by reparameterization, so that the inference stage does not increase the model parameters and computational costs, that is, parameter-free model adaptation is achieved.

[0116] Specifically, in step S1, ψ represents a pre-trained model equipped with a randomly initialized task-specific head, given a target The target task, model adaptation generates the target model ψ by executing a specific migration strategy ζ ★ :

[0117]

[0118] In the field of computer vision, the transfer learning strategy of convolutional networks uses the same network structure as the pre-trained model on the target task, where the network parameters ω are inherited from the pre-trained ω0. These strategies can be roughly divided into two categories: fine-tuning strategies and linear detection strategies. The essential difference between these strategies lies in the learning rate of each network parameter. Without loss of generality, let η m is the reference learning rate, η∈R |i| represents the learning rate multiplier η of all |i| filters (i) ∈R where i represents the filter index and |i| represents the total number of filters. Given the hyperparameter η, the goal of these model adaptation strategies is:

[0119]

[0120] In this framework, the fine-tuning strategy is equivalent to setting a global η for all parameters, while the linear probing strategy is equivalent to setting η = 0 for all pre-trained parameters;

[0121] The soft balance between freezing filters and fine-tuning filters represents the intermediate state between them, that is, tuning filters with a smaller learning rate, while the optimal learning rate for each parameter is affected by data distribution and model structure factors, aiming to automatically learn a reasonable migration strategy ζ for the target task. The goal of automated model adaptation is expressed as:

[0122]

[0123] In order to achieve a soft balance between the frozen pre-trained weights and the fully fine-tuned weights, each convolution-batch normalization layer of the network is expanded into two branches, a frozen branch with parameters from the pre-trained model and a trainable branch for adapting the target task. Both branches are initialized with the same pre-trained network parameters; a scaling factor δ is explicitly introduced for each filter in both branches. Where d represents the filter number in a network layer, and the reparameterized adaptation is defined as:

[0124]

[0125] where ω0 and represents the parameters of the pre-trained network and the parameters of the fine-tuning branch, and diag(·) represents the conversion of the vector into a diagonal matrix.

[0126] In step S2, the process of reparameterization adaptation to simulate fine-tuning at different learning rates for each training iteration is:

[0127] Given a pre-trained linear network layer ψ(ω0), use Indicates that this layer uses a learning rate η m Implement adaptive tuning strategy, using Indicates that all d filters in this layer are respectively adjusted with learning rate Implement fine-tuning strategies;

[0128] If the network functions of the two strategies are the same at the tth training iteration, that is, Then there is always a certain scaling factor So that after one training iteration, the network functions of the two strategies are still the same, that is,

[0129] The network functions of these two strategies are the same at the tth training iteration, and

[0130] First, calculate the adaptive The weight change:

[0131]

[0132] set up The function of this layer after one adaptive tuning iteration is:

[0133]

[0134] For fine-tuning:

[0135]

[0136] Recall and Therefore, there exists:

[0137] Make

[0138] By setting the scaling factor The learning rate is η m The training reparameterization adaptation can express any learning rate for each training iteration The BN layer usually uses the statistics of the current batch to simulate the data distribution during the training phase, which leads to BN nonlinearity and makes the above analysis not strictly true. There is no obvious difference between using batch statistics and using frozen cumulative statistics.

[0139] In step S2, reparameterization adaptation is performed for each layer of the pre-trained model, and the coefficients δ and ζ are made learnable. In the above analysis, δ and ζ are fixed and cannot be learned. In order to make them learnable, it is necessary to use separate training iterations to alternately optimize the scaling factors δ, ζ and the network parameter ω, that is, use one training iteration to fix δ and ζ to optimize ω, so that equations (6) and (8) are established, and then use one training iteration to fix ω to optimize δ and ζ to simulate the optimization of the learning rate, and then repeat these two training iterations.

[0140] To save training cost and simplify the process, the adaptive parameters are jointly optimized in each training iteration. and the factors of all layers Based on the above analysis, there are η=diag(ζ 2 )η m , therefore, the original objective of automated model adaptation in equation (3) is transformed into:

[0141]

[0142] Among them, the pre-trained ω0 and the reference learning rate η mare task-independent network parameters and hyperparameters. Since δ and ζ participate in the loss calculation, they can be simply are optimized together via back-propagation; thus, reparameterized adaptive tuning mimics model fine-tuning with adaptive learning rates for each filter without introducing additional training iterations or search phases.

[0143] The weights γ of the BN layers of the two branches are contained in ω0, In , they have the same shape and play a similar role as the scaling factors δ and ζ, which are directly used as scaling factors; both the frozen branch and the fine-tuning branch are initialized with the same pre-trained network parameters, that is, at the 0th training iteration Then there is Therefore, you need to add for each branch The coefficient of

[0144] In step S3, using soft weights of two branches for adaptation usually introduces twice the model size and computation, resulting in higher inference cost; using a reparameterization strategy to avoid this dilemma, an adaptive network layer It consists of a frozen branch and a fine-tuning branch. Both branches consist of a convolutional Conv layer and a batch normalization BN layer. The weights and optional bias of the convolutional layer can be represented by w and b respectively. The BN layer performs channel-by-channel normalization and scaling, using μ and σ to represent the cumulative mean and variance of the training samples, and using γ and β as channel-by-channel scaling factors. A Conv-BN branch is merged into a single Conv layer as shown below:

[0145]

[0146] Where ★ represents the convolution operation, w ′ and b ′ are the weights and biases of the merged Conv layer; if the outputs of two or more Conv layers with the same configuration are added, merge them into one Conv; let w0, b0 be the weights and biases of the frozen Conv-BN branch, respectively. Fine-tune the weights and biases of the Conv-BN branch respectively, and the weighted sum of the two branches is converted to:

[0147]

[0148] Therefore, combined into a single convolutional layer:

[0149]

[0150] Thus, during the inference phase, the reparameterized adaptive Can be equivalently converted to a single weight layer This enables parameter-free model adaptation.

[0151] The same or similar reference numerals correspond to the same or similar components;

[0152] The positional relationships described in the drawings are only for illustrative purposes and should not be construed as limiting the present patent.

[0153] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A parameter-free automatic adaptive method based on a re-parameterized pre-trained neural network, the method being used for image recognition, characterized in that: The following steps are involved: S1: Each convolution-batch normalization layer of the network is expanded into a weighted sum of two branches, a frozen branch whose parameters come from the pre-trained model and a trainable branch for adapting to the target task; S2: Perform reparameterized adaptive tuning on the target task, jointly optimize network parameters and branch scaling factors, and simulate fine-tuning at arbitrary learning rates for each filter; S3: In the inference phase, reparameterization is used to convert the two branches equivalently into a single weight layer; In the step S1, each convolution-batch normalization layer of the network is expanded into a weighted sum of two branches, a frozen branch whose parameters come from the pre-trained model and a trainable branch for adapting the target task, that is, each convolution-batch normalization layer of the image recognition neural network is expanded into a weighted sum of two convolution-batch normalization branches, both branches are initialized with the same pre-trained network parameters, one branch is frozen, and the other branch is not frozen; In step S1, ψ represents a pre-trained model equipped with a randomly initialized task-specific head, given a target The target task, model adaptation by executing a specific migration strategy Generate the target model ψ*: In the field of computer vision, the transfer learning strategy of convolutional networks uses the same network structure as the pre-trained model on the target task, where the network parameters ω are inherited from the pre-trained ω0. These strategies are divided into two categories: fine-tuning strategies and linear detection strategies. The essential difference between these strategies lies in the learning rate of each network parameter. Without loss of generality, let η m is the reference learning rate, η∈R |i| represents the learning rate multiplier η of all |i| filters (i) ∈R where i represents the filter index and |i| represents the total number of filters. Given the hyperparameter η, the goal of these model adaptation strategies is: In this framework, the fine-tuning strategy is equivalent to setting a global η for all parameters, while the linear probing strategy is equivalent to setting η = 0 for all pre-trained parameters; The soft balance between freezing filters and fine-tuning filters represents an intermediate state between them, that is, tuning filters with a smaller learning rate, while the optimal learning rate for each parameter is affected by data distribution and model structure factors, aiming to automatically learn a reasonable migration strategy for the target task. The goal of automated model adaptation is expressed as: In order to achieve a soft balance between the frozen pre-trained weights and the fully fine-tuned weights for the filters, each convolution-batch normalization layer of the network is expanded into two branches, a frozen branch with parameters from the pre-trained model and a trainable branch for adapting the target task, both of which are initialized with the same pre-trained network parameters; each filter in both branches explicitly introduces a scaling factor Where d represents the filter number in a network layer, and the reparameterized adaptation is defined as: where ω0 and represents the parameters of the pre-trained network and the parameters of the fine-tuning branch, and diag(·) represents the conversion of the vector into a diagonal matrix; In step S3, using the soft weights of the two branches for adaptation usually introduces twice the model size and computational complexity, resulting in higher inference costs. A reparameterization strategy is used to avoid this dilemma. An adaptive network layer It consists of a frozen branch and a fine-tuning branch. Both branches consist of a convolutional Conv layer and a batch normalization BN layer. The weights and optional bias of the convolutional layer can be represented by w and b respectively. The BN layer performs channel-by-channel normalization and scaling, using μ and σ to represent the cumulative mean and variance of the training samples, and using γ and β as channel-by-channel scaling factors. A Conv-BN branch is merged into a single Conv layer as shown below: Where * denotes the convolution operation, w′ and b′ are the weights and biases of the merged Conv layer; if the outputs of two or more Conv layers with the same configuration are added, they are merged into one Conv; let W0, b0 be the weights and biases of the frozen Conv-BN branch, respectively. Fine-tune the weights and biases of the Conv-BN branch respectively, and the weighted sum of the two branches is converted to: Therefore, combined into a single convolutional layer: Thus, during the inference phase, the reparameterized adaptive Can be equivalently converted to a single weight layer This enables parameter-free model adaptation.

2. The parameter-free automatic self-adaptation method based on re-parameterized pre-trained neural network according to claim 1, characterized in that: In the step S2, re-parameterized adaptive tuning is performed on the target task, the network parameters and branch scaling factors are jointly optimized, and fine-tuning at any learning rate is simulated for each filter. The network in step S1 is end-to-end trained on the target task, and no additional training iterations or search stages are introduced to simulate model fine-tuning in which each filter has an adaptive learning rate.

3. The parameter-free automatic self-adaptation method based on re-parameterized pre-trained neural network according to claim 2, characterized in that: In step S3, reparameterization is used in the inference stage to convert the two branches equivalently into a single weight layer without introducing additional model parameters and computational costs. Every two convolution-batch normalization branches of the network trained in step S2 are merged into a single weight layer by reparameterization, so that the model parameters and computational costs are not increased in the inference stage, that is, parameter-free model adaptation is achieved.

4. The parameter-free automatic self-adaptation method based on re-parameterized pre-trained neural network according to claim 3, characterized in that: In step S2, the process of re-parameterization adaptation to simulate fine-tuning at different learning rates for each training iteration is: Given a pre-trained linear network layer ψ(ω0), use Indicates that this layer uses a learning rate η m Implement an adaptive tuning strategy, using Indicates that all d filters in this layer are respectively adjusted with learning rate Implement fine-tuning strategies; If the network functions of the two strategies are the same at the tth training iteration, that is, Then there is always a certain scaling factor So that after one training iteration, the network functions of the two strategies are still the same, that is, The network functions of these two strategies are the same at the tth training iteration, and First, calculate the adaptive The weight change: set up The function of this layer after one adaptive tuning iteration is: For fine-tuning: Recall and Therefore, there exists: Make 5. The parameter-free automatic self-adaptation method based on re-parameterized pre-trained neural network according to claim 4, characterized in that: By setting the scaling factor The learning rate is η m The training reparameterization adaptation can express any learning rate for each training iteration The BN layer usually uses the statistics of the current batch to simulate the data distribution during the training phase, which leads to BN nonlinearity and makes the above analysis not strictly true. There is no obvious difference between using batch statistics and using frozen cumulative statistics.

6. The parameter-free automatic self-adaptation method based on re-parameterized pre-trained neural network according to claim 5, characterized in that: In step S2, reparameterization adaptation is performed for each layer of the pre-trained model, and the coefficients δ and can be learned; in the above analysis, δ and is fixed and cannot be learned. To make it learnable, separate training iterations are needed to alternately optimize the scaling factors δ and and network parameters ω, i.e., fix δ and To optimize ω, make equations (6) and (8) hold, and then use a training iteration to fix ω to optimize δ and To simulate the optimization of the learning rate, these two training iterations are then repeated; To save training cost and simplify the process, the adaptive parameters are jointly optimized in each training iteration. and the factors of all layers Based on the above analysis, there are Therefore, the original goal of automated model adaptation in equation (3) is transformed into: Among them, the pre-trained ω0 and the reference learning rate η m are task-independent network parameters and hyperparameters, since δ and participate in the loss calculation, so it can be simply are optimized together via back-propagation; thus, reparameterized adaptive tuning mimics model fine-tuning with adaptive learning rates for each filter without introducing additional training iterations or search phases.

7. The parameter-free automatic self-adaptation method based on re-parameterized pre-trained neural network according to claim 6, characterized in that: The weights γ of the BN layers of the two branches are contained in ω0, , have the same shape and are scaled by factors δ and Play a similar role and use them directly as scaling factors; both the frozen branch and the fine-tuned branch are initialized with the same pre-trained network parameters, i.e., at the 0th training iteration Then there is Therefore, you need to add for each branch The coefficient of

Images