Modular approach to neural network models

Abstract

The application realizes a modular method of a neural network model by methods in the fields of software engineering and artificial intelligence, and generally includes three parts of a search space, a performance evaluation strategy and a search strategy; an original model is an N-class pre-training model composed of a plurality of neural network layers and having L weights, and a K-class image or text dataset of a target task is input at the same time, and six steps are used to realize the modularization of the pre-training neural network model for the classification task; the obtained module only retains the relevant part of the weights of the N-class model related to the target task and is used to classify the image or text data to be classified of the target task; the method provided by the application adopts a gradient-based discrete space search method, improves the recognition accuracy of the relevant weights through the accuracy index of the target task, and at the same time, reduces the time and calculation overhead in the classification of the image or text by setting the irrelevant weights to zero.

Description

A modular approach to neural network models Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a modular method for neural network models. Background Technology

[0002] As intelligent software systems play an increasingly important role in numerous application scenarios such as autonomous driving, question-answering systems, and scientific research, the training of neural network models, as the core of intelligent software systems, has become a crucial task in the software development process. However, with neural network models becoming increasingly complex and large, the overhead of model construction (such as training time and computational resource costs) is also increasing dramatically. Neural network models are considered "Software 2.0," and inspired by software reuse, the reuse of existing pre-trained neural network models to reduce (or even eliminate) the high cost of model construction has been widely researched and applied.

[0003] Existing model reuse techniques are mainly divided into two categories: model selection and network-based deep transfer learning. Model selection techniques compare the similarity between the features of candidate pre-trained models and the features of the target task data, or test the candidate models using a small number of test samples from the target task, to select the candidate model that can be directly reused and has the highest performance as accurately as possible, while minimizing the time and manpower costs associated with model selection. Network-based deep transfer learning techniques reuse pre-trained models and further fine-tune all or some network layers on the target task data (through manual or automatic selection). Compared to training a new model from scratch, transfer learning requires less time and labeled data for the target task to train a new model for the target task. However, both of these techniques focus more on whether the performance of the reused pre-trained model (after fine-tuning) meets the requirements of the target task, ignoring the issues of overhead and security in model reuse. The original task solved by the pre-trained model and the target task may be inclusive or overlapping; for example, the target task may be a sub-classification task within the original classification task. While pre-trained models can be directly reused for the target task, some of their neurons and weights may be responsible for identifying categories that are not part of the target task. The computational overhead generated by these neurons and weights that are irrelevant to the target task is additional and can be expensive, and may also introduce security risks such as "backdoors".

[0004] To address the aforementioned issues, some researchers have attempted to modularize pre-trained models, with each module retaining only a portion of the model's neurons and weights, and reusing only modules relevant to the target task. Existing neural network modularization methods include attention-based and slice-based approaches. Essentially, both of these modularization methods identify neurons and weights relevant to the target task based on neuron coverage, thereby decomposing the pre-trained neural network model into reusable modules. A module contains a portion of the model's neurons or weights, capable of solving a subtask of the original task (e.g., a binary classification subtask within a multi-class classification task). By reusing modules on demand rather than simply reusing the entire model, the overhead and security issues of reuse are mitigated.

[0005] Existing modularization methods for neural network models rely on neuron coverage to identify neurons and weights relevant to the target task. However, the effectiveness of neuron coverage remains controversial. Activated neurons may not be essential for the target task, thus existing methods may fail to accurately identify relevant weights, resulting in the retention of irrelevant neurons and weights in modules. This leads to high reuse costs for modules and slices and still presents significant security risks. Furthermore, existing work often generates modules of the same size as the model, resulting in high module reuse costs.

[0006] This invention addresses the modularization of neural network models by proposing a gradient-based discrete space search modularization method. This method solves the problem of inaccurate identification of relevant weights in existing works, enabling the generated modules to retain only weights relevant to the target task, thereby reducing module reuse overhead and improving reuse security.

[0007] Specifically, the main problems addressed include: (1) Training image or text classification models, especially large models, leads to significant time and computational overhead. This invention reduces the training overhead of image or text classification models by reusing modules of pre-trained models. (2) Existing model reuse techniques may result in significant additional model reuse overhead when reusing pre-trained models, and face security risks such as adversarial defects and backdoor inheritance. This invention improves the inference speed and reuse security of models in image or text classification tasks by reusing modules. (3) Existing research uses neuron coverage to indirectly evaluate the correlation between weights and the target task, thus the identification of relevant weights is not accurate enough. This invention adopts a modular method based on gradient-based discrete space search, directly evaluating the correlation between weights and the target task through the accuracy index of the target task, thereby improving the accuracy of relevant weight identification. (4) Existing methods generate modules by setting irrelevant neurons to zero, and the module size is the same as the model size, resulting in large reuse overhead. This invention resets irrelevant weights to zero, enabling image or text classification modules to be compressed and accelerated, thus reducing reuse overhead. Summary of the Invention

[0008] For this reason, the present invention first proposes a modularization method for a neural network model, which is characterized in that:

[0009] Process an input multi-class pre-trained neural network model and a multi-class dataset such as images or texts of a target task through a processing unit, which includes three parts: a search space, a performance evaluation strategy, and a search strategy; the input of the method specifically includes: a pre-trained multi-class model such as an image or text as the original model, which is composed of several neural network layers and is an N-class model with a total of L weights; and a K-class dataset such as images or texts of a target task, where K < N, including data for modularization and data to be classified; the method realizes the modularization of the original model through six steps, generates a K-class neural network module for the target task and only contains J weights, where J < L, and further uses the neural network module to classify the data to be classified in the target task, and finally outputs the classified data such as images or texts;

[0010] Step 1, construct the search space. The search space contains all candidate modules. A candidate module is constructed by selectively deleting some weights of the original model using a mask. A mask is a bit vector [0,1] with a length of L L , where L is the number of weights in the original model. The bit "0" represents deleting the corresponding weight, and the bit "1" represents retaining the corresponding weight. There are a total of 2 L candidate masks, so the search space consists of 2 L candidate modules;

[0011] Step 2, perform search initialization in the search space. When starting the search, the values of all elements of the initialized candidate mask are 1, indicating that the initialized module retains all weights. This initialized candidate mask is passed to the performance evaluation strategy;

[0012] Step 3, use the performance evaluation strategy to evaluate the performance of the candidate module. Given a candidate mask, the performance evaluation strategy first removes the weights in the model corresponding to the elements with a value of 0 in the mask, and retains the weights corresponding to the elements with a value of 1. Then, at the last layer of the model, a new output head is appended as the output layer to construct a candidate module; the output head is a fully connected network layer used to make the candidate module adapt to the number of classifications of the target task; then, on the input data such as images or texts for modularization, calculate the objective function value to evaluate the candidate module; the objective function is defined as the weighted sum of the following two parts: the weight retention rate of the candidate module, and the cross-entropy loss between the prediction of the candidate module and the labels on the target dataset; the obtained objective function value will be fed back to the search strategy;

[0013] Step 4: Search for candidate modules using the search strategy. The search strategy uses a gradient-based search method. Based on the objective function value, it calculates the gradient of each value in the mask and output header, and updates the mask and output header through gradient descent. In each round of search, the search strategy passes the updated mask and output header as new candidate modules to the performance evaluation strategy.

[0014] Step 5: Iterate through Steps 3 and 4 until the number of iterations reaches a given value or the objective function value no longer changes. The resulting module retains only the weights related to the target task in the N-class model and can be used for K-class target tasks.

[0015] Step six: Using the neural network module obtained in step five, classify the input data such as images or text to be classified. Finally, the method outputs the classified images or text.

[0016] The method for calculating the objective function value used to evaluate the candidate module is as follows: First, a candidate module is constructed by multiplying the weights of the original model by the mask to remove some of the weights from the original model, thus obtaining an intermediate model. Since the output layer has N neurons, the intermediate model is an N-class classification model. A new output head, i.e., a fully connected network layer, is appended after the intermediate model to complete the construction of the candidate module. The output head is randomly initialized in the first round of search and is updated along with the mask in subsequent search processes. The output head has K neurons, which converts the N-class classification prediction result of the intermediate model into a K-class classification prediction result, enabling the candidate module to adapt to the target task.

[0017] After constructing the candidate modules, the cross-entropy loss L between the prediction results of the candidate modules on the target dataset and the true labels is calculated. ce Calculate using the following formula:

[0018]

[0019] Where K is the number of categories for the target task, M and H are the mask and output header, respectively, and P... i (M,H) represents the prediction of the i-th category by the candidate module constructed using M and H, t iThis represents the probability of the i-th class in the one-hot encoding of a label. Here, the one-hot encoding of a label is a binary vector of length K, where each element corresponds to one of the K classes. Only the element corresponding to that label class in the one-hot encoding has a value of 1, while the other elements have a value of 0. A lower cross-entropy loss means that the candidate module retains more weights relevant to the target task, thus achieving higher classification accuracy on the target dataset.

[0020] Weight retention rate L wr It can be obtained directly through mask calculation:

[0021]

[0022] Where L is the number of weights in the original model. A lower weight retention rate indicates that the candidate module retains fewer weights. Based on L... ce and L wr The objective function O is defined as follows:

[0023] O = L ce +α×L wr ,

[0024] Here, α is the weighting coefficient, which is usually set to 1.0 in experiments. To minimize O, the search process tends to find a candidate module that retains only weights relevant to the target task, because this candidate module retains as few weights as possible while achieving the lowest cross-entropy loss.

[0025] The gradient-based search method is as follows: In each round of search, based on the objective function value of the candidate modules from the previous round, the mask and output head are updated using gradient descent to find a new candidate module with a smaller objective function value for updating, as shown in the following formula:

[0026]

[0027]

[0028] Where ξ is the learning rate, and M' is the updated mask, corresponding to a candidate module with a smaller objective function value;

[0029] When applying gradient descent to search for a new mask in the search space, since the values ​​of the elements in the mask are discrete (i.e., 0 or 1), and gradient descent requires the search space to be continuous, each element in the mask is assigned a continuous value as the weight corresponding to that element and its relevance to the target task. Then, based on the indicator function... Elements with a relevance greater than 0 corresponding to their weights are set to 1, while other elements are set to 0. Once the condition that the search space is continuous is met, a pass-through estimator technique is used to estimate the gradient of the indicator function. The pass-through estimator estimates the gradient of the current network layer by using the gradient of the previous neural network layer.

[0030] The gradient of the output head is obtained by... The calculated value is updated along with the mask. After updating the mask and output header, the search strategy passes the new mask and output header as a new candidate module to the performance evaluation strategy, and after receiving the objective function value returned by the performance evaluation strategy, begins the next round of search.

[0031] The technical effects to be achieved by this invention are as follows:

[0032] (1) The present invention adopts a gradient-based discrete space search method, which directly evaluates the relevance of each weight of the pre-trained model to the target task by the classification accuracy index of classification tasks such as images or text, and searches for relevant weights, thereby improving the recognition accuracy of relevant weights.

[0033] (2) The present invention constructs modules by resetting irrelevant weights to zero, which enables the modules to be compressed and accelerated, thereby reducing the time and computational overhead of classifying images or texts. Attached Figure Description

[0034] Figure 1. Overall workflow of SeaM;

[0035] Figure 2 shows a module constructed using a mask and an output header; Detailed Implementation

[0036] The following are preferred embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0037] This invention proposes a modular approach to neural network models, applicable to classification tasks such as images or text.

[0038] As shown in Figure 1, this invention proposes a modular framework for neural network models, SeaM, and designs a modular method based on gradient-based discrete space search. The SeaM framework takes a pre-trained neural network model for multi-class classification (e.g., images or text) and a multi-class dataset for the target task (e.g., images or text) as input. The logic for processing the pre-trained neural network model includes three parts: search space, performance evaluation strategy, and search strategy. Given an N-class original model (e.g., the pre-trained three-class model in Figure 1, i.e., N=3), which consists of several neural network layers with a total of L weights (3 layers in the example in Figure 1, with a total of 15 weights), and a K-class dataset for the target task (a binary dataset in the example in Figure 1, i.e., K=2), including the image or text data used for modularization and the image or text data to be classified.

[0039] The process of SeaM generating modules and using these modules to classify images or text involves the following six steps:

[0040] Step one: Construct the search space. The search space contains all candidate modules. A candidate module is constructed by selectively removing some weights from the original model using a mask. A mask is a bit vector of length L [0,1]. L Where L is the number of weights in the original model, bit "0" represents deleting the corresponding weight, and bit "1" represents keeping the corresponding weight. There are a total of 2... L There are 2 candidate masks, therefore the search space is 2 L It consists of 2 candidate modules. The example in Figure 1 contains a total of 2 15 One candidate mask;

[0041] Step two involves initializing the search within the search space. At the start of the search, all elements of the initialized candidate mask are set to 1, indicating that the initialization module retains all weights. This initialized candidate mask is then passed to the performance evaluation strategy.

[0042] Step 3: Evaluate the performance of the candidate module using the performance evaluation strategy described above. Given a candidate mask, the performance evaluation strategy first removes the weights in the model corresponding to elements with a value of 0 in the mask, while retaining the weights corresponding to elements with a value of 1. Then, a new output head is appended as the output layer in the last layer of the model, thereby constructing a candidate module. The output head is a fully connected layer used to adapt the candidate module to the number of classifications of the target task. Then, on the input image or text data used for modularization, the objective function value is calculated to evaluate the candidate module. The objective function is defined as a weighted sum of the following two parts: the weight retention rate of the candidate module, and the cross-entropy loss between the predictions of the candidate module and the labels on the target dataset. The obtained objective function value is fed back to the search strategy;

[0043] Step 4: Search for candidate modules using the search strategy. The search strategy uses a gradient-based search method. Based on the objective function value, it calculates the gradient of each value in the mask and output header, and updates the mask and output header through gradient descent. In each round of search, the search strategy passes the updated mask and output header as new candidate modules to the performance evaluation strategy.

[0044] Step five: Iterate through steps three and four until the number of iterations reaches a given value or the objective function value essentially stops changing. The resulting module retains only the weights relevant to the target task from the N-class model, making it suitable for K-class target tasks. In the example shown in Figure 1, the module retains 7 of the original model's 15 weights, enabling binary classification.

[0045] Step six involves using the neural network module obtained in step five to classify the input image or text data. Since the module retains only some weights from the original model, its classification process incurs lower time and computational costs compared to the original model. Finally, the method outputs classified image or text data. The following details the process of modularizing and reusing the pre-trained model on the target task dataset.

[0046] Specific methods for constructing the search space:

[0047] The goal of modularization is to obtain a module that retains only the weights relevant to the target task from the original model. Neural network modularization is defined as a search problem that seeks a module from all candidate modules that selectively removes some weights from the original model. The module found that retains only the weights relevant to the target task is considered the optimal module. In this problem, the search space consists of all possible modules. To facilitate technically solving this problem in practice, a mask that records which weights are removed and retained in a candidate module is used to represent it, thus omitting unnecessary details such as max-pooling and dropout layers. Therefore, in SeaM, the search space consists of all candidate masks.

[0048] Specifically, the mask is a bit vector of length L [0,1]. L Where L is the number of weights in the original model, and 0 (or 1) represents the corresponding weight that is removed (or retained). Figure 2 illustrates the process of building modules by removing weights from the original model using a mask. By multiplying the weights of the pre-trained model by the mask, SeaM sets irrelevant weight values ​​to zero and retains the values ​​of relevant weights. During inference, weights with values ​​set to zero participate in the calculation but have no effect on the prediction, thus achieving the effect of removing irrelevant weights. After modularization, calculations involving zero-value weights in the module can be eliminated using sparse model runtime libraries (such as DeepSparse).

[0049] After constructing the search space, a mask initialized with all elements set to 1 is passed to the performance evaluation strategy. In other words, the search starts with a candidate module that retains all the original model weights.

[0050] Specific methods for evaluating the performance of candidate modules:

[0051] The goal of the search is to find the optimal mask, which acts as a candidate module that retains only the weights relevant to the target task and performs good classification of the target task. To achieve this, the performance evaluation strategy defines the objective function of the search as a weighted sum of the weight retention rate and the cross-entropy loss. The weight retention rate measures the number of weights retained by the candidate module. The cross-entropy loss on the target dataset measures the classification performance of the candidate module on the target task.

[0052] Specifically, when evaluating the performance of a candidate module, SeaM first constructs the candidate module because the calculation of cross-entropy loss requires running the candidate module on the target dataset. Figure 2 illustrates the construction process of a module. SeaM first multiplies the weights of the original model by the mask to remove some of the weights from the original model, thus obtaining an intermediate model. Since the output layer has three neurons, the intermediate model is still a three-class classification model. To adapt the candidate module to the number of classes in the target task, a new output head, i.e., a fully connected network layer, is appended after the intermediate model. The output head is randomly initialized in the first round of search and will be updated along with the mask in subsequent search processes. In this example, the output head has two neurons, which convert the three-class prediction result of the intermediate model into a two-class prediction result, enabling the candidate module to adapt to the target task.

[0053] After constructing the candidate modules, the cross-entropy loss L between the prediction results of the candidate modules on the target dataset and the true labels is calculated. ce Calculate using the following formula:

[0054]

[0055] Where K is the number of categories for the target task, M and H are the mask and output header, respectively, and P... i (M,H) represents the prediction of the i-th category by the candidate module constructed using M and H, t i This represents the probability of the i-th class in the one-hot encoding of the label. In the example shown in Figure 1, the one-hot encoding of a label is a binary vector of length 2, i.e., [0,1] or [1,0], where each element corresponds to one of the two classes. Only the element corresponding to the label's class in the one-hot encoding has a value of 1, while other elements have a value of 0. A lower cross-entropy loss means that the candidate module retains more weights relevant to the target task, thus achieving higher classification accuracy on the target dataset.

[0056] Weight retention rate L wr Calculate directly using the mask:

[0057]

[0058] Where L is the number of weights in the original model. A lower weight retention rate indicates that the candidate module retains fewer weights. Based on L... ce and L wr The objective function O is defined as follows:

[0059] O = L ce +α×Lwr ,

[0060] Here, α is the weighting coefficient, which is usually set to 1.0 in experiments. To minimize O, SeaM tends to search for a candidate module that retains only weights relevant to the target task, because this candidate module achieves the highest classification accuracy while retaining as few weights as possible.

[0061] The specific method for constructing the search candidate modules is as follows:

[0062] Large models can have billions of parameters, resulting in an extremely large search space. To effectively explore this vast search space, SeaM employs a gradient-based discrete space search strategy. In each round of the search, this strategy uses gradient descent to find a new candidate module with a smaller objective function value, based on the objective function value of the candidate modules from the previous round. In other words, the mask is updated using gradient descent, as shown in the following formula:

[0063]

[0064]

[0065] Where ξ is the learning rate, and M' is the updated mask, corresponding to a candidate module with a smaller objective function value.

[0066] When updating a mask using gradient descent, it's important to note that gradient descent computation requires a continuous and differentiable search space; however, the search space of the mask is discrete and non-differentiable. To ensure a continuous search space, the search strategy assigns a continuous value to each element in the mask, which can be considered as the relevance between the element's weight and the target task. Then, based on the indicator function... In the mask, elements with a relevance greater than 0 corresponding to their weights are set to 1, while other elements are set to 0. Since relevance is continuous, the search strategy can use gradient descent to update the relevance, and thus update the mask.

[0067] After satisfying the condition that the search space is continuous, another problem arises: the indicator function is not differentiable at x = 0, and its derivative is zero everywhere outside of x = 0. This problem prevents the commonly used gradient descent-based backpropagation from being directly applied to update relevance. To address this issue, a technique called the Straight-through Estimator (STE) is used to estimate the gradient of the indicator function. STE estimates the gradient of the current network layer by using the gradient of the previous neural network layer.

[0068] The gradient of the output head is passed through The calculated value is updated along with the mask. After updating the mask and output header, the search strategy passes the new mask and output header as a new candidate module to the performance evaluation strategy, and after receiving the objective function value returned by the performance evaluation strategy, begins the next round of search.

Claims

1. A modular method for neural network models, characterized in that: The input multi-class pre-trained neural network model and an image or text multi-class dataset of a target task are processed by a processing unit, which includes three parts: search space, performance evaluation strategy and search strategy. The input to the method specifically includes: a pre-trained image or text multi-classification model as the original model, which consists of several neural network layers, totaling... Each weight Classification models; and an image or text target task. Classification dataset, in which This includes data for modularization and data to be classified; the method achieves modularization of the original model through six steps, generating a model for the target task that contains only the data for modularization. Each weight The classification neural network module, in which The system further uses a neural network module to classify the data to be classified for the target task, ultimately outputting classified image or text data. Step one involves constructing the search space, which contains all candidate modules. A candidate module is constructed by selectively removing some weights from the original model using a mask. A mask is a sequence of lengths... bit vector ,in This represents the number of weights in the original model. Bit "0" indicates that the corresponding weight is deleted, while bit "1" indicates that the corresponding weight is retained. There are a total of [number missing]. There are 10 candidate masks, therefore the search space is 10 ... The process consists of several candidate modules; Step 2: Initialize the search in the search space; When the search begins, all elements of the initialized candidate mask are 1, indicating that the initialized module retains all weights, and this initialized candidate mask is passed to the performance evaluation strategy; Step 3: Evaluate the performance of the candidate module using the performance evaluation strategy; Given a candidate mask, the performance evaluation strategy first removes the weights in the model corresponding to elements with a value of 0 in the mask, while retaining the weights corresponding to elements with a value of 1. Then, a new output head is added as the output layer in the last layer of the model, thereby constructing a candidate module; The output head is a fully connected network layer used to adapt the candidate module to the number of classifications of the target task; Then, the input image or text data is used for modularization. The process begins by evaluating candidate modules using a weighted objective function. The objective function is defined as the weighted sum of two parts: the weight retention rate of the candidate module and the cross-entropy loss between the candidate module's predictions and the labels on the target dataset. The obtained objective function value is fed back to the search strategy. Step four involves searching for candidate modules using a gradient-based search method. Based on the objective function value, the gradient of each value in the mask and output header is calculated, and the mask and output header are updated using gradient descent. In each round of the search, the search strategy passes the updated mask and output header as new candidate modules to the performance evaluation strategy. Step five involves iteratively executing steps three and four until the number of iterations reaches a given value or the objective function value essentially stops changing. Only the obtained modules are retained. The weights in a classification model that are relevant to the target task can be used for... The classification target task; Step six, using the neural network module obtained in step five, classifies the input image or text data to be classified, and finally, the method outputs the classified image or text data.

2. The modularization method for a neural network model as described in claim 1, characterized in that: The method for calculating the objective function value used to evaluate the candidate module is as follows: First, a candidate module is constructed by multiplying the weights of the original model by the mask to remove some weights from the original model, thereby obtaining an intermediate model; since the output layer has There are 100 neurons, therefore the intermediate model is 1000 neurons. The classification model is followed by an intermediate model with a new output head, i.e., a fully connected network layer, to complete the construction of candidate modules. The output head is randomly initialized in the first round of search and is updated along with the mask in subsequent search processes. Output head has One neuron, which will be the intermediate model's... Classification prediction results converted The classification and prediction results enable candidate modules to adapt to the target task; After constructing the candidate modules, the cross-entropy loss between the prediction results of the candidate modules on the target dataset and the true labels is calculated. Calculate using the following formula: ,in, It is the number of categories of the target task. and These are the mask and the output header, respectively. Through and The constructed candidate modules for the first Predictions for each category, It is the first one-hot encoding of the tag. The probability of each category; here, the one-hot encoding of a label is of length . A binary vector, where each element corresponds to... One of the categories; the one-hot encoding of a label has only the element corresponding to that label category as having a value of 1, while other elements have a value of 0; the lower the cross-entropy loss, the more weights the candidate module retains that are relevant to the target task, thus achieving higher classification accuracy on the target dataset; weight retention rate It can be obtained directly through mask calculation: ,in, This refers to the number of weights in the original model; a lower weight retention rate indicates that the candidate module retains fewer weights; based on and objective function The definition is as follows: ,in, These are weighting coefficients, typically set to 1.0 in experiments; to minimize... The search process tends to find a candidate module that retains only the weights relevant to the target task, because this candidate module retains as few weights as possible while achieving the lowest cross-entropy loss.

3. The modularization method for a neural network model as described in claim 2, characterized in that: The gradient-based search method is as follows: In each round of search, based on the objective function value of the candidate modules from the previous round, the mask and output head are updated using gradient descent to find a new candidate module with a smaller objective function value for updating, as shown in the following formula: ， ,in, It's the learning rate. This is the updated mask, corresponding to a candidate module with a smaller objective function value. When applying gradient descent to search for a new mask in the search space, since the values ​​of the elements in the mask are discrete (i.e., 0 or 1), and gradient descent requires the search space to be continuous, each element in the mask is assigned a continuous value as the relevance between the weight of that element and the target task. Then, based on the indicator function... In the mask, elements with a correlation greater than 0 to their corresponding weights are set to 1, and other elements are set to 0. Once the search space is continuous, a pass-through estimator is used to estimate the gradient of the indicator function. This pass-through estimator estimates the gradient of the current network layer by using the gradient of the previous neural network layer. The gradient of the output head is obtained through... The result is calculated and updated along with the mask. After updating the mask and output header, the search strategy passes the new mask and output header as a new candidate module to the performance evaluation strategy. After obtaining the objective function value returned by the performance evaluation strategy, the next round of search begins.

Images