Transform neural network model-oriented modular decomposition method

Abstract

The invention provides a modular decomposition method for a Transform neural network model, and the method comprises the steps: carrying out the modular training of a randomly initialized model, and achieving the structural decomposition of a mask after the modular training, thereby achieving the flexible on-demand multiplexing. The method specifically comprises the following steps: S1, modular training; a neural network model is given, and firstly, all parameters of the model are randomly initialized; the device comprises a neuron recognizer used for recognizing neurons related to a specific function; s2, carrying out structured decomposition; carrying out modular decomposition on the model subjected to modular training; s3, multiplexing as required; by removing neurons in a structured manner, on-demand multiplexing is realized, and the memory and calculation overhead are minimized. According to the method, the structured decomposition of the Transform structure model and the flexible on-demand multiplexing of functions can be realized, and the expansibility is high.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network model reuse in computer intelligent software engineering, and specifically to a modular decomposition method for Transformer neural network models. Background Art

[0002] In recent years, deep learning technology has flourished, and neural network models have demonstrated their unique superiority in various aspects such as image classification, object detection, speech recognition, and natural language processing. However, as neural network models become more and more complex and large, the cost of model construction (such as training time and computing resource overhead) has also increased dramatically, placing a heavy burden on model developers. Fortunately, a large number of pre-trained neural network models have been publicly released in various open source communities. For example, there are more than 80,000 public pre-trained models on the NLP open source community Hugging Face, and there are as many as 90,000 repositories marked with machine learning tags on GitHub. Inspired by software reuse, reusing existing pre-trained neural network models to reduce (or even eliminate) the high cost of model construction has been widely studied and applied.

[0003] In order to implement the pre-trained model reuse technology, the industry generally adopts two types of technologies: model selection and network-based deep transfer learning.

[0004] Model selection technology can quickly select pre-trained models that can be directly reused, while network-based deep transfer learning technology can complete the training of new models for target tasks with a small amount of time cost and labeled data of the target task. However, these two types of technologies focus more on whether the performance of the model after reuse can meet the requirements of the target task, ignoring the reuse cost of the model, including the training cost and inference cost of the model on the target task data.

[0005] So in order to solve these problems, there are already immature technologies in the industry:

[0006] Prior art 1, for N-classification CNN models (Binhang Qi, Hailong Sun, Hongyu Zhang, and Xiang Gao. 2024. Reusing Convolutional Neural Network Models through Modularization and Composition. ACM Trans. Softw. Eng. Methodol. 33, 3, Article 70 (mar 2024), 39 pages. https: / / doi.org / 10.1145 / 3632744), this technology proposes a compressed convolutional neural network modularization algorithm and applies it to the quality assurance of CNN models. For each of the N categories, the algorithm searches for the convolution kernel used to extract the features of the category in the pre-trained model and removes other irrelevant convolution kernels to form a module.

[0007] The first existing technology can effectively realize the modularization of CNN models. At the same time, for a category, since the generated module only retains some convolution kernels related to the category in the pre-trained model, the number of parameters of the module is smaller than that of the pre-trained model. Compared with other modularization technologies, the combined model composed of modules generated by this technology has smaller parameters and lower inference overhead. However, this technology still has the following shortcomings: First, the modularization method of this technology belongs to post-training modularization, which decomposes the trained model, but because the original model does not take the modularization feature into account during the training process, the different convolution kernels of the model are highly coupled, which greatly limits the effect of modularization; secondly, this technology takes the substructure of the convolution kernel as the modularization target, which makes this technology only applicable to CNN and cannot be extended to other models.

[0008] Prior art 2, CN116340808A Disclosed is a modularization method for a neural network model. The method processes an input multi-classification pre-trained neural network model and an image or text multi-classification data set of a target task through a processing unit. The processing unit includes three parts: a search space, a performance evaluation strategy and a search strategy. The input of the method specifically includes: a pre-trained image or text multi-classification model as an original model, the original model is composed of a plurality of neural network layers, and is an N-classification model with a total of L weights; and an image or text K-classification data set of a target task, wherein K is less than N, and includes data for modularization and data to be classified. The method realizes modularization for the original model through six steps, generates a K-classification neural network module for the target task and only includes J weights, wherein J is less than L, and further uses the neural network module to classify the data to be classified of the target task, and finally outputs the classified image or text data.

[0009] The second existing technology can effectively realize the modularization of CNN model and can be extended to other models. At the same time, for a category, since the generated module only retains part of the weights related to the category in the pre-trained model, the number of parameters of the module is less than that of the pre-trained model. Compared with other modularization technologies, this technology has better scalability because it sets the target as "weight", a unit that is commonly present in all models. However, this technology still has the following shortcomings: First, the modularization method of this technology still belongs to post-training modularization. It decomposes the trained model, but because the original model does not consider the modularization feature during the training process, the different convolution kernels of the model are highly coupled, which greatly limits the effect of modularization; secondly, although this technology takes weight as the target and achieves a sparse weight matrix by resetting a large number of weights to 0, it cannot structurally remove these weights. At present, the inference acceleration of the model on the sparse weight matrix depends on a specific sparse acceleration engine, which is difficult to achieve acceleration on highly parallel devices such as GPUs, and then cannot be applied in actual production environments.

[0010] The technical solution of the existing technology three (Binhang Qi, Hailong Sun, Hongyu Zhang, Ruobing Zhao, and Xiang Gao. 2024. Modularizing while Training: A New Paradigm for Modularizing DNN Models. In Proceedings of the IEEE / ACM 46th International Conference on Software Engineering (ICSE'24). Association for Computing Machinery, New York, NY, USA, Article 31, 12 pages. https: / / doi.org / 10.1145 / 3597503.3608135) is aimed at the N-classification CNN model. This technology proposes a convolutional neural network modularization algorithm during training and applies it to the modular decomposition and on-demand reuse of the CNN model. During the training process, for each of the N categories, the algorithm adds a convolution kernel identifier to each convolutional neural network layer to identify which convolution kernels are related to the target category and define the cohesion and coupling losses. Finally, the model is trained to have high cohesion and low coupling features, and the module is obtained by structurally removing the convolution kernels that are not related to the corresponding category.

[0011] The third existing technology can effectively realize the modularization of CNN model during training. At the same time, for a category, since the generated module only retains part of the convolution kernels related to the category in the pre-trained model, the number of parameters of the module is smaller than that of the pre-trained model. Compared with other modularization technologies, since this technology introduces modularization restrictions in the training stage of the model, the combined model composed of modules generated by this technology has smaller parameters and lower inference overhead. However, this technology still has the following shortcomings: First, this technology takes the convolution kernel substructure as the modularization target, which makes this technology only applicable to CNN and cannot be extended to other models. Secondly, the optimization function used by this technology is the weighted sum of cohesion loss, coupling loss and cross entropy loss, which has some difficulties in multi-objective optimization, which is specifically reflected in the decrease in accuracy and the room for further reduction of the convolution kernel retention rate. Summary of the invention

[0012] The present invention aims to solve the problem of on-demand reuse of deep neural network models and aims to provide a modular decomposition method for neural network models with Transformer structure, which supports modular decomposition when training the model at the neuron granularity.

[0013] Specifically, the main issues to be addressed include:

[0014] (1) The existing modularization method during training can only be implemented for CNN models and cannot be applied to the most advanced model structure Transformer. The present invention determines the granularity of modularization on neurons and designs a neuron identifier to determine the corresponding function of the neuron, thereby realizing modularization on the Transformer model;

[0015] (2) We designed and implemented a neuron-based structured decomposition method. Considering that a neuron is an abstract concept in the DNN model, which represents the output result of a row of the weight matrix of the corresponding layer, we designed a method for structured removal of the weight matrix in the attention mechanism calculation and the fully connected layer calculation.

[0016] (3) We designed a loss function suitable for optimizing larger-scale models, integrating methods such as cohesion, coupling, and contrastive learning. We achieved significant improvements in indicators such as neuron retention rate, accuracy, cohesion, and coupling in modular training.

[0017] The specific technical solution is:

[0018] A modular decomposition method for Transformer neural network models performs modular training on a randomly initialized model and applies masks to achieve structured decomposition after modular training, thereby achieving flexible on-demand reuse.

[0019] S1. Modular training.

[0020] Given a neural network model, all parameters of the model are randomly initialized. A neuron identifier is included to identify neurons related to specific functions.

[0021] Specifically, the neuron identifier contains several mask generators, which are composed of multiple linear layers and are used to learn the relationship between input features and corresponding categories. Each mask generator corresponds to a linear layer in the neural network model. For each mask generator, its input is exactly the same as the input of the corresponding linear layer. During the forward propagation of the model, the mask generator generates a mask with the same dimension as the number of neurons in the corresponding linear layer, where each value represents whether the corresponding neuron is related to a category. The mask is used to calculate cohesion and coupling losses, and is finally optimized in the contrastive learning loss. After the forward propagation of the model is completed, the cross entropy loss is calculated for the final classification result. The objective function is the weighted sum of the cross entropy loss, cohesion loss, and coupling loss.

[0022] S2. Structured decomposition.

[0023] For the modularly trained model, it will be modularly decomposed.

[0024] Specifically, based on the mask generated by the mask generator, determine which neurons are related to the current target classification. Considering that the neuron actually corresponds to the output of a row of the weight matrix in the linear layer, remove the corresponding row in the weight matrix according to the mask. In addition, considering that the weight matrix of the previous layer will also be partially removed, remove the corresponding row or column of the current layer according to the mask of the previous layer; after decomposition, a decomposed module is obtained, and a classification head of the target task is added for simple fine-tuning to obtain the final module for on-demand reuse.

[0025] S3, on-demand reuse

[0026] By structurally removing neurons, on-demand reuse is achieved, minimizing memory and computational overhead.

[0027] Specific:

[0028] S1. Modular training

[0029] Training a DNN model with modularity involves three steps: (1) identifying relevant neurons, (2) evaluating modularity and accuracy metrics, and (3) optimizing modularity and classification accuracy performance.

[0030] (1) Identify relevant neurons. A neuron identifier is used to identify neurons related to the target category in the linear layer and attention layer in the Transformer structure. For a model, its neuron identifier I DNN Contains several mask generators G, each of which is a fully connected neural network consisting of two linear layers and attached to a linear layer L of the model DNN Generate mask m and apply m to linear layer L DNN The output h of The process is expressed as follows using formula 1:

[0031]

[0032] In formula 2, the input x is average pooled to generate a downsampled feature x pooled and used as the input of the mask generator G. In Formula 3, G encodes x pooled And generate mask m, indicating that in L DNN Which neuron corresponds to the current input sample.

[0033] In the ViT model, the feature dimension of the input x is (B, N, D), where B represents the batch size of the input data, N represents the number of patches in the ViT model, and D represents the feature size of each patch. During the forward propagation process, the training data is input into the model and the neuron recognizer. DNN Output a feature h, the dimension is (B, N, D), the neuron recognizer is for each L DNN Generate a mask m whose dimensions (B, 1, D) correspond to the output of each neuron. In formula 4, for the output of the neuron, the mask m is broadcasted in the patch dimension and multiplied to the masked output. G maps the value of m to the interval [0, 1) by adding the activation function Tanh and ReLU. 0 means that the corresponding neuron is irrelevant to the input sample.

[0034] (2) Evaluate modularity and classification performance. The concepts of cohesion and coupling are introduced to evaluate the performance of modularity. Combining the concepts of cohesion and coupling, the positive sample is constructed as the overlap of neurons activated by data samples belonging to the same category, that is, cohesion. Similarly, the negative sample is the overlap of neurons activated by data samples belonging to different categories, that is, coupling.

[0035] Specifically, a dataset containing n categories, for category c i , belongs to category c i n i The samples are represented as Correspondingly, the set of neurons activated by each sample is represented as For category c i , module mN i Contains neurons used by all samples For category c i , module mN i The cohesion is expressed as:

[0036]

[0037] The coupling degree is expressed as:

[0038] Coupling(m i , m j )=Overlap(mN i , mN j ) 6

[0039] The Overlap index is derived from the JaccardIndex. For two sets A and B, their JaccardIndex is:

[0040]

[0041] (3) Modular training and optimization. Use cosine similarity instead of overlap to obtain an optimized cohesion and coupling loss:

[0042]

[0043] n i Indicates that in each batch, it belongs to category c i The number of samples. Indicates that it belongs to category c i The mask of the jth sample. Cos(a, b) represents the cosine similarity of two tensors a and b. The temperature parameter τ and the exponential function in contrastive learning are introduced to scale the distribution of samples in the feature space; the loss of cohesion is expressed as:

[0044]

[0045] The loss of coupling is expressed as:

[0046]

[0047] After obtaining the cohesion and coupling losses, the contrast loss is defined as follows;

[0048]

[0049] The target loss function consists of two parts: cross entropy loss and contrast loss:

[0050]

[0051] Among them, α is the weight of contrast loss. Based on the above loss function, the model is optimized by the gradient descent algorithm, so as to train the corresponding part of the neurons to recognize samples of specific categories.

[0052] S2. Structured decomposition

[0053] The modular model is decomposed into multiple modules according to the mask generated by the mask generator. Specifically, it is first determined which part the neuron corresponds to in the model, and then the model is decomposed according to the mask.

[0054] (1) Determine the neuronal structure.

[0055] By removing a corresponding row in the weight matrix, the purpose of removing neurons is achieved, while reducing the corresponding computational overhead.

[0056] (2) Decompose the linear layer by partial weight matrix.

[0057] To decompose the modular model into modules, we first generate masks for each sample. i , n i belongs to c i The samples are represented as Their corresponding masks are represented as definition:

[0058]

[0059] For category c i , in order to select neurons that are valid for all relevant samples and filter out noise, the mask of the module

[0060]

[0061] For the fully connected layer, for two consecutive layers A and B, B accepts the output of A. Then for the weight matrix B, it needs to remove the weight rows where the mask B is 0 in the output dimension. At the same time, since B accepts the output of A, the weight matrix B also needs to remove the weight columns where the mask A is 0 in the input dimension; for the attention layer, there are three weight matrices Query, Key and Value, all of which input the same features and obtain the corresponding Q, K and V. Then, the attention score is calculated as:

[0062]

[0063] Where Q = [q1, q2, ..., q n ],K=[k1,k2,...,k n ], For Query and Key, take the intersection of the two masks, that is, m QK =m Q ∩m K , only the common parts of the two are retained. The output dimension of the weight matrix V is m QK to remove.

[0064] When the linear layers at both ends of the residual connection are structured and removed, the output and input dimensions change. An on-demand filling and patching solution is adopted to fill the input brought by the residual connection according to the input dimension of the back layer of the residual connection and remove the redundant part.

[0065] Beneficial effects brought by the technical solution of the present invention:

[0066] 1. The present invention provides a modular decomposition method for a neural network model of a Transformer structure model, which can realize the structured decomposition of the Transformer structure model and the flexible on-demand reuse of functions. Compared with the unstructured method of the second technique, the neurons of the model can be physically removed using the method of the present invention, thereby reducing storage and reasoning overhead.

[0067] 2. The present invention selects neurons as the modular granularity and provides a complete neuron structured decomposition method. Compared with techniques 1 and 3, this granularity is widely present in all models and is highly scalable.

[0068] 3. The module obtained by the neuron structured decomposition method designed by the present invention on the ViT model retains at least 18.8% of the neurons of the original model (corresponding to 12.04% of the weight). At the same time, its classification accuracy is comparable to that of other methods, with almost no decrease.

[0069] 4. The loss function designed by the present invention greatly reduces the retention rate of neurons or convolution kernels. The ablation test on the CNN model (using the same CNN model as the third technique instead of the ViT model of the present invention, only replacing the loss function) shows that the average convolution kernel retention rate of the method of the present invention is 12.8%, and the average convolution kernel retention rate of the method of the third technique is 17.55%. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is a flow chart of the present invention;

[0071] Figure 2 This is a schematic diagram of removing neurons in the attention and fully connected layers of the present invention;

[0072] Figure 3 Schematic diagram of dimensional mismatch and repair of residual connection of the present invention;

[0073] Figure 4It is a schematic diagram of the module on-demand multiplexing of the present invention. DETAILED DESCRIPTION

[0074] The invention designs and implements a modular method NeMo for structured training of neuron granularity. The core workflow framework of the invention is as follows: Figure 1 As shown in Figure 3, NeMo mainly consists of two processes: modular training for a randomly initialized model and applying masks to achieve structured decomposition after modular training, thereby achieving flexible on-demand reuse.

[0075] S1. Modular training.

[0076] Given a neural network model, NeMo first randomly initializes all parameters of the model. NeMo contains a neuron identifier to identify neurons related to specific functions. Specifically, NeMo's neuron identifier contains several mask generators, which are composed of multiple linear layers to learn the relationship between input features and corresponding categories. Each mask generator corresponds to a linear layer in the neural network model. For each mask generator, its input is exactly the same as the input of the corresponding linear layer. During the forward propagation of the model, the mask generator generates a mask with the same dimension as the number of neurons in the corresponding linear layer, where each value represents whether the corresponding neuron is related to a category (0 means irrelevant, 1 means relevant). The mask is used to calculate the cohesion and coupling losses, and finally optimized in the contrastive learning loss. After the forward propagation of the model is completed, the cross entropy loss is calculated for the final classification result. The objective function is the weighted sum of the cross entropy loss, cohesion loss and coupling loss. As the training process proceeds and the back-propagation optimization loss is optimized, the model is finally trained as a modular model with high accuracy, high cohesion and low coupling.

[0077] S2. Structured decomposition.

[0078] For the modularly trained model, NeMo will perform modular decomposition on it. Specifically, NeMo determines which neurons are related to the current target classification based on the mask generated by the mask generator. Considering that the neuron actually corresponds to the output of a row of the weight matrix in the linear layer, NeMo will remove a row in the corresponding weight matrix according to the mask. In addition, considering that the weight matrix of the previous layer will also be partially removed, NeMo will also remove the corresponding rows or columns of the current layer according to the mask of the previous layer, depending on the connection method between different layers, such as the attention layer or the fully connected layer. After decomposition, NeMo will obtain a decomposed module, and add a classification head of the target task for simple fine-tuning to obtain the final module for on-demand reuse.

[0079] The following are the details of the above steps:

[0080] S1. Modular training

[0081] Training a DNN model with modularity mainly involves three steps: (1) identifying relevant neurons, (2) evaluating modularity and accuracy indicators, and (3) optimizing modularity and classification accuracy performance. Finally, the modularly trained model can be decomposed into smaller modules and flexibly reused on demand.

[0082] (1) Identify relevant neurons. Identifying neurons related to a certain category is the most important part of modular training. The present invention proposes a neuron identifier in NeMo, which is used to identify neurons related to the target category in the linear layer and attention layer in the Transformer structure. For a model, its neuron identifier I DNN Contains several mask generators G, each of which is a fully connected neural network consisting of two linear layers and attached to a linear layer L of the model DN N. Generate mask m and apply m to linear layer L DNN The output h of The process is expressed as follows using formula 1:

[0083] In order to reduce the additional training cost, the present invention performs average pooling on the input x in formula 2 to generate a downsampled feature x pooled and used as the input of the mask generator G. In Formula 3, G encodes x pooled And generate mask m, indicating that in L DNN Which neuron in corresponds to the current input sample. Specifically, in the ViT model, the feature dimension of the input x is (B, N, D), where B represents the batch size of the input data, N represents the number of patches in the ViT model, and D represents the feature size of each patch. For example, for an input image of size 224×224 and dividing it into 14×14 patches, the size of each patch is 16×16. Therefore, N=196+1 (an additional patch is used to represent the global feature), and D=256. During the forward propagation process, the training data is input into the model and the neuron identifier. L DNN It will output a feature h with dimensions (B, N, D). The neuron recognizer will generate a feature h for each L DNN Generate a mask, whose dimensions (B, 1, D) correspond to the output of each neuron. In formula 4, for the output of the neuron, the mask m is broadcasted in the patch dimension and multiplied to the masked output. G maps the value of m to the interval [0, 1) by adding the activation function Tanh and ReLU. 0 means that the corresponding neuron is irrelevant to the input sample.

[0084] (2) Evaluate modularity and classification performance. In modular training, the present invention introduces the concepts of cohesion and coupling, and uses them to evaluate the performance of modularity. Among them, cohesion quantifies the degree of overlap of neurons used by samples with the same label; coupling quantifies the degree of overlap of neurons used by samples with different labels. In order to optimize these two indicators, the contrastive learning method mainly focuses on distinguishing positive samples from negative samples. The optimization goal is to make the distance between positive samples in the feature space closer and the distance between negative samples farther. The existing method of constructing positive samples is mainly to perform data enhancement on a single image to obtain a series of positive samples. Combining the concepts of cohesion and coupling, the present invention constructs positive samples as the overlap of activation neurons of data samples belonging to the same class, that is, cohesion. Similarly, negative samples are the overlap of activation neurons of data samples belonging to different classes, that is, coupling. Specifically, for a data set containing n categories, for category c i , belongs to category c i n i The samples are represented as Correspondingly, the set of neurons activated by each sample can be expressed as For category c i , module mN i Contains neurons used by all samples For category c i , module mN i The cohesion can be expressed as:

[0085]

[0086] The coupling degree can be expressed as:

[0087] Coupling(m i , m j )=Overlap(mN i , mN j ) 6

[0088] Among them, the Overlap index is derived from JaccardIndex, which is widely used to calculate the overlap between two modules. For two sets A and B, their JaccardIndex is:

[0089]

[0090] (3) Modular training and optimization. Since the calculation of the above cohesion and coupling is not differentiable, in order to continuously optimize the modular performance during the training process, the present invention uses cosine similarity instead of Overlap, thereby obtaining an optimized cohesion and coupling loss.

[0091]

[0092] Here, n i Indicates that in each batch, it belongs to category c i The number of samples. Indicates that it belongs to category c i The mask of the jth sample. Cos(a, b) represents the cosine similarity of two tensors a and b. The present invention introduces the temperature parameter τ and the exponential function in contrastive learning to scale the distribution of samples in the feature space, so as to better focus on some samples that are difficult to optimize at the edge, and achieve the effect of lower retention rate and higher accuracy. Finally, the loss of cohesion is expressed as:

[0093]

[0094] The loss of coupling is expressed as:

[0095]

[0096] After obtaining the cohesion and coupling losses, the present invention defines the contrast loss as follows:

[0097]

[0098] In addition, in order to optimize the classification accuracy of the model, the present invention uses cross entropy loss. The target loss function consists of two parts: cross entropy loss and contrast loss:

[0099]

[0100] Among them, α is the weight of the contrast loss. Based on the above loss function, the model can be optimized by the gradient descent algorithm, so as to train the corresponding part of the neurons to recognize samples of specific categories.

[0101] S2. Structured decomposition

[0102] In order to obtain a module corresponding to a class sample, after modular training, the present invention decomposes the modular model into multiple modules according to the mask generated by the mask generator. Specifically, the present invention first determines which part the neuron corresponds to in the model, and then decomposes the model according to the mask.

[0103] (1) Determine the neuron structure. Neurons are widely present in deep neural networks (DNNs). However, they are more of a descriptive concept rather than a specific structure. In the linear layer, a neuron represents the final value of the input after passing through the weight matrix, adding the bias, and finally applying the activation function. Therefore, unlike the convolution kernel, the neuron is not a specific structure. However, the present invention can achieve the purpose of removing neurons by removing a corresponding row in the weight matrix, while reducing the corresponding computational overhead.

[0104] (2) Decompose the linear layer by partial weight matrices. In order to decompose the modular model into modules, NeMo first generates a mask for each sample. For category c i , n i belongs to c i The sample can be represented as Their corresponding masks can be expressed as The present invention defines:

[0105]

[0106] For category c i , in order to select neurons that are valid for all relevant samples and filter out noise, the mask of the module

[0107]

[0108] For fully connected layers, weight removal is very simple. For two consecutive layers A and B, B receives the output of A. Then for the weight matrix B, it needs to remove the weight rows where the mask B is 0 in the output dimension. At the same time, since B receives the output of A, the weight matrix B also needs to remove the weight columns where the mask A is 0 in the input dimension, such as Figure 2 As shown in the blue part. For the attention layer, the situation is much more complicated. In the calculation process of the attention mechanism, there are three weight matrices Query, Key and Value. All three input the same features and get the corresponding Q, K and V. Then, the attention score is calculated as:

[0109]

[0110] Where Q = [q1, q2, ..., q n ],K=[k1,k2,...,k n ], When i After being masked by 0, the corresponding q i ×k i The value is 0, which means k i Therefore, for Query and Key, the intersection of the two masks should be taken, that is, mQK =m Q ∩m K , only the common parts of the two are retained. For Value, the situation is similar to the fully connected layer. The output dimension of the weight matrix V can be represented by m v to remove, but since it accepts the result of multiplying Q and K, its input dimension should be m QK To remove, see Figure 2 Yellow part.

[0111] In addition, since residual connections are widely present in neural networks, they jump between two different blocks, which brings additional problems to modularization. Specifically, when the linear layers at both ends of the residual connection are structurally removed, the output and input dimensions change. Although the present invention has previously processed the continuous layers to ensure that the dimensions between the continuous layers are correctly matched after removal, since the residual connection usually spans multiple layers or blocks, the dimensions at both ends are not matched. Figure 3 As shown in (b) in .

[0112] Existing solutions include Figure 3 (c) Adding additional neurons to patch, and Figure 3 (d) is the output of the previous and next layers filled with 0. However, these methods all have additional overhead. Considering that when part of the weight of the next layer is removed, its input only needs to consider the retained part, the present invention proposes Figure 3 (e) is the on-demand filling and patching solution, which fills the input brought by the residual connection only according to the input dimension of the back layer of the residual connection and removes the redundant part. Obviously, this is the most efficient solution.

[0113] S3, on-demand reuse

[0114] NeMo achieves efficient and flexible on-demand reuse by structurally removing neurons, minimizing memory and computational overhead. For example, when a task requires identifying "cat" or "dog", NeMo can decompose a 2-classification module from a 10-classification model and apply it, reducing the cost of reasoning by removing additional irrelevant neurons.

[0115] like Figure 4 As shown in the figure, after modular training, developers can share the mask file on the model sharing platform. Users can get the corresponding module by applying the mask and obtain a high-accuracy module by simple fine-tuning.

[0116] The present invention provides a neuron-sized modular training method and a detailed solution for the ViT model. For other models with similar structures, such as DeiT, the modular training and structured decomposition process can be implemented according to this method, and excellent results can be achieved.

[0117] The "neuron identifier" used in the present invention includes multiple "mask generators", and the mask generator is composed of multiple linear layers. In technical implementation, the linear layer in the mask generator can be replaced by a convolutional layer or other network layer, and the purpose is to output the mask to identify the activated neuron.

Claims

1. A modular decomposition method for Transformer neural network model, characterized in that: Modular training of a randomly initialized model and structured decomposition by applying masks after modular training, thus enabling flexible on-demand reuse; The specific steps include: S1, modular training; Given a neural network model, all parameters of the model are first randomly initialized; a neuron identifier is included to identify neurons related to specific functions; S2, structured decomposition; For the modularly trained model, it will be modularly decomposed; S3, on-demand reuse; By structurally removing neurons, on-demand reuse is achieved, minimizing memory and computational overhead.

2. A modular decomposition method for a Transformer neural network model according to claim 1, characterized in that: S1. Specific methods of modular training: The neuron identifier contains several mask generators, which are composed of multiple linear layers and are used to learn the relationship between input features and corresponding categories. Each mask generator corresponds to a linear layer in the neural network model. For each mask generator, its input is exactly the same as the input of the corresponding linear layer. During the forward propagation of the model, the mask generator generates a mask with the same dimension as the number of neurons in the corresponding linear layer, where each value represents whether the corresponding neuron is related to a category. The mask is used to calculate the cohesion and coupling losses, and is finally optimized in the contrastive learning loss. After the forward propagation of the model is completed, the cross entropy loss is calculated for the final classification result. The objective function is a weighted sum of cross entropy loss, cohesion loss, and coupling loss.

3. A modular decomposition method for a Transformer neural network model according to claim 2, characterized in that: Training a DNN model with modularity in S1 consists of three steps: (1) identifying relevant neurons, (2) evaluating modularity and accuracy metrics, and (3) optimizing modularity and classification accuracy performance; (1) Identify relevant neurons; use a neuron identifier to identify neurons related to the target category in the linear layer and attention layer in the Transformer structure; for a model, its neuron identifier L DNN Contains several mask generators G, each of which is a fully connected neural network consisting of two linear layers and attached to a linear layer L of the model DNN ; Generate mask m and apply m to linear layer L DNN The output h of The process is expressed as follows using formula 1: In formula 2, the input x is average pooled to generate a downsampled feature x pooled And used as input to the mask generator G; In Formula 3, G encodes x pooled And generate mask m, indicating that in L DNN Which neuron corresponds to the current input sample? In the ViT model, the feature dimension of the input x is (B, N, D), where B represents the batch size of the input data, NN represents the number of patches in the ViT model, and D represents the feature size of each patch; during the forward propagation process, the training data is input into the model and the neuron recognizer; L DNN Output a feature h, the dimension is (B, N, D), the neuron recognizer is for each L DNN Generate a mask m, whose dimensions (B, 1, D) correspond to the output of each neuron; in formula 4, for the output of the neuron, the mask m is broadcasted in the dimension of the patch and multiplied to the masked output; G maps the value of mn to the interval [0, 1) by adding the activation function Tanh and ReLU; 0 represents that the corresponding neuron is irrelevant to the input sample; (2) Evaluate modularity and classification performance; introduce the concepts of cohesion and coupling, and use them to evaluate the performance of modularity; combine the concepts of cohesion and coupling, and construct positive samples as the overlap of neurons activated by data samples belonging to the same category, that is, cohesion; similarly, negative samples are the overlap of neurons activated by data samples belonging to different categories, that is, coupling; Specifically, a dataset containing n categories, for category c i , belongs to category c i n i The samples are represented as Correspondingly, the set of neurons activated by each sample is represented as For category c i , module mN i Contains neurons used by all samples For category c i , module mN i The cohesion is expressed as: The coupling degree is expressed as: Coupling(m i ,m j )=Overlap(mN i ,mN j ) 6 The Overlap index is derived from the JaccardIndex. For two sets A and B, their JaccardIndex is: (3) Modular training and optimization: Use cosine similarity instead of overlap to obtain an optimized cohesion and coupling loss: n i Indicates that in each batch, it belongs to category c i The number of samples; Indicates that it belongs to category c i The mask of the jth sample of ; Cos(a, b) represents the cosine similarity of two tensors a and b; the temperature parameter τ in contrastive learning and the exponential function are introduced to scale the distribution of samples in the feature space; the loss of cohesion is expressed as: The loss of coupling is expressed as: After obtaining the cohesion and coupling losses, the contrast loss is defined as follows; The target loss function consists of two parts: cross entropy loss and contrast loss: Among them, α is the weight of the contrast loss; based on the above loss function, the model is optimized by the gradient descent algorithm, so as to train the corresponding part of the neurons to recognize samples of specific categories.

4. A modular decomposition method for a Transformer neural network model according to claim 1, characterized in that: S2. The specific method of structured decomposition is to determine which neurons are related to the current target classification based on the mask generated by the mask generator; considering that the neuron actually corresponds to the output of a row of the weight matrix in the linear layer, remove the corresponding row in the weight matrix according to the mask; in addition, considering that the weight matrix of the previous layer will also be partially removed, remove the corresponding row or column of the current layer according to the mask of the previous layer; after decomposition, a decomposed module is obtained, and a classification head of the target task is added for simple fine-tuning to obtain the final module for on-demand reuse.

5. A modular decomposition method for a Transformer neural network model according to claim 4, characterized in that: In S2, the modular model is decomposed into multiple modules according to the mask generated by the mask generator; specifically, it is first determined which part the neuron corresponds to in the model, and then the model is decomposed according to the mask; (1) Determine the neuronal structure; By removing a corresponding row in the weight matrix, the purpose of removing neurons is achieved, while reducing the corresponding computational overhead; (2) Decomposing the linear layer by partial weight matrices; To decompose the modular model into modules, we first generate a mask for each sample; for category c i , n i belongs to c i The samples are represented as Their corresponding masks are represented as definition: For category c i , in order to select neurons that are valid for all relevant samples and filter out noise, the mask of the module For the fully connected layer, for two consecutive layers A and B, B accepts the output of A; then for the weight matrix B, it needs to remove the weight rows where the mask B is 0 in the output dimension; at the same time, since B accepts the output of A, the weight matrix B also needs to remove the weight columns where the mask A is 0 in the input dimension; for the attention layer, there are three weight matrices Query, Key and Value, all of which input the same features and obtain the corresponding Q, K and V; then, the attention score is calculated as: Where Q = [q1, q2, ..., q n ],K=[k1,k2,...,k n ], For Query and Key, take the intersection of the two masks, that is, m QK =m Q ∩m K , only the common part of the two is retained; the output latitude of the weight matrix V is composed of m QK to remove; When the linear layers at both ends of the residual connection are structured and removed, the output and input dimensions change. An on-demand filling and patching solution is adopted to fill the input brought by the residual connection according to the input dimension of the back layer of the residual connection and remove the redundant part.

Images