A neural network cross-architecture initialization method based on weight alignment and remapping
By employing robust hierarchical block decomposition and weight alignment remapping, the weight transfer problem under changing neural network architectures is solved, enabling cross-architecture initialization, improving the stability and efficiency of model training, and reducing costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- UNIV OF CHINESE ACAD OF SCI
- Filing Date
- 2026-02-04
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies struggle to effectively reuse pre-trained model weights when neural network architectures change, resulting in low training efficiency, poor stability, and the need to rely on original training data or retrain, increasing costs and time.
A cross-architecture initialization method for neural networks based on weight alignment and remapping is adopted. Through robust hierarchical block decomposition, self-normalized sequence alignment, and orthogonality-preserving weight projection, the weight parameters of the pre-trained model are transferred to the target neural network to achieve cross-architecture initialization.
It improves the stability and efficiency of model training without requiring the original training data, reduces the cost of retraining, and is suitable for scenarios such as model pruning, redesign, or neural architecture search.
Smart Images
Figure CN122133705A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to computers and deep learning, and more specifically, to a method for cross-architecture initialization of neural networks based on weight alignment and remapping. Background Technology
[0002] In recent years, with the continuous expansion of deep learning model scale and the ever-growing training datasets, the computational resources and time costs required to train deep neural networks have increased significantly. To reduce training costs and shorten model development cycles, reusing pre-trained models has become a common technique in the field of artificial intelligence. Deep learning models have demonstrated good performance in various tasks such as computer vision. Compared to training models from scratch, utilizing pre-trained models can effectively improve R&D efficiency and reduce computational costs.
[0003] In the actual deployment of deep learning models, due to performance optimization, resource constraints, or adaptation to specific hardware platforms, it is often necessary to adjust or redesign the network structure. For example, the network structure can be changed through model pruning, Neural Architecture Search (NAS), or manual design. To address these needs, various weight transfer or knowledge transfer methods have been proposed in the existing technology. For example, network morphism (Chen T., Goodfellow I., Shlens J., Net2Net: Accelerating Learning via KnowledgeTransfer[C], ICLR 2016) transfers weights from the original network to the modified network while keeping the network function mapping unchanged; in addition, knowledge distillation can be used to transfer knowledge learned in large models to target models with different structures.
[0004] However, the aforementioned existing technologies still have certain limitations when dealing with changes in network architecture. Traditional weight transfer methods are typically only applicable to scenarios where the source and target models have completely identical architectures. While network morphism methods can support limited structural adjustments, they are usually only suitable for simple structural changes and may introduce redundant parameters, leading to reduced model training efficiency. Although knowledge distillation methods have some versatility, they often rely on the original training data and require a complete training process, thus increasing the time and computational cost of model transfer. Furthermore, some heuristic initialization methods may weaken the feature information already learned in the source model. When the network structure undergoes significant changes (such as model pruning or redesign), existing technologies usually still require random initialization and retraining of the model, which can easily lead to unstable training processes and limited model performance.
[0005] Therefore, there is an urgent need for an initialization method that can effectively transfer the weights of a pre-trained model without relying on the original training data when the network architecture changes, so as to reduce the cost of model transfer and retraining and improve the stability and efficiency of model training. Summary of the Invention
[0006] The purpose of this invention is to provide a cross-architecture initialization method for neural networks based on weight alignment and remapping. This method can effectively reuse parameter information in the pre-trained model when the neural network architecture changes without relying on the original training data, thereby overcoming the problem that existing weight transfer methods are difficult to apply in scenarios such as model pruning, redesign, or neural architecture search, and providing a stable and high-quality initialization starting point for the target neural network.
[0007] To achieve the above objectives, according to one aspect of the present invention, a method for cross-architecture initialization of neural networks based on weight alignment and remapping is provided. The method includes: performing robust hierarchical block decomposition on pre-trained source and target neural networks; dividing the network architectures of the source and target neural networks into ordered source macroblock sequences and target macroblock sequences respectively, based on the graph connectivity of the network structure; performing self-normalized sequence alignment on the source and target macroblock sequences under conditions without training data to construct a block-level mapping relationship between the target and source macroblocks, and back-allocating target macroblocks that have not established a mapping relationship through the sequence alignment; performing orthogonality-preserving weight projection operation on the weight parameters in the source neural network based on the block-level mapping relationship to remap the weight parameters to the network layer in the target neural network corresponding to the target macroblock; and training the target neural network using the remapped weight parameters as initialization parameters to obtain a final model.
[0008] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein computer-readable instructions are stored therein, which, when executed by a processor, are used to implement the above-described method for cross-architecture initialization of neural networks based on weight alignment and remapping.
[0009] The technical solution described in this invention enables weight transfer across network architectures without relying on the original training data, providing the target neural network with a parameter starting point superior to random initialization. This improves the stability and efficiency of the model training process and reduces the retraining cost of the model during structural optimization and actual deployment, demonstrating good engineering practicality and application value. Attached Figure Description
[0010] Figure 1This is a schematic diagram of the overall process of a neural network cross-architecture initialization method based on weight alignment and remapping according to an embodiment of the present invention;
[0011] Figure 2 This is a schematic flowchart of robust hierarchical block decomposition according to an embodiment of the present invention;
[0012] Figure 3 This is a flowchart illustrating the self-normalized sequence alignment process according to an embodiment of the present invention;
[0013] Figure 4 This is a schematic diagram of a process for weighted width remapping using Hadamard permutation tiling according to an embodiment of the present invention; and
[0014] Figure 5 This is a schematic diagram of the process of using spectral mirroring to perform convolution kernel scale remapping according to an embodiment of the present invention. Detailed Implementation
[0015] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that those skilled in the art can readily understand the present invention.
[0016] This document only describes the parts necessary for understanding the technical content of the present invention, and the description of the remaining parts will be omitted to avoid confusion about the essence of the present invention. This should be noted. Moreover, in this process, for the sake of clarity and convenience of description, the thickness of the lines or the size of the constituent elements shown in the figures may be exaggerated.
[0017] The terminology used herein is for describing embodiments and is not intended to limit or restrict the invention. When describing a component as being "connected," "combined," or "joined" with another component, this includes not only direct connections but also indirect connections where other components are present in between. Furthermore, terms such as "comprising," "including," or "having" indicate the presence of features, numbers, steps, operations, components, or combinations thereof described in the specification, and do not exclude the existence or additional possibility of one or more other features, numbers, steps, operations, components, or combinations thereof. Additionally, terms such as "first" and "second," which may be used herein, are used only to distinguish one component from another and, unless specifically stated otherwise, do not limit the order or importance of the components. Therefore, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.
[0018] The structure and method of the present invention will be described in detail below with reference to the accompanying drawings.
[0019] Figure 1 This is a schematic diagram of the overall process of a neural network cross-architecture initialization method based on weight alignment and remapping according to an embodiment of the present invention.
[0020] In this embodiment, a method for initializing neural network weights without relying on the original training data is provided, for transferring weight parameters from a pre-trained source neural network to a target neural network with a changed architecture. (Refer to...) Figure 1 The method includes the following steps:
[0021] In step S1, a robust hierarchical block decomposition operation is performed. The network architectures of the source neural network and the target neural network are analyzed, and based on the graph connectivity of the network structure, the network architecture is divided into a stable sequence of macroblocks, where the truncated average degree is used as the segmentation threshold.
[0022] In step S2, a self-normalized sequence alignment operation is performed. Using a dynamic programming algorithm combined with a data-free cost function, the macroblock sequences of the source neural network and the target neural network are optimally matched to establish a block-level mapping relationship.
[0023] In step S3, an orthogonality-preserving weight projection operation is performed. Based on the mapping relationship established in step S2, the weight parameters in the source neural network are remapped and projected to generate the initial weights of the target neural network. Specifically, for network layers with increased width, a permutation tiling method can be used for weight remapping; for network layers with changed convolutional kernel scale, a frequency domain mapping method can be used for weight remapping.
[0024] In step S4, the initial weights generated in step S3 are used as starting parameters to train or fine-tune the target neural network, thereby obtaining the final target model.
[0025] Specifically, refer to Figure 2 The detailed process of robust hierarchical block decomposition is explained.
[0026] like Figure 2 As shown, in step S101, the network structures of the source neural network and the target neural network are represented as directed acyclic graphs (DAGs), where each layer in the neural network is represented as a node in the graph, and the connection relationship between layers is represented as a directed edge.
[0027] In step S102, the connectivity degree of each node in the directed acyclic graph is calculated, and a segmentation threshold is determined based on the connectivity degree. Specifically, after removing a certain proportion of high-connectivity nodes and low-connectivity nodes, the connectivity of the remaining nodes can be statistically analyzed to obtain the segmentation threshold. As an illustrative example, nodes with connectivity in the top 10% and bottom 10% can be removed to reduce the impact of abnormal nodes at the network input or output on the threshold calculation results.
[0028] In step S103, it is identified that the connectivity in the directed acyclic graph is greater than the segmentation threshold. The nodes serve as high-connectivity merge nodes, wherein the merge nodes may include residual connection points or feature splicing points.
[0029] In step S104, the directed acyclic graph is segmented at the identified merging nodes, dividing the neural network into several macroblocks arranged in the order of the network structure, thereby forming the source macroblock sequence corresponding to the source neural network. and the target macroblock sequence corresponding to the target neural network .
[0030] Specifically, refer to Figure 3 The detailed process of self-normalized sequence alignment is explained.
[0031] like Figure 3 As shown, in step S201, a normalized attribute vector is extracted for each layer in each macroblock. The attribute vector describes the structural features of the corresponding network layer. The operator type is one-hot encoded, and hyperparameters such as the number of channels and the kernel size are normalized to the [0,1] interval. To ensure the comparability of attribute vectors between the source and target neural networks, the normalization process uses a preset global upper bound, rather than based on the statistical values of the current network. For example, the number of channels can be divided by a preset maximum number of channels (e.g., 8192), and the kernel size can be divided by a preset maximum kernel size (e.g., 15), thus ensuring that attribute vectors from different network architectures are within the same scale space.
[0032] In step S202, the matching cost matrix is constructed. The source macroblock is calculated. With target macroblock The matching cost between them, wherein the matching cost is composed of a penalty term for operator type mismatch and a normalized L1 distance term between corresponding layer attribute vectors.
[0033] In step S203, a gap penalty term is set. The gap penalty value is adaptively determined based on the length difference between the source macroblock sequence and the target macroblock sequence. The greater the difference in sequence length, the higher the penalty value for the corresponding insertion or deletion operation.
[0034] In step S204, the Needleman-Wunsch dynamic programming algorithm is executed to find the macroblock pairing path that minimizes the total matching cost, and the optimal mapping relationship is output. Specifically, dynamic programming is used to construct the scoring matrix. ,in Indicates the source macroblock sequence before The sequence of macroblocks and the target macroblock The minimum cumulative cost for matching macroblocks is given by the following state transition equation:
[0035]
[0036] in, Indicates source macroblock With target macroblock The matching cost between them This represents the adaptive gap penalty value calculated based on the macroblock sequence length difference. After calculation, from... Begin backtracking to determine the optimal alignment path.
[0037] In step S205, for the remaining macroblocks that fail to match using the dynamic programming algorithm, a backsliding assignment is performed using center kernel alignment (CKA) similarity under data-free conditions. The calculation of the CKA similarity can refer to neural network representation similarity analysis methods. Specifically, to calculate representation similarity under conditions without real data, this embodiment constructs a set of deterministic orthogonal input probes, which are generated based on a Hadamard matrix with a fixed random seed. These input probes are input into the source macroblock and the unmatched target macroblock, respectively, to obtain the corresponding output feature maps. The CKA similarity is then calculated based on the output feature maps, thereby assigning the unmatched target macroblock to the source macroblock with the highest CKA similarity.
[0038] In step S3, an orthogonality-preserving weight projection operation is performed. This step first determines the hierarchical correspondence between the source neural network and the target neural network, and then transforms the weight parameters based on the hierarchical correspondence to generate the initial weights of the target neural network.
[0039] First, we handle the case where the number of layers is inconsistent in a pair of aligned macroblocks. When the number of layers in the target macroblock... Greater than the number of layers in the source macroblock At this time, a circular mapping strategy is used to determine the hierarchical correspondence. Specifically, for the target macroblock with index , The network layer, whose initial weights correspond to the index in the source macroblock. The network layer, where The modulo mapping method allows newly added network layers in the target macroblock to obtain weight parameters corresponding to those of existing network layers in the source macroblock.
[0040] Secondly, after determining the hierarchical correspondence, based on the source layer With target layer The mapping relationship between them is used to detect the shape differences of the weight parameters and perform the corresponding projection operation.
[0041] When the target layer increases in width relative to the source layer, the weight parameters of the source layer are expanded using Hadamard permutation tiling and orthogonalization techniques.
[0042] When the target layer is enlarged relative to the source layer in terms of kernel size, the weight parameters of the source layer are expanded using spectral mirroring techniques.
[0043] When both the width increase and the convolution kernel size increase occur simultaneously, the Hadamard permutation tiling and spectral mirroring techniques are combined to expand the weight parameters of the source layer, thereby generating the initial weights of the corresponding layer of the target neural network.
[0044] Specifically, refer to Figure 4 This section explains the process of weight width remapping using Hadamard permutation tiling. For example... Figure 4 As shown, this process is applicable when the width of the target network layer is greater than the width of the source network layer.
[0045] In step S301, the target layer width is calculated. With source layer width The proportional relationship between them is determined, and the tiling ratio c is determined based on the proportional relationship.
[0046] In step S302, the source layer weights are generated. Each copy is assigned a weighted copy, and a permutation matrix is applied to the weighted copy. This reduces the symmetry between weighted replicas. The permutation matrix P is a permutation matrix that satisfies the orthogonality preservation requirement, and its generation method can be determined based on preset rules or a fixed random seed.
[0047] In step S303, using the dimension of Hadamard matrix The orthogonal linear combination operation is performed on the weighted replicas after the permutation process, and the calculation formula is as follows:
[0048]
[0049] in, The dimension is The identity matrix, Indicates length is The target layer weights generated through the above operations maintain the variance characteristics of the original weights while ensuring that the newly generated row vectors are orthogonal to each other, which is beneficial for optimizing the gradient propagation performance during subsequent training.
[0050] Specifically, refer to Figure 5 The process of remapping the convolution kernel scale using spectral mirroring is explained. For example... Figure 5 As shown, this process is applicable when the target convolutional kernel size is larger than the source convolutional kernel size.
[0051] In step S351, the source convolution kernel is... Performing a two-dimensional discrete Fourier transform (DFT) yields the corresponding frequency domain representation. .
[0052] In step S352, a frequency domain tensor with the target convolution kernel size is constructed. and the frequency domain representation Embedded into the frequency domain tensor The central low-frequency region is initialized, while the remaining regions are initialized using a zero-fill method.
[0053] In step S353, the frequency domain representation is expressed with the Nyquist frequency as the axis of symmetry. The high-frequency components in the frequency domain are reflected and mirrored, and then filled into the frequency domain tensor. In the high-frequency region, to maintain the continuity of the spectrum.
[0054] In step S354, the frequency domain tensor after filling is... Performing a two-dimensional discrete Fourier inverse transform yields the target convolution kernel in the spatial domain. This completes the kernel scale remapping operation.
[0055] This method can be implemented by program instructions executed by a processor and stored in a computer-readable storage medium.
[0056] The scope of the claims of this invention is not limited to the specific embodiments described above. Various other embodiments, including modifications or alterations that can be made by those skilled in the art without departing from the spirit and intent of the invention as described in the claims, should also be included within the scope of the claims of this invention.
Claims
1. A cross-architecture initialization method for neural networks based on weight alignment and remapping, the method comprising: Robust hierarchical block decomposition is performed on the pre-trained source neural network and target neural network. Based on graph connectivity, the architectures of the source neural network and target neural network are divided into ordered source macroblock sequences and target macroblock sequences, respectively. The source macroblock sequence and the target macroblock sequence are self-normalized and aligned according to the no-data cost function and dynamic programming algorithm to construct the mapping relationship between the target macroblock and the source macroblock, and the target macroblock that fails to be matched by dynamic programming is backed up based on representation similarity. Based on the mapping relationship, the orthogonal preserving projection technique is used to remap the weight parameters of the source neural network to the corresponding layer of the target neural network; as well as The target neural network is trained using the remapped weight parameters as initial values to obtain the final model.
2. The method according to claim 1, wherein, Robust hierarchical block decomposition of pre-trained source and target neural networks includes: Represent the network architecture as a directed acyclic graph; The truncated average degree of all nodes in the graph is used as the degree threshold. The truncated average degree is calculated by removing the highest and lowest degree nodes by a preset percentage. The directed acyclic graph is segmented at high connectivity nodes where the degree exceeds the degree threshold, forming several stable macroblocks.
3. The method according to claim 1, wherein, The self-normalized sequence alignment of the source macroblock sequence and the target macroblock sequence based on the no-data cost function and dynamic programming algorithm includes: Extract normalized attribute vectors for each layer in each macroblock, the attribute vectors encoding layer operator types, hyperparameters, and structural properties; Define a matching cost function, which includes an operator type mismatch penalty term and a normalized distance term between layer attribute vectors; A gap penalty term is set, which is adaptively adjusted based on the difference between the source macroblock sequence length and the target macroblock sequence length. The Needleman-Wunsch algorithm is used to find the macroblock pairing path that minimizes the total matching cost based on the matching cost function and the gap penalty term.
4. The method according to claim 1, characterized in that, Based on the mapping relationship, remapping the weight parameters of the source neural network to the corresponding layer of the target neural network using orthogonality-preserving weight projection includes: When the target layer in the target neural network increases in width relative to the source layer in the mapped source neural network, the Hadamard permutation tiling method is adopted. By applying the permutation matrix to multiple copies of the source layer weights and using the Hadamard matrix for orthogonal linear combination, the target layer weights that preserve the variance are generated. When the size of the target layer convolution kernel in the target neural network is increased relative to the size of the source layer convolution kernel in the mapped source neural network, the spectral mirroring method is adopted. By performing a two-dimensional discrete Fourier transform on the source layer convolution kernel, high-frequency coefficients are embedded in the center of the frequency domain and reflected with the Nyquist frequency as the axis. After inverse transformation, the target layer convolution kernel is generated. When the length of the target macroblock being aligned is greater than the length of the source macroblock, a cyclic mapping strategy is adopted, so that each layer in the target macroblock cyclically reuses the weight parameters of the corresponding layer order in the source macroblock.
5. The method according to claim 4, wherein, The calculation formula for the Hadamard displacement tiling method is as follows: ,in For the target weight, For source weights, For Hadamard matrices, Let be the permutation matrix. For Kronecker product, It is a vector of all 1s. To widen by a factor of 1.
6. The method according to claim 1, wherein, The layer attribute vector in the dataless cost function is obtained by normalizing the numerical hyperparameters of each layer to the [0,1] interval by dividing them by a preset fixed upper limit value. The fixed upper limit value is independent of the source neural network and the target neural network.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, are used to implement the neural network cross-architecture initialization method based on weight alignment and remapping as described in claim 1.