A maximum update parameterization method for wide and deep expansion of a large model based on spectral conditions
By adopting the maximum update parameterization method based on spectral conditions for large-scale model width and depth expansion, the problems of unstable feature learning and difficulty in hyperparameter transfer during the width and depth expansion of large-scale deep neural network models are solved, thereby achieving stability and performance improvement in model training and reducing the cost of parameter tuning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- RENMIN UNIVERSITY OF CHINA
- Filing Date
- 2026-02-27
- Publication Date
- 2026-06-09
AI Technical Summary
Existing technologies suffer from unstable feature learning and difficulty in hyperparameter transfer during the widening and depth expansion of large-scale deep neural network models, leading to problems such as unstable training and high parameter tuning costs.
We employ a maximum update parameterization method based on spectral conditions for large model width and depth expansion. We determine the basic hyperparameters through Bayesian hyperparameter optimization and train the network using the Muon-Kimi optimizer. We apply uniform spectral constraints to control the weight parameters and their update amounts, including the parameterization of weight initialization variance, layer scaling factor, and learning rate.
It achieves stable feature learning behavior across models of different sizes, reduces the difficulty of hyperparameter transfer, improves the stability and performance of model training, and reduces engineering costs.
Smart Images
Figure CN122174927A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer science, and more specifically, to a maximum update parameterization method for large model width-depth extension based on spectral conditions. Background Technology
[0002] The rapid development of artificial intelligence technology has driven the widespread application of deep neural network models in fields such as natural language processing, computer vision, and multimodal understanding. In recent years, generative foundational models have continued to grow in scale, with their parameter size increasing and network structures showing a trend of simultaneously increasing width and depth. Expanding the model size can significantly improve the model's expressive power and task performance, but at the same time, it also places higher demands on the stability and engineering efficiency of model training.
[0003] In the training of deep neural networks, hyperparameters such as the initialization method of model parameters, learning rate settings, and parameter update magnitude have a decisive impact on feature learning behavior. When the model size changes, if a fixed default standard parameterization method (SP) is still used, it often leads to numerical explosion or vanishing of features within the network, resulting in problems such as training instability, slow convergence, or even failure to converge. In addition, the optimal hyperparameters differ significantly between models of different sizes, requiring extensive hyperparameter tuning every time the model size is increased, resulting in extremely high training costs.
[0004] Maximum Update Parameterization (µP) [1] is a method that reparameterizes hyperparameters as the model size changes, so that models of different sizes can maintain consistent feature learning behavior during training and can achieve direct transfer of hyperparameters between models of different sizes. Existing µP theory has achieved good results in scenarios where only the network width is expanded, but for practical application scenarios where both width and depth are expanded at the same time, there is still a lack of a unified, concise and easy-to-implement theoretical framework and engineering method.
[0005] Current research on µP (micropixel density) scaling, which involves simultaneous expansion of both width and depth, typically relies on complex mathematical analysis tools, and its conclusions are often strongly correlated with specific network structures or optimizers. For example, different works have proposed different parameterizations for stochastic gradient descent (SGD), AdamW, or matrix preconditioning optimizers under different network structures, lacking a unified descriptive approach. Furthermore, these methods are complex to derive and implement, making it difficult to generalize to new optimizers or network structures in engineering practice. In addition, most existing schemes fail to characterize the propagation of weight parameters and their updates within the network from a unified scale constraint perspective, making it difficult to systematically explain the intrinsic relationships between different methods and hindering the formation of universal design principles. Summary of the Invention
[0006] The purpose of this disclosure is to provide a maximum update parameterization method for large-scale model width and depth expansion based on spectral conditions, aiming to solve the problems of unstable feature learning and difficult hyperparameter transfer during the training of large models in the process of width and depth expansion. Specifically, the unstable feature learning phenomenon for large models of text or images is mainly manifested in feature explosion, feature vanishing, or inter-layer signal amplitude imbalance as the model size increases due to the neural network structure of image and text processing. Similarly, the difficulty in hyperparameter transfer stems from the sheer size of the neural network structure of image and text processing, leading to the need for repeated manual adjustment of learning rate, weight initialization scale, and inter-layer scaling coefficients between models of different sizes. Existing technologies typically only design hyperparameter configuration methods for a single model and optimizer, failing to establish a unified correspondence between the model parameter update amplitude and network width and depth from the perspective of spectral scale constraints. This results in inconsistent feature learning behavior when the model size changes, thus simultaneously causing training instability and high parameter tuning costs.
[0007] In general, a maximum update parameterization method based on spectral conditions for large model width and depth expansion is provided for parameter updates of neural network models used to address the problems of unstable feature learning and difficulty in hyperparameter transfer that arise as the width and depth scale expands during the training of large language / image models. Step 1: The base model is a neural network model, including an input layer, multiple hidden residual blocks, and an output layer. The network width n_base and network depth L_base of the base model are determined. Then, Bayesian hyperparameter optimization is used to determine the optimal base hyperparameters, such as the learning rate, inter-layer scaling coefficients, and weight initialization scale. Specifically, a vector of base hyperparameters to be optimized is constructed on the base model. Based on the training dataset and validation dataset Define the objective function of the base model on the validation set as follows: A Bayesian hyperparameter optimization method is employed to construct a hyperparameter profile within the feasible region. proxy model And obtain the optimal basic hyperparameters by maximizing the acquisition function. ,in This is the optimal loss observed so far; Step 2: When constructing the target model, calculate the width ratio r_n = n / n_base and the depth ratio r_L = L / L_base based on the network width n and depth L of the target model; Step 3: Scale the hyperparameters according to the following ratios: Hidden layer initialization variance = basic initialization variance / r_n; Hidden layer scaling factor = basic scaling factor / r_L; Hidden layer learning rate = basic learning rate / Output layer scaling factor = base scaling factor / r_n; Step 4: Train the network using the Muon-Kimi optimizer and parameterize the hyperparameters.
[0008] The hyperparameters include the base learning rate, the base initial variance, and the base scaling factor.
[0009] Each of the hidden residual blocks contains an identity mapping branch and a main branch, the main branch consisting of at least two layers of linear transformations. Let x be the input data. Features of each layer of the neural network, For the parameters of each layer of the neural network, Let L be the scaling factor for each layer of the neural network, and L be the number of hidden residual blocks in the network. Then the network architecture can be represented as: , , .
[0010] For the initial input and output layer weight matrices, the μP spectral condition is that the product of the RMS operator norm and the corresponding layer scaling factor remains constant; that is, the input and output weights satisfy: .
[0011] For each weight matrix of the main branch in each hidden residual block, the μP spectral condition is that the product of the RMS operator norms and the product of the scaling factor corresponding to that residual block should be inversely proportional to the network depth; that is, the hidden weights satisfy: ,in The RMS operator norm is the spectral norm multiplied by the matrix of size m x n. Θ is a symbol representing the same order in an asymptotic sense.
[0012] The specific method for training the network using the Muon-Kimi optimizer is as follows: First, determine the spectral constraints of the weight update amount. During the training process, apply the same form of spectral constraints as the initial weight values to the weight update amounts of each order of the input layer, output layer, and hidden layer.
[0013] The specific hyperparameter parameterization method of the Muon-Kimi optimizer is as follows: Perform singular value decomposition on the gradient matrix to extract its principal directions: , U l V l These are gradients The left and right singular vector matrices.
[0014] To ensure that Muon-Kimi's update rules satisfy the unified spectral constraints, the following three types of hyperparameters are jointly parameterized: Weight initialization variance parameters: The initial variance of the hidden layer weight matrix is inversely proportional to the network width; the input and output layer weight matrices use constant-order initialization variance. Block scaling factor parameters: The input layer scaling factor is set to a constant order; the output layer scaling factor is inversely proportional to the network width; the scaling factor for each hidden residual block is inversely proportional to the network depth. Learning rate parameters: The learning rate of the input layer is set to a constant order; the learning rate of the output layer is set to a constant order; the learning rate of the hidden layer is inversely proportional to the square root of the network width.
[0015] The technical effects to be achieved by the embodiments of the present invention are as follows: A unified µP method is proposed for scenarios where the width and depth of neural networks are expanded simultaneously. By characterizing the propagation of network weight parameters and their updates within the network from the perspective of spectral norm, a unified scale constraint is established, and a set of general hyperparameter parameterization rules is formed accordingly. This invention does not depend on a specific network structure or optimizer, and can uniformly map the update rules of different optimization algorithms to the spectral constraint framework, thereby maintaining stable feature learning behavior across different model scales and achieving reliable hyperparameter transfer. Attached Figure Description
[0016] The above and other objects and features of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings.
[0017] Figure 1 This is a schematic diagram illustrating the architecture of a maximum update parameterization method for large model width-depth expansion based on spectral conditions according to an embodiment of the present disclosure; Figure 2 This is a schematic diagram illustrating test results according to an embodiment of the present disclosure. Detailed Implementation
[0018] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.
[0019] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become clear upon understanding the disclosure of this application.
[0020] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.
[0021] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.
[0022] In the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" another element, directly "connected to," or "bonded to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, no other elements may be present in between.
[0023] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.
[0024] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.
[0025] Furthermore, in the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is believed that such detailed descriptions would lead to a vague interpretation of this disclosure.
[0026] Figure 1 This is a schematic diagram illustrating a maximum update parameterization method for large model width-depth expansion based on spectral conditions according to an embodiment of the present disclosure.
[0027] To achieve the aforementioned objectives, the present invention employs the following technical framework: Figure 1 As shown.
[0028] This invention addresses residual neural network structures that simultaneously expand width and depth, constructing a unified spectral constraint parameterization method to ensure that the network satisfies the following µP criteria at different scales: (1) the numerical scale of features in each layer remains stable; (2) parameter updates produce the largest possible effective change in features. To this end, this invention uniformly constrains the spectral norm of weight parameters and their update amounts from the perspectives of network forward propagation and parameter update propagation, and derives the parameterization methods for the corresponding initialization parameters, layer scaling factors, and learning rates of the Muon-Kimi optimizer.
[0029] Unified wide-depth extended µP spectral conditions: In residual neural networks that simultaneously expand width and depth, as the number of network layers increases, if the scale of the weight parameters and their updates is not controlled, feature changes from each residual block will accumulate along the network depth direction, leading to feature value explosion or disappearance, and thus disrupting stable network training. To address this problem, this invention utilizes the RMS operator norm (assuming a matrix size of m x n, the RMS operator norm is the spectral norm multiplied by...). Therefore, we still refer to it as the spectral condition) to uniformly characterize the propagation behavior of network weight parameters and their single-step update, and propose the following unified spectral constraint conditions.
[0030] The neural network includes an input layer, multiple hidden residual blocks, and an output layer. Each hidden residual block contains an identity mapping branch and a main branch, with the main branch consisting of at least two linear transformations. Let x be the input data. Features of each layer of the neural network, For the parameters of each layer of the neural network, Let L be the scaling factor for each layer of the neural network, and L be the number of hidden residual blocks in the network. Then the network architecture can be represented as: , , .
[0031] This invention requires: (a) Weight constraints of input and output layers For the input layer weight matrix and the output layer weight matrix, the product of their RMS operator norm and the corresponding layer scaling factor is kept constant, so that the scale of the input features and output features does not change with the network size.
[0032] (ii) Initial constraints for hiding residual block weights For each weight matrix of the main branch in each hidden residual block, the product of its RMS operator norm and the product of the scaling factor corresponding to that residual block should be inversely proportional to the network depth. This constraint ensures that the overall impact of each residual block on the features during forward propagation decreases with increasing depth, thus preventing the amplification of features caused by the accumulation of numerous residual blocks.
[0033] (III) Spectral Constraints on Weight Updates During training, the update amounts of the weights at each order of the input layer, output layer, and hidden layer are subject to spectral constraints of the same form as the initial weight values, so that the feature changes caused by a single-step update remain consistent across networks of different sizes.
[0034] μP spectral conditions under combined width and depth scaling: To ensure the μP principle holds, the initial weights and their updates at each step should satisfy: Initial conditions Input and output weights:
[0035] Hidden weights:
[0036] Update conditions Input and output weights:
[0037] Hidden weights (first-order weight update): ,
[0038] Hidden weights (second-order weight update):
[0039] (iv) Constraint Effect Through the aforementioned unified spectral constraints, this invention theoretically guarantees that the µP criterion can still be achieved in scenarios where both width and depth are expanded simultaneously.
[0040] Spectral Conditional Implementation of the Muon-Kimi Optimizer Under the premise of satisfying the unified spectrum constraint, this invention provides a specific hyperparameter parameterization implementation method for the Muon-Kimi optimizer.
[0041] Muon-Kimi is an optimizer that performs singular value direction normalization updates on matrix parameters and has recently been used for pre-training of state-of-the-art large language models. Its basic idea is to perform singular value decomposition on the gradient matrix, extract its principal directions, and normalize the update magnitude to facilitate the use of hyperparameters from existing optimizers such as AdamW. Its specific update method is as follows: , U l V l These are gradients The left and right singular vector matrices.
[0042] To ensure that Muon-Kimi's update rules satisfy the unified spectral constraints, this invention performs joint parameterization on the following three types of hyperparameters: (a) Initialization of variance parameters for weights For the hidden layer weight matrix, its initial variance is inversely proportional to the network width; for the input and output layer weight matrices, a constant-order initial variance is used. These settings ensure that the RMS operator norm of the hidden layer weights remains stable as the network width increases during the initialization phase.
[0043] (ii) Block scaling factor parameter The input layer scaling factor is set to a constant; the output layer scaling factor is inversely proportional to the network width; and the scaling factor for each hidden residual block is inversely proportional to the network depth. This setting ensures that as the number of residual blocks increases, the contribution of each residual block to the overall features decreases accordingly, thus satisfying the unified spectral constraint.
[0044] (iii) Learning rate parameter The learning rate of the Muon-Kimi optimizer is parameterized as follows: the input layer learning rate is set to a constant order; the output layer learning rate is set to a constant order; and the hidden layer learning rate is inversely proportional to the square root of the network width.
[0045] The above settings ensure that the spectral norm of the hidden layer weight update and the residual block scaling factor together satisfy the unified spectral constraint.
[0046] Method Implementation Steps The specific steps are shown in Table 1 below.
[0047] Table 1: μP implementation of Muon-Kimi (Liu et al., 2025) under width and depth scaling Red entries represent the differences between μP and SP (He et al., 2015), while gray entries show the corresponding SP methods. and These represent the width and depth scaling ratios relative to the baseline model, respectively. For the language task, the variance of the input weights is... For image tasks, it is: .
[0048]
[0049] Step 1: Determine the network width n_base and network depth L_base of the base model, and determine the hyperparameters such as the base learning rate, base initialization variance, and base scaling factor on the base model, such as by obtaining the corresponding optimal hyperparameters through grid search or random search; Step 2: When constructing the target model, calculate the width ratio r_n = n / n_base and the depth ratio r_L = L / L_base based on the network width n and depth L of the target model; Step 3: Scale the hyperparameters according to the following ratios: Hidden layer initialization variance = Basic initialization variance / r_n; Hidden layer scaling factor = Basic scaling factor / r_L; Hidden layer learning rate = Basic learning rate / Output layer scaling factor = base scaling factor / r_n; Step 4: Train the network using the Muon-Kimi optimizer.
[0050] The beneficial effects of the technical solution of this invention are as follows: Compared with existing standard parameterization methods (SP), the maximum update parameterization method (µP) based on unified spectral constraints proposed in this invention shows significant technical performance in scenarios where the width and depth of neural networks are expanded simultaneously.
[0051] First, this invention can maintain a basically constant numerical scale of internal network features in large language models with different widths and depths. Figure 2 (a, 2b) effectively avoids the feature explosion or feature vanishing problem that occurs as the model size increases, thereby significantly improving the stability of the large-scale model training process.
[0052] Secondly, this invention achieves reliable transfer of hyperparameters between models of different scales. Experimental results show that ( Figure 2 c, 2d) The base learning rate and related hyperparameters determined on small-scale models can be directly applied to larger-scale models after parameterization by this invention, without the need for large-scale parameter tuning, which significantly reduces the engineering cost of training large models.
[0053] Furthermore, under the same training settings, the model employing the technical solution of this invention can achieve lower validation loss across different width and depth scales. Figure 2 (c, 2d), and this advantage becomes more pronounced as the model size increases, indicating that the present invention further improves the training effect and performance of the model while ensuring stability.
[0054] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.
Claims
1. A maximum update parameterization method for large model width and depth expansion based on spectral conditions, characterized in that, During the training of a large model for language or image processing tasks, stable feature learning and transfer hyperparameters are performed as the width and depth scale expands, thereby obtaining the expanded parameter update of the large model for language / image processing tasks. Step 1: The base model is a large model for language / image processing tasks, consisting of a neural network model suitable for processing images or text, including an input layer, multiple hidden residual blocks, and an output layer. The network width n_base and network depth L_base of the base model are determined, and the optimal base hyperparameters are determined by Bayesian hyperparameter optimization on the base model, such as learning rate, inter-layer scaling factor, and weight initialization scale. Specifically, based on the aforementioned basic model, a basic hyperparameter vector to be optimized is constructed. Based on the training dataset and validation dataset Define the objective function of the base model on the validation set as follows: A Bayesian hyperparameter optimization method is used to construct a hyperparameter profile within the feasible region. proxy model And obtain the optimal basic hyperparameters by maximizing the acquisition function. ,in This is the optimal loss observed so far; Step 2: When constructing the target model, calculate the width ratio r_n = n / n_base and the depth ratio r_L = L / L_base based on the network width n and depth L of the target model; Step 3: Scale the hyperparameters according to the following ratios: Hidden layer initialization variance = basic initialization variance / r_n; Hidden layer scaling factor = basic scaling factor / r_L; Hidden layer learning rate = basic learning rate / Output layer scaling factor = base scaling factor / r_n; Step 4: Train the network using the Muon-Kimi optimizer and scaled hyperparameters to obtain a large model with updated parameters for language or image processing tasks.
2. The maximum update parameterization method for large model width and depth expansion based on spectral conditions as described in claim 1, characterized in that, The hyperparameters include the base learning rate, the base initial variance, and the base scaling factor.
3. The maximum update parameterization method for large model width and depth expansion based on spectral conditions as described in claim 2, characterized in that, Each of the hidden residual blocks contains an identity mapping branch and a main branch, the main branch consisting of at least two layers of linear transformations; let x be the input data. W represents the features of each layer of the neural network. l For the parameters of each layer of the neural network, Let L be the scaling factor for each layer of the neural network, and L be the number of hidden residual blocks in the network. Then the network architecture can be represented as: , ， 。 4. The maximum update parameterization method for large model width and depth expansion based on spectral conditions as described in claim 3, characterized in that, For the initial input and output layer weight matrices, the μP spectral condition is that the product of the RMS operator norm and the corresponding layer scaling factor remains constant; that is, the input and output weights satisfy: ,in The RMS operator norm is the spectral norm multiplied by the matrix of size m x n. Θ is a symbol representing the same order in an asymptotic sense.
5. The maximum update parameterization method for large model width and depth expansion based on spectral conditions as described in claim 4, characterized in that, For each weight matrix of the main branch in each hidden residual block, the μP spectral condition is that the product of the RMS operator norms and the product of the scaling factor corresponding to that residual block should be inversely proportional to the network depth; that is, the hidden weights satisfy: .
6. The maximum update parameterization method for large model width and depth extension based on spectral conditions as described in claim 1, characterized in that, The specific method for training the network using the Muon-Kimi optimizer is as follows: First, determine the spectral constraints of the weight update amount. During the training process, apply the same form of spectral constraints as the initial weight values to the weight update amounts of each order of the input layer, output layer, and hidden layer. The specific hyperparameter parameterization method of the Muon-Kimi optimizer is as follows: Perform singular value decomposition on the gradient matrix to extract its principal directions: ， U l V l These are gradients The left and right singular vector matrices.
7. The maximum update parameterization method for large model width and depth extension based on spectral conditions as described in claim 1, characterized in that, To ensure that Muon-Kimi's update rules satisfy the unified spectral constraints, the following three types of hyperparameters are jointly parameterized: Weight initialization variance parameters: The initial variance of the hidden layer weight matrix is inversely proportional to the network width; the input and output layer weight matrices use constant-order initialization variance. Block scaling factor parameters: The input layer scaling factor is set to a constant order; the output layer scaling factor is inversely proportional to the network width; the scaling factor for each hidden residual block is inversely proportional to the network depth. Learning rate parameters: The learning rate of the input layer is set to a constant order; the learning rate of the output layer is set to a constant order; the learning rate of the hidden layer is inversely proportional to the square root of the network width.