Distributed pruning method for intelligent traffic large language model based on deep expansion network

Abstract

The application relates to the technical field of large language model pruning optimization, and particularly discloses a smart traffic large language model distributed pruning method based on a deep unfolding network, which comprises the following steps: modeling a large language model pruning task as a distributed optimization problem, and performing layer-by-layer pruning on each transformer layer; solving the distributed optimization problem by using ADMM; unfolding the ADMM into a deep neural network structure, wherein the deep unfolding network is composed of layer network blocks, each block comprises a local weight updating module, a global sparse updating module and a dual variable updating module, and the three modules correspond to one ADMM iteration; and introducing an asynchronous communication mechanism in the pruning optimization framework. By using the above method, the deployability and real-time performance of the traffic large language model in travel services, traffic prediction and scheduling optimization are improved, the convergence speed and training efficiency are significantly improved, and the method is suitable for large-scale smart traffic edge deployment environments across intersections and regions.

Description

Technical Field

[0001] This invention relates to the field of large language model pruning and optimization technology, and in particular to a distributed pruning method for intelligent transportation large language models based on deep unfolded networks. Background Technology

[0002] In recent years, large language models have demonstrated their potential to surpass traditional algorithms in the field of intelligent transportation, particularly in complex cognitive tasks such as traffic flow prediction, intelligent signal control strategy generation, natural language interactive traffic condition query and analysis, and vehicle-roadside collaborative decision-making. However, the enormous computational and storage overhead of foundational models such as GPT and LLaMA, which have hundreds of billions of parameters, makes them difficult to deploy directly on intelligent transportation edge devices (such as roadside units, traffic signal controllers, and onboard computing units) where computing power, memory, and power consumption are severely limited. Model compression technology is crucial to bringing this capability to the edge. Among various compression methods, model pruning is considered an effective path for edge deployment because it can directly reduce model parameters and computational load. Currently, the mainstream methods in this field can be summarized into two categories, but both have significant limitations.

[0003] The first approach is a centralized pruning and redeployment method for edge computing. This method first trains a large language model using aggregated global traffic data in a powerful cloud computing center, then compresses it into a lightweight model through pruning techniques, and finally distributes and deploys it to various edge devices. This approach has two drawbacks: First, the model obtained by this method is difficult to fully adapt to the local traffic characteristics of each edge node. This "one-size-fits-all" optimization objective leads to model parameters compromising with mainstream patterns. While it has a certain degree of generalization on a macro level, its accuracy and adaptability are often unsatisfactory when dealing with specific local scenarios with unique data patterns, failing to reach the performance level of dedicated models. Second, this method requires the continuous transmission of large-scale, continuous raw video data from various edge nodes to a single central cloud for centralized model training and pruning. Transmitting and storing city-level raw data places enormous pressure on network bandwidth and central storage facilities.

[0004] The second approach is the isolated, single-device pruning method. This method abandons data collaboration and prunes and fine-tunes the pre-trained model solely on a single edge device, utilizing its limited local data. The drawback of this method is the extremely limited observation field and data volume of a single edge device. The pruned model trained based on this limitation overfits to local patterns and fails to learn broad, general traffic patterns. This makes it difficult to support a city-level intelligent transportation system that requires unified, highly robust intelligent services.

[0005] In summary, existing technologies face the core contradiction of data incompatibility and resource constraints in the specific application scenario of intelligent transportation. Therefore, a new distributed communication and optimization framework is needed to collaboratively utilize distributed traffic data knowledge without aggregating privacy data, generating lightweight and robust specialized models for edge devices. The existing ADMM-Grad technology transforms the LLM pruning problem into a convex optimization problem and solves it based on the ADMM algorithm. However, this technology still has the following drawbacks:

[0006] (1) ADMM-Grad is mainly designed for single-machine or centralized computing environments, and its optimization goal is to obtain a globally unified model. However, when applied to intelligent transportation, a typical distributed scenario, this centralized paradigm is difficult to adapt to the unique local traffic patterns of each edge node. Due to the inability to perform targeted optimization on node data scattered in different geographical locations, the accuracy of this model is limited when dealing with specific traffic conditions at specific intersections. At the same time, the method itself lacks an effective mechanism for handling distributed parameter partitioning and cross-node consistency constraints, making it difficult to achieve efficient and reliable model compression in distributed transportation systems.

[0007] (2) ADMM-Grad typically requires many iterations to converge, especially in cases with a large number of parameters in large models, where the computational cost of training and pruning remains too high. In addition, ADMM-Grad requires inverting large matrices during iteration, which may become a bottleneck in actual computation. In real-world deployments, traffic edge devices are often constrained by power consumption and heat dissipation conditions, and methods with excessive computational burden can easily lead to excessive energy consumption and overheating of the devices.

[0008] (3) The ADMM-Grad pruning process only updates weights under a fixed mask, which is usually generated by an external heuristic. This mask selection method cannot meet the real-time requirements of intelligent transportation tasks and is prone to causing a mismatch between the sparse pattern and the actual distribution of traffic data, resulting in unsatisfactory pruning effect and inference efficiency on edge devices. Summary of the Invention

[0009] The purpose of this invention is to address the problems of poor local adaptability in existing centralized large language model pruning methods, the large number of iterations and slow convergence speed of traditional ADMM pruning methods, and the fact that existing ADMM-Grad methods can only update weights under a given mask, and the mask itself still relies on heuristic strategies. This invention proposes a distributed pruning framework based on deep unfolded ADMM, modeling the pruning problem as a distributed consistency optimization problem; unfolding the iterative steps of ADMM into a deep neural network; introducing global sparse variables into the ADMM framework; and setting the threshold parameter as a learnable variable in the deep unfolded network.

[0010] To achieve the above objectives, this invention provides a distributed pruning method for intelligent transportation large language models based on deep unfolded networks, comprising the following steps:

[0011] S1. Model the large language model pruning task as a distributed optimization problem: perform layer-by-layer pruning for each transformer layer;

[0012] S2. Iterative solution: ADMM is used to solve the distributed optimization problem;

[0013] S3. Unfold ADMM into a deep neural network structure, i.e., a deep unfolded network: The deep unfolded network consists of... The network consists of layers and blocks. Each block contains a local weight update module, a global sparse update module, and a dual variable update module, corresponding to one ADMM iteration.

[0014] Preferably, S1 is as follows:

[0015] Let the original weights of a certain layer be... The corresponding input activation is The output is ,right Each line of application Norms are used to promote the sparsity of the entire matrix, and the expression is:

[0016] ;

[0017] in, express The OK, It is a sparse regularization weight. It is the Frobenius norm. for Norm, For row index;

[0018] For each node , Define the total number of nodes. For nodes Local weighted partitioning, For nodes Local activation data, Output the corresponding target fragment. Global consensus variable The Each row block introduces a global consensus variable. The expression for the distributed optimization problem is:

[0019] ;

[0020] in, for The OK, Constraints to ensure that local weights are consistent with global consensus.

[0021] Preferably, S2 is as follows:

[0022] ADMM is used to solve distributed optimization problems, and the augmented Lagrangian function corresponding to the expression of the distributed optimization problem is introduced:

[0023] ;

[0024] in, It is the penalty coefficient. For nodes The dual variable;

[0025] ADMM update is written as:

[0026] ;

[0027] ;

[0028] ;

[0029] in, For learnable parameters, Index for iteration count, For nodes The local weights in the first place The update results of the iteration For the Master node at the The global sparse variables are obtained through communication and aggregation calculations among all nodes after iteration. To find the variable that minimizes the objective function, For nodes In the The dual variable after iterative update For nodes In the The dual variable after iterative update For nodes The global sparse variable in the first place The update results of the iteration;

[0030] The stopping criterion for the iterative algorithm is:

[0031] .

[0032] Preferably, the , , The specific update formulas for the three variables in a distributed scenario are as follows:

[0033] S21, renew:

[0034] For nodes , The updated objective function is:

[0035] ;

[0036] By taking Updated objective function pair Taking the first derivative and setting it to zero yields an explicit closed-form solution. Define , , It is the identity matrix, and the expression for the closed-form solution is as follows:

[0037] ;

[0038] in, It is a local feature correlation matrix. It is the local input-output covariance matrix;

[0039] S22, renew:

[0040] Define intermediate variables Merge all line blocks, then , The updated optimization problem expression is:

[0041] ;

[0042] remember The Rows are vectors ,correspond The Rows are vectors This leads to the row-level subproblem:

[0043] ;

[0044] This question is standard. The problem has a closed-form solution using soft thresholding:

[0045] ;

[0046] in, For the first In the next iteration, the matrix Any element in the array; This is a sign function that returns the sign of the original data. For the first The matrix at the next iteration Located in the line, number Column elements; For column indexes;

[0047] The vector form is denoted as:

[0048] ;

[0049] in, Indicates soft threshold operation;

[0050] Therefore, the complete matrix The update is as follows:

[0051] ;

[0052] S23, renew:

[0053] conduct The updated expression is:

[0054] .

[0055] Preferably, a distributed Master-Worker asynchronous communication architecture is adopted to support parallel pruning optimization of large language models, specifically:

[0056] A partial barrier mechanism is introduced, meaning the master node receives no less than [a certain number of] [items] in each iteration. After updating the results of each worker, execute the global variable. Update operation, ;

[0057] A bounded delay mechanism is introduced, which sets a global delay cap parameter to constrain the maximum number of updates that each worker node can tolerate before being ignored by the master node. That is, only if the update delay of all worker nodes satisfies Under certain conditions, the master node is allowed to trigger a global update.

[0058] Preferably, the local weight update module, global sparse update module, and dual variable update module mentioned in S3 are specifically as follows:

[0059] S31, Local Weight Update Module:

[0060] Based on the relationships between variables in the closed-form solution expression described in S21, a neural network module is designed to parameterize the variables in this process and use learnable matrices. approximate We obtain the approximate update formula:

[0061] ;

[0062] in, , It is a learnable vector;

[0063] S32, Global Sparse Update Module:

[0064] The complete matrix described in S22 Fixed threshold for updating formula Replace with learnable parameters , complete matrix The updated formula is rewritten as follows:

[0065] ;

[0066] S33, Dual Variable Update Module:

[0067] right The update formula for the dual variable is modified, specifically, given three inputs. , , The dual variable update module in the first... The output of the layer network block is:

[0068] ;

[0069] in, It is the first Learnable parameters in layered network blocks.

[0070] Preferably, all learnable parameters are not shared between different network blocks, that is, the parameters of each layer are learned independently for the optimization characteristics of its iteration stage.

[0071] This invention employs the aforementioned distributed pruning method for intelligent transportation large language models based on deep unfolded networks, and its beneficial effects are as follows:

[0072] (1) This invention systematically models the large language model pruning task in intelligent transportation scenarios as a distributed constrained optimization problem. The single-layer Transformer weight matrix is ​​divided into rows and distributed to different traffic edge nodes. Each node independently completes the local weight update of the corresponding parameter sub-block. Then, the master node is responsible for the synchronization and aggregation of global sparse variables, thereby ensuring the consistency and global optimality of the overall pruning. This modeling method effectively solves the storage bottleneck and computational burden of centralized pruning on ultra-large-scale traffic models, enabling traffic large language models with hundreds of billions of parameters to achieve efficient sparsification on edge devices with limited computing power, improving their deployability and real-time performance in travel services, traffic prediction and scheduling optimization.

[0073] (2) This invention unfolds the distributed ADMM pruning algorithm into a deep neural network structure, constructing a differentiable network composed of a local weight update module, a global sparse update module, and a dual variable update module. Compared with traditional pruning methods based on heuristics or simple optimization, the deep unfolding structure proposed in this invention significantly improves the convergence speed and training efficiency, and achieves a better balance between sparse structure and task performance, enabling it to simultaneously consider inference efficiency and the accuracy of travel prediction and scheduling recommendation in intelligent transportation scenarios.

[0074] (3) This invention introduces an asynchronous communication mechanism into the pruning optimization framework of the intelligent transportation large language model, and combines a partial obstacle mechanism and a bounded delay mechanism to ensure algorithm convergence. The master node receives no less than Once the update results of each worker node are obtained, global variables can be updated to avoid global blocking caused by slow nodes; at the same time, a global delay limit parameter can be set. This design ensures that all nodes can participate in updates within a certain number of iterations, thereby guaranteeing global consistency and convergence stability. It effectively reduces communication latency and energy consumption between edge nodes, improves system fault tolerance and scalability, and is particularly suitable for large-scale intelligent transportation edge deployment environments spanning intersections and regions. Attached Figure Description

[0075] Figure 1 This is the first step in the distributed pruning method for intelligent transportation large language models based on deep unfolded networks, as proposed in this invention. Layer-depth unfolded network block structure diagram. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0077] The distributed pruning method for intelligent transportation large language models based on deep unfolded networks, as described in this embodiment of the invention, includes the following steps:

[0078] S1. Model the large language model pruning task as a distributed optimization problem: perform layer-by-layer pruning for each transformer layer, specifically:

[0079] Let the original weights of a certain layer be... The corresponding input activation is The output is The goal is to find the sparsified weights. This minimizes the output reconstruction error while satisfying the sparsity constraint. Due to direct application... Norms lead to nonconvex optimization problems; this invention employs their convex approximation, i.e. Norms. Specifically, this is to effectively achieve sparsity while ensuring the weights after pruning. The computational feasibility, for Each line of application The norm is used to promote the sparsity of the entire matrix. The task of this invention can be expressed in the following form:

[0080] ;

[0081] in, express The OK, It is a sparse regularization weight. It is the Frobenius norm. for Norm, This is a row index.

[0082] For each node , Define the total number of nodes. For nodes Local weighted partitioning, For nodes Local activation data, Output the corresponding target fragment. Global consensus variable The Each line is a block. To transform the above task expression into a distributed consensus form, a global consensus variable is introduced. The expression for the distributed optimization problem is:

[0083] ;

[0084] in, for The OK, Constraints to ensure that local weights are consistent with global consensus.

[0085] S2, Solving using iterative algorithm:

[0086] To adapt to distributed solutions, ADMM is used to solve distributed optimization problems, and the augmented Lagrangian function corresponding to the expression of the distributed optimization problem is introduced:

[0087] ;

[0088] in, It is the penalty coefficient. For nodes The dual variable.

[0089] ADMM update is written as:

[0090] ;

[0091] ;

[0092] ;

[0093] in, For learnable parameters, Index for iteration count, For nodes The local weights in the first place The update results of the iteration For the Master node at the The global sparse variables are obtained through communication and aggregation calculations among all nodes after iteration. To find the variable that minimizes the objective function, For nodes In the The dual variable after iterative update For nodes In the The dual variable after iterative update For nodes The global sparse variable in the first place The update results of the iteration;

[0094] The stopping criterion for the iterative algorithm is:

[0095] .

[0096] , , The specific update formulas for the three variables in a distributed scenario are as follows:

[0097] S21, renew:

[0098] For nodes , The updated objective function is:

[0099] ;

[0100] Obviously The objective function for updating is a convex problem, which can be solved by taking... Updated objective function pair Taking the first derivative and setting it to zero yields an explicit closed-form solution. Definition , , It is the identity matrix, and the expression for the closed-form solution is as follows:

[0101] ;

[0102] in, It is a local feature correlation matrix. It is the local input-output covariance matrix.

[0103] S22, renew:

[0104] Define intermediate variables Merge all line blocks, then , The updated optimization problem expression is:

[0105] ;

[0106] remember The Rows are vectors ,correspond The Action is This leads to the row-level subproblem:

[0107] ;

[0108] This question is standard. The problem has a closed-form solution using soft thresholding:

[0109] ;

[0110] in, For the first In the next iteration, the matrix Any element in the array; This is a sign function that returns the sign of the original data. For the first The matrix at the next iteration Located in the line, number Column elements; For column indexes;

[0111] The vector form is denoted as:

[0112] ;

[0113] in, Indicates soft threshold operation;

[0114] Therefore, the complete matrix The update is as follows:

[0115] ;

[0116] S23, renew:

[0117] conduct The updated expression is:

[0118] .

[0119] conduct After the update, continue with the next iteration until the stopping criterion of the iterative algorithm is met, at which point the iteration stops.

[0120] S24, Distributed Communication Architecture:

[0121] A distributed Master-Worker asynchronous communication architecture is adopted to support parallel pruning optimization of large language models, specifically:

[0122] A partial hindrance mechanism is introduced, meaning that the master node does not need to wait for all worker nodes to complete their local weight updates in each iteration, but only needs to receive at least... After the update results of each worker ( ), which allows the execution of global variables This mechanism effectively reduces overall blocking caused by latency in a few nodes or uneven computational load, thus improving the parallel efficiency of global iteration.

[0123] To ensure the convergence and fairness of asynchronous updates, a bounded delay mechanism is introduced. This mechanism sets a global delay cap parameter to constrain the maximum number of updates that each worker node can tolerate before being ignored by the master node. That is, only if the update delay of all worker nodes satisfies Under certain conditions, the master node is allowed to trigger a global update. This design improves communication asynchronicity while effectively preventing convergence instability caused by a single node not participating in updates for a long time, ensuring that the distributed pruning algorithm can maintain convergence and optimization accuracy in an asynchronous environment.

[0124] S3. Unfold ADMM into a deep neural network structure, i.e., a deep unfolded network:

[0125] To solve the model pruning problem in the distributed optimization problem expression of S1, the ADMM algorithm typically requires hundreds of iterations to obtain a satisfactory result. Furthermore, the ADMM algorithm for S21... Matrix inversion in the updated closed-form solution is an unavoidable part of the iteration process. Both of these factors consume significant computational resources, leading to excessively long actual computation times. Furthermore, the selection of appropriate hyperparameters is also a challenge. This is also a challenge in practical applications.

[0126] To overcome these difficulties, this invention employs a deep unrolling method, defining a deep network architecture by unrolling the iterative process in the three expressions of ADMM update in S2 and transforming some variables into trainable variables. The deep unrolling method optimizes parameters by minimizing the loss on the training set, integrating model-based and learning-based methods. Compared to iterative optimization, these methods transform the iterative process into the forward propagation of the neural network, significantly accelerating computation. Unlike typical deep learning models, they incorporate the computational relationships between parameters, reducing model complexity and enhancing interpretability.

[0127] The depth unfolding network of this invention is composed of The network consists of layers of blocks, each containing a local weight update module, a global sparse update module, and a dual variable update module, corresponding to one ADMM iteration. The structure of a layer-deep unfolded network is as follows: Figure 1 As shown.

[0128] The local weight update module, the global sparse update module, and the dual variable update module are as follows:

[0129] S31, Local Weight Update Module:

[0130] In the iterative optimization, variables are updated at each node according to the closed-form solution described in S21. In this module, a neural network module is designed based on the relationships between variables in the closed-form solution expression described in S21. Some variables in this process are parameterized to enhance its expressive power. When the input is high-dimensional data, matrix inversion operations typically consume significant computational resources. Therefore, matrix inversion operations need to be avoided. To achieve this, learnable matrices are used. approximate We obtain the approximate update formula:

[0131] ;

[0132] in, , It is a learnable vector;

[0133] This diagonal approximation parameterization strategy has many advantages. First, the computational complexity and storage requirements are reduced from the original... and Downgraded to and This significantly reduces the communication and computational overhead of distributed systems. Secondly, each worker only needs to learn... corresponding The parameterization strategy uses only diagonal parameters, eliminating the need to learn complex row-column coupling parameters. This reduces the likelihood of gradient explosion / vanishing during training, making it ideal for end-to-end optimization of neural networks. Finally, compared to the common method of using fully connected layers for feature mapping in deep unfolded networks, the parameterization strategy of this invention requires fewer parameters and is more feasible.

[0134] S32, Global Sparse Update Module:

[0135] In deep unfolded networks, to improve flexibility and learning ability, the complete matrix of S22 is... Fixed threshold for updating formula Replace with learnable parameters , complete matrix The updated formula is rewritten as follows:

[0136] ;

[0137] By introducing learnable The model no longer relies on a globally fixed, sparse penalty parameter that requires repeated trial and error to determine. Instead, it can dynamically learn and adapt an optimal threshold for each layer during end-to-end training. This allows the model to adaptively adjust the aggressiveness of pruning based on the current feature distribution and training progress. Furthermore, this design transforms the threshold selection problem, traditionally a hyperparameter tuning problem, into a gradient optimization problem of the model's internal parameters. This not only avoids complex and computationally expensive manual parameter searching but also enables the threshold to converge to a better value in a data-driven manner.

[0138] S33, Dual Variable Update Module:

[0139] In this module, for The update formula for the dual variable is modified, specifically, given three inputs. , , The dual variable update module in the first... The output of the layer network block is:

[0140] ;

[0141] in, It is the first Learnable parameters in layered network blocks.

[0142] The learnable parameters of the model proposed in this invention are not shared between different network blocks. That is, the parameters of each layer are independently learned for the optimization characteristics of its iteration stage, thereby enhancing the model's adaptability to different iteration dynamics, avoiding the decrease in convergence speed or the solidification of sparse patterns caused by parameter coupling, and ensuring the flexibility and expressiveness of the pruning process.

[0143] Therefore, this invention adopts the above-mentioned distributed pruning method for intelligent transportation large language models based on deep unfolded networks. (1) For the first time, the large language model pruning task in intelligent transportation scenarios is systematically modeled as a distributed constrained optimization problem. Specifically, the single-layer Transformer weight matrix is ​​divided into rows and distributed to different traffic edge nodes. Each node independently completes the local weight update of the corresponding parameter sub-block. Then, the master node is responsible for the synchronization and aggregation of global sparse variables, thereby ensuring the consistency and global optimality of the overall pruning. This modeling method effectively solves the storage bottleneck and computational burden of centralized pruning on ultra-large-scale traffic models, enabling traffic large language models with hundreds of billions of parameters to achieve efficient sparsification on edge devices with limited computing power, improving their deployability and real-time performance in travel services, traffic prediction and scheduling optimization; (2) The distributed ADMM pruning algorithm is unfolded into a deep neural network structure, constructing a differentiable network composed of a local weight update module, a global sparse update module and a dual variable update module. Each module strictly corresponds to the mathematical update steps in the optimization algorithm, realizing the unity of interpretability at the algorithm level and trainability at the network level. Compared with traditional heuristic or simple optimization-based pruning methods, the deep unfolding structure proposed in this invention significantly improves convergence speed and training efficiency, and achieves a better balance between sparse structure and task performance, enabling it to simultaneously consider inference efficiency and the accuracy of travel prediction and scheduling recommendation in intelligent transportation scenarios; (3) For the first time, an asynchronous communication mechanism is introduced into the pruning optimization framework of intelligent transportation large language model, and combined with partial obstacle mechanism and bounded delay mechanism to ensure algorithm convergence. Specifically, the master node receives no less than Once the update results of each worker node are obtained, global variables can be updated to avoid global blocking caused by slow nodes; at the same time, a global delay limit parameter can be set. This design ensures that all nodes can participate in updates within a certain number of iterations, thereby guaranteeing global consistency and convergence stability. It effectively reduces communication latency and energy consumption between edge nodes, improves system fault tolerance and scalability, and is particularly suitable for large-scale intelligent transportation edge deployment environments spanning intersections and regions.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A distributed pruning method for intelligent transportation large language models based on deep unfolded networks, characterized in that, Includes the following steps: S1. Model the large language model pruning task as a distributed optimization problem: perform layer-by-layer pruning for each transformer layer, divide the single-layer Transformer weight matrix into rows and distribute it to different traffic edge nodes, each node independently completes the local weight update of the corresponding parameter sub-block, and then the master node is responsible for the synchronization and aggregation of global sparse variables. S2. Iterative solution: ADMM is used to solve the distributed optimization problem; S3. Unfold ADMM into a deep neural network structure, i.e., a deep unfolded network: The deep unfolded network consists of... The network consists of layers and blocks, each block containing a local weight update module, a global sparse update module, and a dual variable update module, corresponding to one ADMM iteration. S1 specifically refers to: Let the original weight of a certain layer be... The corresponding input activation is The output is ,right Each line of application Norms are used to promote the sparsity of the entire matrix, and the expression is: ； in, express The OK, It is a sparse regularization weight. It is the Frobenius norm. for Norm, For row index, express The optimal solution; For each node , Define the total number of nodes. For nodes Local weighted partitioning, For nodes Local activation data, Output the corresponding target fragment. Global consensus variable The Each row block introduces a global consensus variable. The expression for the distributed optimization problem is: ； in, for The OK, express The optimal solution. Constraints to ensure that local weights are consistent with global consensus.

2. The distributed pruning method for intelligent transportation large language models based on deep unfolded networks according to claim 1, characterized in that, S2 specifically refers to: ADMM is used to solve distributed optimization problems, and the augmented Lagrangian function corresponding to the expression of the distributed optimization problem is introduced: ； in, It is the penalty coefficient. For nodes The dual variable; ADMM update is written as: ； ； ； in, For learnable parameters, Index for iteration count, For nodes The local weights in the first place The update results of the iteration For the Master node at the The global sparse variables are obtained through communication and aggregation calculations among all nodes after iteration. To find the variable that minimizes the objective function, For nodes In the The dual variable after iterative update For nodes In the The dual variable after iterative update For nodes The global sparse variable in the first place The update results of the iteration; The stopping criterion for the iterative algorithm is: 。 3. The distributed pruning method for intelligent transportation large language models based on deep unfolded networks according to claim 2, characterized in that, S21, renew: For nodes , The updated objective function is: ； By taking Updated objective function pair Taking the first derivative and setting it to zero yields an explicit closed-form solution. Define , , It is the identity matrix, and the expression for the closed-form solution is as follows: ； in, It is a local feature correlation matrix. It is the local input-output covariance matrix; S22, renew: Define intermediate variables Merge all line blocks, then ,get The updated optimization problem expression; remember The Rows are vectors ,correspond The Rows are vectors This yields a row-level subproblem; this problem is the standard The problem has a closed-form solution using soft thresholding: ； in, For the first In the next iteration, the matrix Any element in the array; This is a sign function that returns the sign of the original data. For the first The matrix at the next iteration Located in the line, number Column elements; For column indexes; The vector form is denoted as: ； in, Indicates soft threshold operation; Therefore, the complete matrix The update expression is: ； S23, renew: conduct The updated expression is: 。 4. The distributed pruning method for intelligent transportation large language models based on deep unfolded networks according to claim 3, characterized in that, A distributed Master-Worker asynchronous communication architecture is adopted to support parallel pruning optimization of large language models, specifically: A partial barrier mechanism is introduced, meaning the master node receives no less than [a certain number of] [items] in each iteration. After updating the results of each worker, execute the global variable. Update operation, ; A bounded delay mechanism is introduced, which sets a global delay cap parameter to constrain the maximum number of updates that each worker node can tolerate before being ignored by the master node. That is, only if the update delay of all worker nodes satisfies Under certain conditions, the master node is allowed to trigger a global update.

5. The distributed pruning method for intelligent transportation large language models based on deep unfolded networks according to claim 4, characterized in that, The local weight update module, global sparse update module, and dual variable update module mentioned in S3 are specifically as follows: S31, Local Weight Update Module: Based on the relationships between variables in the closed-form solution expression described in S21, a neural network module is designed to parameterize the variables in this process and use learnable matrices. approximate We obtain the approximate update formula: ； in, , It is a learnable vector; S32, Global Sparse Update Module: The complete matrix described in S22 Fixed threshold for updating formula Replace with learnable parameters , complete matrix The updated formula is rewritten as follows: ； S33, Dual Variable Update Module: right The update formula for the dual variable is modified, specifically, given three inputs. , , The dual variable update module in the first... The output of the layer network block is: ； in, It is the first Learnable parameters in layered network blocks.

6. The distributed pruning method for intelligent transportation large language models based on deep unfolded networks according to claim 5, characterized in that, All learnable parameters are not shared between different network blocks; that is, the parameters of each layer are learned independently for the optimization characteristics of its iteration stage.

Images