Improvement method of large language model, electronic equipment and storage medium

Abstract

The invention discloses an improvement method of a large language model, electronic equipment and a storage medium, and the method comprises the steps: calculating a correlation matrix between middle layers through a public data set, and calculating mutual information redundancy between the layers based on the correlation matrix between the middle layers, an inter-layer redundancy matrix is constructed through mutual information redundancy among the layers; calculating the total redundancy of each layer based on the redundancy matrix, and generating a layer importance factor through function conversion; establishing an objective function based on the layer importance factors and constructing a linear optimization model of sparse rate distribution of each layer; solving the optimization model by adopting a linear programming algorithm to obtain an optimal sparse rate distribution scheme of each layer; differentiated pruning is carried out on each layer of weight matrix according to the distribution scheme, a weight parameter with the maximum amplitude in each layer of weight matrix is reserved, the rest parameters are set to be zero, and a sparse weight matrix is obtained; according to the method, the features can be selectively pruned in the middle layer, so that fine-grained optimization of a large language model is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of language models, and in particular relates to an improvement method for a large language model, an electronic device, and a storage medium. Background Art

[0002] Among related technologies, OWL (Outlier Weighed Layerwise Sparsity) is a non-uniform inter-layer sparsity technique that assigns different sparsity ratios based on the proportion of outliers in each layer. This method exploits the fact that certain features in large language models exhibit significantly larger values ​​and optimizes the parameter settings by adjusting hyperparameters such as the outlier threshold and upper and lower sparsity limits. While this method shows potential for improving sparsity allocation, it relies heavily on empirical analysis and lacks a strong theoretical basis to demonstrate its effectiveness, limiting its generalizability and robustness across different architectures and datasets. SparseGPT introduces a Hessian matrix-based importance estimation technique to prune less important weights and reconstruct inter-layer outputs based on the importance measure. This technique reduces computational overhead and allows the application of traditional OBS (Optimal Brain Surgeon) neural network pruning methods to large-scale models. The Wanda method adopts a simplified strategy, relying solely on the product of weights and activations to identify and remove unimportant weights, thereby accelerating pruning.

[0003] These technologies all address the critical issues of model compression and optimization, specifically using sparse techniques to reduce model size without significantly impacting performance. Each offers unique solutions to address the redundancy inherent in large language models, better adapting to limited computing resources. These approaches not only improve model efficiency but also offer new perspectives for developing more powerful and accessible language models.

[0004] The inventors found that the relevant technologies have the following common shortcomings when dealing with knowledge distillation of sparse large language models: Defect 1: Minimizing the sum of introduced errors based on layer-by-layer pruning. However, such a greedy strategy will lead to the accumulation of errors in the entire model, ultimately resulting in unsatisfactory overall performance. In addition, these methods face more challenges in retraining models on large-scale datasets. Defect 2: Many methods use a uniform sparsity rate without considering the different contributions of each layer to the final model performance. Defect 3: When the existing technology is applied to a new model, complex adjustments and sufficient manual experience are required to determine the potential optimal parameter combination.

[0005] Among them, the following flaws include: 1. Greedy layer-by-layer pruning strategies: This type of approach attempts to optimize the model by minimizing pruning error layer by layer. However, this approach can easily trap the model in local optima, and pruning error accumulates with each layer, ultimately affecting overall model performance. Consequently, retraining the model on large-scale datasets presents difficulties. 2. Many methods, including Wanda and SparseGPT, set the same sparsity ratio for each layer, failing to consider the varying contributions of different layers to the model's final performance. This means that in practice, manual adjustment of sparsity may be necessary for different layers, which undoubtedly increases implementation complexity and uncertainty. 3. While the OWL method attempts to improve model optimization through outlier-based non-uniform inter-layer sparsification techniques, its effectiveness in practical applications relies heavily on empirical analysis and lacks a solid theoretical foundation. The selection of hyperparameters in this approach (such as the outlier threshold and upper and lower bounds on sparsity) is sensitive to model performance, and the effective ranges and thresholds of these hyperparameters often require manual tuning, which introduces additional complexity when applied to new models.

[0006] The inventors also found that the limitations of existing technologies make it impossible to provide a more intelligent and adaptive layer sparsity strategy, resulting in the strategy being unable to automatically adapt to the different characteristics of each layer within the model and unable to achieve more effective compression and optimization. Summary of the Invention

[0007] The embodiments of the present invention aim to solve at least one of the above technical problems.

[0008] In the first aspect, an embodiment of the present invention provides an improved method for a large language model, including: calculating the correlation matrix between intermediate layers through a public data set, calculating the mutual information redundancy between each layer based on the correlation matrix between the intermediate layers, and constructing an inter-layer redundancy matrix through the mutual information redundancy between the layers; calculating the total redundancy of each layer based on the redundancy matrix, and generating a layer importance factor through function conversion; establishing an objective function based on the layer importance factor, and constructing a linear optimization model for the sparsity rate allocation of each layer with the model target size as a constraint condition; solving the optimization model with a linear programming algorithm to obtain the optimal sparsity rate allocation scheme for each layer; performing differential pruning on the weight matrix of each layer according to the allocation scheme, retaining the weight parameter with the largest amplitude in the weight matrix of each layer, and setting the remaining parameters to zero to obtain a sparse weight matrix.

[0009] In a second aspect, an embodiment of the present invention provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any one of the above-mentioned methods for improving the large language model of the present invention.

[0010] In a third aspect, an embodiment of the present invention provides a storage medium, in which one or more programs including execution instructions are stored. The execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute any of the above-mentioned large language model improvement methods of the present invention.

[0011] In a fourth aspect, an embodiment of the present invention further provides a computer program product, comprising a computer program stored on a storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer executes any one of the above-mentioned methods for improving a large language model.

[0012] This embodiment of the present invention estimates the correlation matrix of intermediate layers by measuring the orthogonality of information between layers, thereby accurately measuring the importance of each layer in the model. Based on this estimated correlation matrix, a linear optimization algorithm is then used to design an adaptive sparse allocation strategy. This method selectively prunes features in intermediate layers, achieving fine-grained optimization of large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0014] Figure 1 A flowchart of an embodiment of a method for improving a large language model of the present invention; Figure 2 A flowchart of a process for improving a large language model provided by one embodiment of the present invention; Figure 3 FIG. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0016] It should be noted that, unless there is any conflict, the embodiments and features in the embodiments of this application can be combined with each other.

[0017] The present invention may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0018] In the present invention, "module", "device", "system" and the like refer to related entities applied to a computer, such as hardware, a combination of hardware and software, software or software in execution, etc. Specifically, for example, an element can be, but is not limited to, a process running on a processor, a processor, an object, an executable element, an execution thread, a program and / or a computer. In addition, an application or script running on a server, or a server can also be an element. One or more elements can be in an execution process and / or thread, and an element can be localized on a computer and / or distributed between two or more computers, and can be run by various computer-readable media. An element can also communicate through local and / or remote processes based on signals having one or more data packets, for example, signals from a data packet interacting with another element in a local system, a distributed system, and / or a network over the Internet that interacts with other systems via signals.

[0019] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article or device. In the absence of further limitations, the elements defined by the phrase "include..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.

[0020] The present invention provides an improved method for a large language model, which can be applied to electronic devices. The electronic devices can be computers, servers, or other electronic products, and the present invention is not limited thereto.

[0021] Please refer to Figure 1 , which illustrates an improved method for a large language model provided by an embodiment of the present invention.

[0022] like Figure 1 As shown, in step 101, the correlation matrix between the intermediate layers is calculated using the public data set, the mutual information redundancy between the layers is calculated based on the correlation matrix between the intermediate layers, and the inter-layer redundancy matrix is ​​constructed using the mutual information redundancy between the layers; In step 102, the total redundancy of each layer is calculated based on the redundancy matrix, and a layer importance factor is generated through function conversion; In step 103, an objective function is established based on the layer importance factor, and a linear optimization model for the sparse rate allocation of each layer is constructed with the model target size as a constraint condition; In step 104, a linear programming algorithm is used to solve the optimization model to obtain the optimal sparse rate allocation scheme for each layer; In step 105, differential pruning is performed on the weight matrix of each layer according to the allocation scheme, the weight parameter with the largest amplitude in the weight matrix of each layer is retained, and the remaining parameters are set to zero to obtain a sparse weight matrix.

[0023] In this embodiment, for step 101, the correlation matrix between the intermediate layers is calculated using a public data set, the mutual information redundancy between each layer is calculated based on the correlation matrix between the intermediate layers, and the inter-layer redundancy matrix is ​​constructed using the mutual information redundancy between each layer; for example, a publicly available data set is selected, including the LLaMA-V1 / V2 / V3 family and OPT data, etc., to ensure the verifiability and repeatability of the experiment, and then these data sets are used to estimate the correlation matrix between the intermediate layers. Here, we apply the concept of information orthogonality. We define a mutual information redundancy metric to quantify the inter-layer output matrix X i and X j For a given neural network F = {f1, f2, ...,fL}, where L is the number of layers, f i is the function of the i-th layer. For random samples , calculate the output hidden state of each layer, where the dimension is d0.

[0024] Using kernel density estimation and Monte Carlo sampling method, the mutual information I(x i ; x j ). Define the mutual information redundancy measure: , where X i and X j Denote the output matrices of the i-th layer and the j-th layer respectively. Finally, the mutual information redundancy between each layer is calculated based on the output matrix. The redundancy matrix is ​​Ψ, where .

[0025] Then, for step 102, the total redundancy of each layer is calculated based on the redundancy matrix, and the layer importance factor is generated by function conversion. According to the redundancy matrix Ψ, the total redundancy of each layer is calculated: . And define the layer importance factor: , where μ is a dynamic hyperparameter.

[0026] Then, for step 103, an objective function is established based on the layer importance factor, and a linear optimization model for the sparse rate distribution of each layer is constructed with the model target size as a constraint condition; the maximization objective function and the constraint condition are defined to maximize the objective function: Constraints: ,in is the sparsity rate of the i-th layer, For the sparse rate The number of parameters of the next i-th layer, B is the target model size.

[0027] Then, for step 104, a linear programming algorithm is used to solve the optimization model to obtain the optimal sparse rate allocation scheme for each layer. For example, a linear programming algorithm (such as the simplex method or the interior point method) is used to solve the above optimization problem to obtain the optimal sparse rate allocation for each layer. .

[0028] Finally, for step 105, the weight matrix of each layer is pruned differentially according to the allocation scheme, the weight parameter with the largest amplitude in each layer weight matrix is ​​retained, and the remaining parameters are set to zero to obtain the sparse weight matrix. According to the optimization results, the corresponding sparsity rate is applied to each layer of the model to achieve differential pruning. For example, for each layer, according to the allocated sparsity rate ,reserve The weight parameters with the largest amplitude are given by |Wi|, where |Wi| is the total number of parameters of the layer. The remaining parameters are set to zero to obtain the sparse weight matrix.

[0029] The method in this application's embodiment estimates the correlation matrix of intermediate layers by measuring the orthogonality of information between layers, thereby accurately measuring the importance of each layer in the model. Based on the estimated correlation matrix, a linear optimization algorithm is then used to design an adaptive sparse allocation strategy. This method can selectively prune features in intermediate layers, thereby achieving fine-grained optimization of large language models.

[0030] In some optional embodiments, the concept of information orthogonality is used to calculate the correlation matrix between the intermediate layers, and the mutual information redundancy between the layers is calculated based on the correlation matrix between the intermediate layers, including defining a mutual information redundancy metric, wherein the mutual information redundancy metric is used to quantify the relationship between the output matrices between the layers. Defining the mutual information redundancy metric is used to quantify the output matrix X between the layers. i and X j The mutual information redundancy metric is defined by performing a joint probability distribution analysis on the output matrices of the two intermediate layers based on the principle of information orthogonality. The redundancy is quantified by calculating the statistical independence index of the feature distributions between the two layers. By measuring the information orthogonality between layers, the correlation matrix of the intermediate layers is estimated, thereby accurately measuring the importance of each layer in the model.

[0031] In some optional embodiments, an objective function is established based on the importance factor, including the sum of the products of the sparsity rate of each layer and the importance weight of its corresponding layer, wherein the importance weight is composed of the cumulative importance factors of the current layer and subsequent layers.

[0032] It should be noted that this application applies the concept of information orthogonality to the evaluation of the correlation between intermediate layers for the first time, in order to accurately measure the importance of each layer in the model. This method breaks through the limitations of traditional methods that rely solely on statistical indicators or empirical rules to determine sparsity allocation. Secondly, based on the correlation matrix calculated by information orthogonality, the present invention develops an adaptive sparse allocation strategy that allows differentiated pruning between different layers, thereby achieving fine-grained optimization. This strategy avoids the suboptimal performance problem caused by fixed sparsity ratios in traditional methods. In addition to hierarchical sparse allocation, the present invention further considers adaptive sparse allocation between different components within the layer, further improving the effect of model compression.

[0033] In some optional embodiments, after the construction of the sparse rate allocation for each layer is completed, component-level sparse allocation is also included, and the component-level sparse allocation includes applying the layer importance factor calculation method to the attention head component and feedforward network component within each layer, establishing a component-level redundancy matrix and re-solving the optimization model. On the basis of the layer-level sparse allocation, an adaptive sparse allocation strategy is further applied to different components in each layer (such as attention head, feedforward network, etc.). Repeat the above steps, but change the optimization object from layer to component. Construct sub-optimization models for the attention head component and feedforward network component in each layer respectively, and repeat the mutual information redundancy calculation and the sparse rate allocation process, wherein in the component-level sparse allocation process, differentiated sparse rate upper limit constraints are set for different types of components, wherein the maximum allowable sparse rate of the attention head component is lower than that of the feedforward network component.

[0034] In some optional embodiments, the performance of the sparsified model is evaluated on a test dataset. If the model performance degrades, iterative fine-tuning is performed to recover the performance loss caused by differential pruning. The performance of the sparsified model is evaluated on a test dataset. If necessary, a small number of iterative fine-tuning iterations may be performed to recover the performance loss caused by pruning.

[0035] It should be noted that this application provides a method for evaluating inter-layer correlation, including the specific algorithm and technical details for estimating the correlation matrix between intermediate layers using the concept of information orthogonality. It also provides an adaptive sparse allocation strategy, including how to construct a linear optimization problem and the specific implementation method for solving the optimal sparse rate allocation using a linear programming algorithm. Furthermore, it provides a strategy and implementation mechanism for adaptive sparse allocation between different components within a layer.

[0036] Please refer to Figure 2 , which shows a flow chart of the implementation of the improved method of the large language model of the present invention.

[0037] like Figure 2As shown in S1, we first selected publicly available datasets, including LLaMA-V1 / V2 / V3 families and OPT, to ensure the verifiability and repeatability of the experiment. Then, we used these datasets to estimate the correlation matrix between intermediate layers, where we applied the concept of information orthogonality. We defined a mutual information redundancy metric to quantify the inter-layer output matrix X i and X j The relationship between them.

[0038] S2. Linear optimization problem construction: Mutual information redundancy is calculated between each layer, thus constructing a redundancy matrix. Based on the redundancy matrix Ψ, the total redundancy of each layer is calculated. Define the layer importance factor: , where μ is a dynamic hyperparameter. Define the maximization objective function and constraints, and use a linear programming algorithm (such as the simplex method or the interior point method) to solve the above optimization problem and obtain the optimal sparsity rate allocation for each layer.

[0039] S4. Adaptive sparse allocation: Based on the optimization results, the corresponding sparse rate is applied to each layer of the model to achieve differential pruning. For each layer, according to the allocated sparse rate , retain the weight parameter with the largest amplitude. Set the remaining parameters to zero to obtain the sparse weight matrix.

[0040] S5. Fine-grained component-level sparse allocation (optional): Based on layer-level sparse allocation, further apply an adaptive sparse allocation strategy to different components within each layer (such as attention heads, feedforward networks, etc.). Repeat the above steps, but change the optimization target from layers to components.

[0041] S6 Model Evaluation and Fine-tuning: Evaluate the performance of the sparsified model on the test dataset. If necessary, perform fine-tuning for a small number of iterations to recover the performance loss caused by pruning.

[0042] It should be noted that the methods or steps involved in any of the following three points fall within the scope of protection of this application. 1. Imitate the ideas and specific steps of using information orthogonality to evaluate the correlation between layers. Even if the algorithm itself is not completely copied, it is possible to draw on its core ideas to develop similar technologies. 2. Attempt to reproduce or modify the linear optimization method used in the present invention to formulate an adaptive sparse allocation strategy, especially in terms of how to define the objective function and constraints. 3. On the basis of the existing foundation, develop an adaptive sparse allocation method suitable for internal components of the layer. Even if the specific implementation details are different, the basic ideas and goals may be similar.

[0043] The inventors have discovered that the following methods may be used to address the deficiencies of the prior art: 1. Optimizing intra-layer sparsity: Practitioners may focus on optimizing intra-layer sparsity, attempting to improve model performance by more fine-grainedly distributing sparsity within a single layer. This attempts to optimize the model without extensive retraining.

[0044] 2. Empirical Tuning and Hyperparameter Optimization: Engineers may rely more on empirical tuning and hyperparameter optimization techniques, such as grid search or Bayesian optimization, to find the best sparsity configuration for a specific model.

[0045] 3. Cross-layer global sparse allocation: To address the problems caused by local optimization, a cross-layer global sparse allocation strategy may be designed. This strategy is no longer limited to the optimization of a single layer, but considers the sparsity allocation of the entire model as a whole to avoid local optimal solutions.

[0046] In response to the defects of the prior art, this application provides an improved method for a large language model, which involves a new perspective on the correlation between layers within the model, which requires breaking away from traditional thinking patterns, understanding the concept of information orthogonality, and applying it to the calculation of the inter-layer correlation matrix. This requires not only a deep theoretical background, but also innovative thinking. This method may be more complicated in design, and it is necessary to combine the theoretical information orthogonality with the inter-layer correlation in the actual model, and to solve the linear programming problem to optimize the sparsity allocation, which may face many challenges in actual operation.

[0047] It should be noted that this paper proposes a large-scale language model compression method based on adaptive layer sparsity (ALS). The core concept of this invention is to quantify the importance of each layer by evaluating inter-layer correlations and to develop an adaptive sparse allocation strategy using a linear optimization algorithm. The specific technical solution is as follows: During our research, we noticed that existing pruning techniques generally adopt a uniform pruning ratio, ignoring the different contributions of different layers to the final model performance. This leads to suboptimal model performance and excessive complexity. Furthermore, these techniques often fail to fully exploit the information correlation between layers. To this end, we proposed a new ALS method to overcome these problems.

[0048] Initial Design and Principle: The design concept of the ALS method is to estimate the correlation matrix of intermediate layers by measuring the information orthogonality between layers, thereby accurately measuring the importance of each layer in the model. Then, based on the estimated correlation matrix, we use a linear optimization algorithm to develop an adaptive sparse allocation strategy. This approach allows us to selectively prune features in intermediate layers, thereby achieving fine-grained optimization of large language models. Inter-layer Correlation Estimation: Leveraging the concept of information orthogonality, we estimate the correlation matrix between intermediate layers.

[0049] The specific steps are: S1. Inter-layer correlation evaluation: S1.1 First, publicly available datasets, including the LLaMA-V1 / V2 / V3 family and OPT, were selected to ensure the verifiability and reproducibility of the experiments.

[0050] S1.2 Next, we use these datasets to estimate the correlation matrix between the intermediate layers, where the concept of information orthogonality is applied. We define a mutual information redundancy measure to quantify the inter-layer output matrix X i and X j The relationship between them.

[0051] •For a given neural network F = {f1, f2, ..., fL}, where L is the number of layers, f i is the function of the i-th layer.

[0052] • For random samples , calculate the output hidden state of each layer, where the dimension is d0.

[0053] •Use kernel density estimation and Monte Carlo sampling method to approximate the mutual information I(X i ; X j ).

[0054] • Define the mutual information redundancy measure: , where X i and X j Represent the output matrices of the i-th layer and the j-th layer respectively.

[0055] S2. Linear optimization problem construction: S2.1: Mutual information redundancy is calculated between each layer, so a redundancy matrix Ψ is constructed, where .

[0056] S2.2: Based on the redundancy matrix Ψ, calculate the total redundancy of each layer: S2.3: Define layer importance factors: , where μ is a dynamic hyperparameter.

[0057] S2.4: Construct a linear optimization problem: Maximize the objective function: Constraints: where is the sparsity rate of layer i, is the number of parameters of the i-th layer under the sparse rate, and B is the target model size.

[0058] S3. Solve the linear optimization problem: Use a linear programming algorithm (such as the simplex method or the interior point method) to solve the above optimization problem and obtain the optimal sparsity rate distribution for each layer. .

[0059] S4. Adaptive sparse allocation: Based on the optimization results, apply the corresponding sparse rate to each layer of the model to achieve differentiated pruning. The specific steps are: S4.1: For each layer, according to the assigned sparsity rate ,reserve The weight parameters with the largest amplitude are given by |Wi|, where |Wi| is the total number of parameters of the layer. The remaining parameters are set to zero to obtain the sparse weight matrix.

[0060] S5 Fine-grained Component-Level Sparse Allocation (Optional): Based on layer-level sparse allocation, we further apply an adaptive sparse allocation strategy to different components within each layer (such as attention heads, feedforward networks, etc.). Repeat the above steps, but change the optimization target from layers to components.

[0061] S6 Model Evaluation and Fine-tuning: Evaluate the performance of the sparsified model on the test dataset. If necessary, perform fine-tuning for a small number of iterations to recover the performance loss caused by pruning.

[0062] Each step of this approach is carefully designed to address specific challenges in pruning large language models. Through this comprehensive approach, this application can significantly reduce model size and improve inference efficiency while maintaining model performance.

[0063] The method in this application achieves parameter compression on large models through pruning. Reducing model parameters reduces the memory usage of large models, making it possible to deploy large models on-device. For example, when the total pruning rate is 50%, the solution automatically calculates the pruning rate of each layer, ultimately compressing the 7B parameters of the large model Agent to 3B, enabling deployment on the vehicle side. While ensuring that the inference performance of the model after compression is comparable to that before compression (each indicator only decreases by 2% to 5%), it also reduces memory usage by 50% and doubles the inference speed, achieving smoother intelligent vehicle-side responses.

[0064] It should be noted that the method of this application can be implemented using the NVIDIA A100 GPU. This method belongs to the compression at the large model level, belongs to the software level, is not strongly bound to the hardware, and can be applied to other GPUs, NPUs, etc. Since the solution of this application belongs to the large model level, no hardware optimization is made. When it comes to hardware deployment of large models, there is no difference with other solutions. The storage of sparse matrices after pruning is no different from the usual large model storage, and supports storage in formats such as pt, safetensors, and bin. This application does not limit this.

[0065] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of combined actions, but those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0066] In some embodiments, an embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores one or more programs including execution instructions, and the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute any of the above-mentioned large language model improvement methods of the present invention.

[0067] In some embodiments, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes any of the above-mentioned methods for improving a large language model.

[0068] In some embodiments, an embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the improved method of the large language model.

[0069] Figure 3 This is a hardware structure diagram of an electronic device for executing an improved method for a large language model provided by another embodiment of the present application. Figure 3 As shown, the device includes: One or more processors 310 and memory 320, Figure 3 A processor 310 is taken as an example.

[0070] The apparatus for executing the improved method for a large language model may further include: an input device 330 and an output device 340 .

[0071] The processor 310, the memory 320, the input device 330 and the output device 340 may be connected via a bus or other means. Figure 3 The bus connection is taken as an example.

[0072] Memory 320, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as the program instructions / modules corresponding to the method for improving a large language model in the embodiments of the present application. Processor 310 executes the non-volatile software programs, instructions, and modules stored in memory 320 to execute various functional applications and data processing of the server, thereby implementing the method for improving a large language model in the above-mentioned method embodiment.

[0073] The memory 320 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the improved device of the large language model, etc. In addition, the memory 320 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 320 may optionally include a memory remotely located relative to the processor 310, and these remote memories may be connected to the improved device of the large language model via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0074] The input device 330 can receive input digital or character information and generate signals related to user settings and function control of the large language model improvement device. The output device 340 can include a display device such as a display screen.

[0075] The one or more modules are stored in the memory 320 , and when executed by the one or more processors 310 , perform the method for improving the large language model in any of the above method embodiments.

[0076] The above-mentioned product can execute the method provided in the embodiment of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of this application.

[0077] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and their primary purpose is to provide voice and data communications. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0078] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers and have computing and processing capabilities, and generally also have mobile Internet access. These terminals include PDAs, MIDs, and UMPCs.

[0079] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0080] (4) Other onboard electronic devices with data interaction functions, such as onboard computer devices installed in vehicles.

[0081] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0082] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a general hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the relevant technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for improving a large language model, comprising: Calculate the correlation matrix between the intermediate layers using a public data set, calculate the mutual information redundancy between the layers based on the correlation matrix between the intermediate layers, and construct an inter-layer redundancy matrix using the mutual information redundancy between the layers; Calculating the total redundancy of each layer based on the redundancy matrix and generating a layer importance factor through function conversion; An objective function is established based on the layer importance factor, and a linear optimization model for the sparsity rate distribution of each layer is constructed with the model target size as a constraint condition; A linear programming algorithm is used to solve the optimization model to obtain the optimal sparse rate allocation scheme for each layer; The weight matrix of each layer is differentially pruned according to the allocation scheme, the weight parameter with the largest amplitude in the weight matrix of each layer is retained, and the remaining parameters are set to zero to obtain a sparse weight matrix.

2. The method according to claim 1, wherein The correlation matrix between the intermediate layers is calculated using the concept of information orthogonality, and the mutual information redundancy between the layers is calculated based on the correlation matrix between the intermediate layers, including: The mutual information redundancy metric is defined, and the mutual information redundancy metric is used to quantify the relationship between output matrices between layers.

3. The method according to claim 2, wherein: Defining the mutual information redundancy metric includes: Based on the principle of information orthogonality, the joint probability distribution of the output matrices of the two intermediate layers is analyzed, and the redundancy is quantified by calculating the statistical independence index of the feature distribution between the two layers.

4. The method according to claim 1, wherein The establishing of an objective function based on the importance factor comprises: The sum of the products of the sparsity rate of each layer and the importance weight of its corresponding layer, wherein the importance weight is composed of the cumulative importance factors of the current layer and subsequent layers.

5. The method according to claim 1, wherein After the construction of the sparse rate allocation of each layer is completed, component-level sparse allocation is also included, and the component-level sparse allocation includes: The layer importance factor calculation method is applied to the attention head component and feedforward network component within each layer, a component-level redundancy matrix is ​​established, and the optimization model is re-solved.

6. The method according to claim 5, wherein: The method further comprises: A sub-optimization model is constructed for the attention head component and the feedforward network component in each layer, and the mutual information redundancy calculation and the sparsity rate allocation process are repeatedly performed. In the component-level sparse allocation process, differentiated sparsity rate upper limit constraints are set for different types of components, and the maximum allowable sparsity rate of the attention head component is lower than that of the feedforward network component.

7. The method according to claim 1, wherein The method further comprises: The performance of the sparsified model is evaluated on the test dataset. If the model performance degrades, iterative fine-tuning is performed to recover the performance loss of the model caused by differential pruning.

8. The method according to claim 1, wherein The public datasets include: LLaMA-V1 / V2 / V3 family and OPT data.

9. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 8.

10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images