Text processing method, model processing method and electronic equipment

Abstract

The invention discloses a text processing method, a model processing method and electronic equipment, and relates to the field of large model technologies and model processing. The method comprises the steps of obtaining a to-be-processed text; reasoning the to-be-processed text by using the target model to obtain a text processing result; wherein the target weight matrix of the target model is obtained by performing pruning operation on an initial weight matrix of the target model according to the target column, the target column is obtained by performing pruning evaluation on the initial weight matrix based on an evaluation matrix, and the evaluation matrix is used for quantifying the influence of weight change in the initial weight matrix on the performance of the target model. The technical problem that the model processing effect is poor in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of large model technology and model processing, and specifically, to a text processing method, a model processing method and an electronic device. Background Art

[0002] The number of parameters in deep learning models is often as high as hundreds of millions, which slows down the online reasoning speed of the model and makes it unable to adapt to the current hardware environment. To solve this problem, various structured pruning methods are proposed in related technologies. Through model processing and quantization, the computational complexity of the model in the reasoning stage is further reduced, thereby reducing the number of model parameters and improving the model reasoning speed.

[0003] However, the structured pruning methods in related technologies often require the introduction of auxiliary tasks during the model training phase, which slows down the model training speed and affects the model performance. In addition, in the model processing process, related technologies usually reduce the number of model parameters by changing the size or number of model weights, while changes in hyperparameters will cause large fluctuations in the performance of the model, resulting in poor model processing results.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a text processing method, a model processing method and an electronic device to at least solve the technical problem of poor model processing effect in the related art.

[0006] According to one aspect of an embodiment of the present application, a text processing method is provided, including: obtaining a text to be processed; using a target model to infer the text to be processed to obtain a text processing result; wherein a target weight matrix of the target model is obtained by pruning an initial weight matrix of the target model according to a target column, and the target column is obtained by pruning and evaluating the initial weight matrix based on an evaluation matrix, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model.

[0007] According to another aspect of an embodiment of the present application, a model processing method is also provided, including: obtaining an initial weight matrix of a target model and an evaluation matrix corresponding to the initial weight matrix, wherein the initial weight matrix is ​​used to represent the initial weight matrix to be pruned in the target model, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model; performing pruning and evaluation on the initial weight matrix based on the evaluation matrix to obtain a target column; performing pruning operations on the initial weight matrix based on the target column to obtain a target weight matrix.

[0008] According to another aspect of an embodiment of the present application, a text processing method is also provided, including: responding to an input instruction on an operation interface, displaying a text to be processed on the operation interface; responding to a text processing instruction on the operation interface, displaying a text processing result on the operation interface, wherein the text processing result is obtained by reasoning about the text to be processed using a target model, and the target weight matrix of the target model is obtained by pruning the initial weight matrix of the target model according to a target column, and the target column is obtained by pruning and evaluating the initial weight matrix based on an evaluation matrix, the initial weight matrix is ​​used to represent the initial weight matrix to be pruned in the target model, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model.

[0009] According to another aspect of an embodiment of the present application, a text processing method is also provided, including: obtaining a text to be processed by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the text to be processed; using a target model to infer the text to be processed to obtain a text processing result, wherein a target weight matrix of the target model is obtained by pruning an initial weight matrix of the target model according to a target column, and the target column is obtained by pruning and evaluating the initial weight matrix based on an evaluation matrix, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model; outputting the text processing result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the text processing result.

[0010] According to another aspect of the embodiments of the present application, an electronic device is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes any one of the methods in the above embodiments when running.

[0011] According to another aspect of the embodiments of the present application, a computer terminal is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the method in each embodiment of the present application is executed when the program is running.

[0012] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in each embodiment of the present application.

[0013] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0014] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0015] According to another aspect of the embodiments of the present application, a computer program is also provided, and when the computer program is executed by a processor, the methods in the various embodiments of the present application are implemented.

[0016] In an embodiment of the present application, a text to be processed is obtained; the target model is used to perform reasoning on the text to be processed to obtain a text processing result; wherein, the target weight matrix of the target model is obtained by pruning the initial weight matrix of the target model according to the target column, and the target column is obtained by pruning and evaluating the initial weight matrix based on the evaluation matrix, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model, thereby achieving the purpose of improving the model processing effect; it is easy to notice that the impact of weight changes on model performance can be quantified based on the evaluation matrix, so as to accurately identify the target column where the weights with less impact on model performance are located, and the initial weight matrix of the target model can be pruned according to the target column, thereby reducing the computational complexity and memory usage of the model while reducing the impact of the pruning operation on the model performance, thereby improving the model processing effect on the text to be processed, thereby solving the technical problem of poor model processing effect in related technologies.

[0017] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a schematic diagram of an application scenario of a text processing method according to an embodiment of the present application;

[0020] Figure 2 is a flowchart of a text processing method according to an embodiment of the present application;

[0021] Figure 3 is a structural schematic diagram of a model processing according to an embodiment of the present application;

[0022] Figure 4 is a flow chart of a model processing method according to an embodiment of the present application;

[0023] Figure 5is a flowchart of a text processing method according to an embodiment of the present application;

[0024] Figure 6 is a flowchart of a text processing method according to an embodiment of the present application;

[0025] Figure 7 is a schematic diagram of a text processing device according to an embodiment of the present application;

[0026] Figure 8 is a schematic diagram of a model processing device according to an embodiment of the present application;

[0027] Fig. 9 is a schematic diagram of a text processing device according to an embodiment of the present application;

[0028] Fig.10 is a schematic diagram of a text processing device according to an embodiment of the present application;

[0029] Fig.11 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] The technical solution provided in this application is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained through large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (Large Language Model, LLM), multi-modal pre-training model (multi-modal pre-training model), etc.

[0033] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples, so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, etc. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, machine translation, etc. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiments of the present application, text processing through the target model in a text processing scenario is taken as an example for explanation,

[0034] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0035] A large model compression framework that was originally designed to simulate precise resection in surgery and aims to remove redundant weights from neural networks while minimizing the impact on model performance. It mainly uses two steps, error estimation and parameter compensation, to determine and remove unimportant model weights, thereby reducing the model size while maintaining high accuracy.

[0036] Layer-wise compression is a compression strategy that prunes or quantizes parameters layer by layer in a neural network to reduce the size and computational cost of the model. Unlike overall compression, layer-wise compression focuses more on local improvements.

[0037] Hessian Matrix, in mathematics and machine learning, the Hessian matrix is ​​a square matrix composed of second-order partial derivatives, which is used to describe the local curvature of the loss function at a certain point. It is crucial to understanding the properties of the function and improving the performance of the algorithm.

[0038] Cholesky decomposition is a method of decomposing a positive definite matrix into the product of a lower triangular matrix and its transpose. It is often used to solve linear equations, square roots of matrices, and related numerical operations. In compression algorithms, it can be used to solve or approximate the inverse of the Hessian matrix.

[0039] Attention Module, in the Large Language Model (LLM), the attention module calculates the relevance of different input parts to assign different weights, thereby achieving dynamic focus on the input. It is mainly composed of multiple attention heads, each of which is responsible for capturing different aspects of the input.

[0040] According to an embodiment of the present application, a text processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] Considering the huge number of model parameters of the large model and the limited computing resources of the mobile terminal, the above text processing method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1 is a schematic diagram of an application scenario of a text processing method according to an embodiment of the present application. Figure 1 In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.

[0042] In an embodiment of the present application, a system composed of a client device and a server can perform the following steps: the client device generates a text to be processed. The server uses the target model to reason about the text to be processed and obtains a text processing result. It should be noted that, if the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiment of the present application can be performed in the client device.

[0043] Under the above operating environment, this application provides Figure 2 The text processing method shown. Figure 2 is a flowchart of a text processing method according to an embodiment of the present application. Figure 2 As shown, the method may include the following steps:

[0044] Step S202, obtaining the text to be processed;

[0045] The above-mentioned text to be processed can be text input that needs to be understood and processed by the language model, and it can be any form of text such as search queries, questions, comments, descriptions, etc.

[0046] In an optional embodiment, by obtaining the text to be processed, the input flexibility of the method is ensured, and it can adapt to various text data sources, such as social media, news articles or user input, providing a broad application basis for subsequent text processing.

[0047] The above steps can receive any form of text input, whether it is a search query, question, comment or description, ensuring the input flexibility and wide applicability of the method, and being able to process text information from different data sources, such as social media, news articles or data directly entered by users. The flexibility of this step provides a solid foundation for subsequent text processing, ensuring that the method can adapt to diverse and changing text data, and providing a wide range of application scenarios for subsequent reasoning and improvement.

[0048] Step S204: use the target model to infer the text to be processed to obtain a text processing result.

[0049] Among them, the target weight matrix of the target model is obtained by pruning the initial weight matrix of the target model according to the target column, and the target column is obtained by pruning and evaluating the initial weight matrix based on the evaluation matrix. The evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model.

[0050] The target model mentioned above refers to a model version that has been pruned and has higher efficiency and resource utilization, and is used to perform reasoning and processing of text data.

[0051] The above-mentioned target weight matrix can be the core component of the target model, which is obtained by pruning and improving the initial weight matrix of the model. By reducing redundant weights, the calculation efficiency and the lightweight degree of the model are improved.

[0052] The target columns mentioned above may be specific columns identified and selected from the weight matrix during the model pruning process, which are evaluated as having a small contribution to the model performance or can be safely removed.

[0053] The above-mentioned evaluation matrix can be a matrix used to quantify the influence of the change of each weight in the weight matrix on the overall performance of the model, and is a key tool for pruning evaluation.

[0054] The target weight matrix of the target model is obtained by pruning the initial weight matrix of the target model based on the target column. The target column is obtained by pruning and evaluating the initial weight matrix based on the evaluation matrix. The evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model. By intelligently identifying non-critical parameters in the model and reducing redundancy through pruning, the demand for computing resources is reduced while maintaining model performance.

[0055] The above evaluation matrix can be a Hessian matrix, which is a square matrix composed of second-order partial derivatives, used to describe the curvature of the loss function (usually refers to the prediction error or objective function in machine learning) relative to the model weights. In the training process of neural networks or large language models, the loss function is a standard for measuring the difference between the model's predicted results and the actual results. Each element of the Hessian matrix represents the second-order partial derivative of the loss function with respect to two different parameters in the model weights, that is, it reflects the degree of influence of weight changes on the loss function changes.

[0056] When pruning large language models to reduce online inference latency, the Hessian matrix is ​​used as an evaluation matrix to help determine which weights have the least impact on model performance when pruned. Specifically, the Hessian matrix can be used to estimate the degree of degradation in model performance after pruning specific weights. This is because the inverse of the Hessian matrix (H^-1) can be viewed as a sensitivity indicator of the impact of weight changes on model output. By calculating the inverse of the Hessian matrix, the weights that have the least impact on model output, that is, the weights that will cause the least performance degradation after pruning, can be found, thereby determining the target columns for pruning.

[0057] In the text processing method of the present application, when the target model processes the text to be processed, its target weight matrix is ​​obtained through pruning operation. This operation is based on the analysis of the Hessian matrix, and the pruning target columns are determined by evaluating the matrix, that is, those weight columns that have the least impact on the model performance after removal. After pruning, the computational complexity of the model is reduced, thereby reducing the delay during online reasoning and improving the speed of text processing.

[0058] This application proposes a low-cost structured pruning method. First, by obtaining the user's text to be processed, and then using the evaluation matrix to prune and evaluate the initial weight matrix of the large model, the matrix of the impact of weight changes on model performance is quantified, and the target column, that is, the weight column with less impact on model performance, is identified. Then, based on the target column, the initial weight matrix is ​​pruned to generate a target weight matrix, and then the improved target model is obtained. This series of operations ensures that the target model reduces the computational complexity and memory usage while maintaining high performance, thereby reducing the delay of online reasoning.

[0059] First, the text to be processed is obtained to provide input for the model. Then, the pruned target model is used to infer the text to obtain the text processing result. By pruning the model, the target columns that have little impact on the model performance are removed from the initial weight matrix to generate the target weight matrix, thereby obtaining a target model with more efficient calculation, lower resource consumption and smaller latency.

[0060] Through the above pruning method, the target model is designed to reduce the consumption of computing resources and inference latency while maintaining performance. Therefore, when the target model is used to infer the acquired text to be processed, the text processing results can be obtained at a faster speed and with less resource overhead. This process not only improves the efficiency of the model in processing text, but also ensures the quality of the processing results, reflecting the significant effect of pruning operations in improving model performance and resource utilization.

[0061] From obtaining the text to be processed to reasoning with the target model after pruning, the speed and efficiency of text processing can be improved while maintaining the quality of the processing results. Specifically, the pruning operation of the target model reduces the computational complexity and memory usage of the model, which means that the speed of online reasoning will be significantly improved and latency will be reduced, thereby improving the user response experience in scenarios such as search systems and customer service conversations. At the same time, due to the quantitative evaluation of the impact of performance and parameter compensation during the pruning process, the loss of model performance is minimized, ensuring the accuracy of the processing results and the ability to understand semantics, so that the improved model can provide high-quality text processing results when processing diverse text data to meet the needs of different scenarios.

[0062] Through the model pruning method proposed in this application, the target model can significantly reduce the delay of online reasoning while maintaining high performance, thereby improving the response speed of the search system and user experience. Specifically, the target weight matrix of the target model is obtained by identifying and pruning weights (i.e., target columns) that have little impact on performance. This not only reduces the computational complexity of the model and reduces unnecessary consumption of computing resources, but also reduces the negative impact of pruning on model performance through a parameter compensation mechanism, thereby achieving efficient compression and performance retention of the model without relying on large-scale post-training.

[0063] Exemplary, take the new query "buy a new laptop" as an example. In traditional search systems, due to the lack of query cache, for such new queries, the system needs to deploy a large language model online for real-time reasoning to perform query rewriting, but this method will result in higher latency. Using the pruning method of the present application, the system first evaluates the target columns with the least impact on performance from the initial weight matrix of the target model, and then prunes these target columns to form a target weight matrix. The pruned target model can greatly reduce the latency of online reasoning while ensuring the quality of query rewriting, such as from a few seconds to hundreds of milliseconds, significantly improving the response speed of the search system and user experience. In this process, the target model not only has a faster reasoning speed, but also ensures minimal performance loss of the model through a parameter compensation mechanism, achieving a dual improvement in efficiency and performance.

[0064] Through the above steps, the text to be processed is obtained; the target model is used to infer the text to be processed to obtain the text processing result; wherein, the target weight matrix of the target model is obtained by pruning the initial weight matrix of the target model according to the target column, and the target column is obtained by pruning and evaluating the initial weight matrix based on the evaluation matrix, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model, thereby achieving the purpose of improving the model processing effect; it is easy to notice that the impact of weight changes on model performance can be quantified based on the evaluation matrix, so as to accurately identify the target column where the weights with less impact on model performance are located, and the initial weight matrix of the target model can be pruned according to the target column, thereby reducing the computational complexity and memory usage of the model while reducing the impact of pruning operations on model performance, thereby improving the model processing effect on the text to be processed, thereby solving the technical problem of poor model processing effect in related technologies.

[0065] In the above embodiment of the present application, the method also includes: performing a pruning operation on the initial weight matrix based on the target column to obtain an intermediate weight matrix; using the intermediate weight matrix as the initial weight matrix, performing a pruning evaluation on the initial weight matrix based on the evaluation matrix to obtain a target column, performing a pruning operation on the initial weight matrix based on the target column to obtain an intermediate weight matrix, until the number of iterations reaches the pruning number; using the intermediate weight matrix as the target weight matrix.

[0066] In the process of model pruning, the target columns, i.e., those weight columns that have relatively little impact on model performance, are first determined based on the evaluation matrix. Then, the initial weight matrix is ​​pruned to remove these target columns, thereby obtaining an intermediate weight matrix. The intermediate weight matrix is ​​used as the new starting point for iterative pruning. The intermediate weight matrix obtained in the previous step is used as the new initial weight matrix. The pruning evaluation and pruning operations based on the evaluation matrix are repeated. This process will continue to iterate until the predetermined number of pruning is reached.

[0067] When the number of iterations reaches the number of prunings determined by the hardware status of the computing device where the target model is to be deployed, the iterative pruning process ends. The final intermediate weight matrix is ​​the target weight matrix after all pruning operations, which is used to construct the final improved target model.

[0068] For example, consider an e-commerce query rewriting system that needs to run in real time on resource-constrained mobile devices to process user search queries. For the relevant modules in the model, first determine a reasonable number of pruning based on the hardware conditions of the mobile device (such as processor computing power, memory size, battery life, etc.) to minimize computing resource consumption and inference latency while ensuring performance. Then, according to the above iterative pruning process, gradually identify and prune the target columns that contribute less to the model performance, and generate an intermediate weight matrix. This process will continue until the predetermined number of pruning is reached.

[0069] The iterative pruning process gradually improves the model structure through cyclic iterations. In each iteration, the target column is determined for pruning based on the evaluation matrix, and then the pruned intermediate weight matrix is ​​used as the starting point for the next iteration. This process continues until the number of prunings determined based on the hardware status is reached. This method ensures the gradual improvement of the model during the pruning process, and can achieve precise control of the model size and complexity according to the characteristics of the target computing device, so that the model can be more adapted to the specific hardware environment while maintaining a certain performance, reducing inference delays and improving efficiency.

[0070] The implementation of the iterative pruning process can improve the operating efficiency and performance stability of the target model on resource-constrained hardware, while reducing inference latency and improving user experience. In this way, not only the problem of excessive online inference latency of large models is solved, but also the model structure and computing resource consumption are improved, providing an effective solution strategy for the deployment and efficient operation of the model in different hardware environments.

[0071] For example, taking the e-commerce query rewriting system as an example, when the user first enters a long-tail query "find a new laptop", the system needs to deploy a large language model online for real-time query understanding and rewriting. In order to solve the high latency problem caused by online reasoning, the number of pruning of related modules is first determined based on the hardware status of the mobile device, such as processor performance, memory size, etc., assuming 8 heads. Then, the initial weight matrix of the relevant modules of the model is pruned and evaluated, and the attention heads with less impact on performance are identified, and pruning operations are performed to obtain the intermediate weight matrix. This intermediate weight matrix is ​​then used as the new initial weight matrix, and pruning evaluation and pruning operations are continued until the predetermined number of pruning is reached. The final target weight matrix will be used to construct the improved target model for real-time processing of user search queries, significantly reducing the inference delay and improving the search experience. At the same time, the performance loss of the model is controlled within an acceptable range, achieving a balance between efficiency and performance.

[0072] This iterative pruning process not only reduces computational complexity and resource consumption, but also reduces the negative impact of pruning on performance through a parameter compensation mechanism, ensuring that the model can adapt to the hardware limitations of mobile devices while maintaining high performance, achieving fast response and high-quality search results, and improving the search user experience.

[0073] In the above embodiment of the present application, the method also includes: in response to the initial weight matrix being a first weight matrix, determining multiple attention units corresponding to the first weight matrix, wherein the weight values ​​of the first weight matrix are used to describe the attention allocation intensity of the multiple attention units; performing pruning evaluation on the multiple attention units based on the evaluation matrix to obtain a target attention unit, wherein the target attention unit is the attention unit that has the least impact on the target model when performing pruning operations on the multiple attention units; and determining the target column corresponding to the target attention unit from the first weight matrix.

[0074] In an optional embodiment, for the first weight matrix of the target model, the weight values ​​of the first weight matrix reflect the intensity of attention distribution among multiple attention units. The intensity of attention distribution determines the weight of the input data during the processing and is the key to the model's understanding and generation of text. These attention units are pruned and evaluated based on the evaluation matrix, which quantifies the impact of weight changes on model performance. Through pruning evaluation, the attention unit with the least impact on model performance can be identified as the target attention unit, ready for pruning. Determine the target columns related to the target attention unit from the first weight matrix. These target columns are determined based on pruning evaluation and represent weight information that can be safely removed without significantly affecting model performance.

[0075] Exemplarily, in the search rewriting system of the large language model, the attention module is the core component of the model, which is responsible for processing the attention allocation between the query terms and the product description. In order to reduce the online reasoning delay, a pruning strategy is designed specifically for the attention module. Specific to the pruning details of the attention module, it is assumed that the weight values ​​in the initial weight matrix (first weight matrix) describe the attention allocation intensity of each attention unit (such as the attention head). The pruning evaluation measures the impact of each attention unit on the model performance after pruning through the evaluation matrix, aiming to find the attention unit with the least impact on the performance, that is, the target attention unit. Once the target attention unit is determined, the next step is to determine the corresponding target columns from the first weight matrix. These columns contain the weight information of the target attention unit and will be used for further pruning operations.

[0076] Overall, this process is an implementation method for structured pruning of attention modules of large models, aiming to reduce model complexity and online inference latency while ensuring stable performance of the pruned model. By identifying and removing target attention units that contribute relatively little to the overall performance of the model, the consumption of computing resources can be effectively reduced and the response speed of the search system can be improved without significantly affecting the quality of search results.

[0077] The above pruning strategy can significantly improve online reasoning latency while maintaining model performance, which is especially effective for processing new queries that appear for the first time, allowing the search system to quickly respond to user input and improve user experience. In addition, this strategy also reduces the computational complexity and memory usage of the model, making the model more suitable for running on resource-constrained devices, such as mobile devices, thereby expanding the application scope of the model and improving the overall efficiency of the search system.

[0078] Exemplarily, in a search scenario, the user enters a query "the latest thin and light laptop" for the first time, and the system needs to use a large language model online for real-time query understanding and rewriting. In order to improve the online reasoning delay, the system uses the pruning method of this application to prune the attention module in the model. First, the attention allocation intensity of each attention unit is identified from the first weight matrix, and then based on the evaluation matrix, the system evaluates the attention unit (target attention unit) with the least impact on the model performance. Assuming that the attention heads that can be safely removed are the 3rd and 6th heads after evaluation, the next step is to determine the target columns corresponding to the 3rd and 6th attention heads from the first weight matrix. Finally, by removing these target columns, a lightweight intermediate weight matrix is ​​formed, and then the improved model is constructed for real-time processing of user queries. The pruned model can significantly reduce the online reasoning delay and improve the search response speed. At the same time, the performance of the model is maintained through the parameter compensation mechanism, achieving a dual improvement in efficiency and effect. The above values ​​are only for example and can be adjusted according to actual conditions.

[0079] In the above embodiment of the present application, a plurality of attention units are pruned and evaluated based on an evaluation matrix to obtain a target attention unit, including: based on the attention head sizes of the plurality of attention units, the evaluation matrix is ​​decomposed along the diagonal direction of the evaluation matrix to obtain a plurality of sub-matrices, wherein the plurality of sub-matrices correspond one-to-one to the plurality of attention units; and a plurality of attention units are pruned and evaluated based on the plurality of sub-matrices to obtain a target attention unit.

[0080] The above attention head size is the size of the attention head in the attention module, which determines the number of input features that each attention head can process. Understanding the attention head size is crucial for subsequent pruning evaluation.

[0081] The above-mentioned decomposition of the evaluation matrix is ​​to perform a specific decomposition operation on the evaluation matrix based on the size of the attention head. This decomposition is performed along the diagonal direction of the evaluation matrix, and the multiple sub-matrices obtained correspond to the multiple attention units of the attention module one by one, and each sub-matrix reflects the performance impact of the corresponding attention unit.

[0082] The above attention unit pruning evaluation is to use the decomposed sub-matrix to perform pruning evaluation on multiple attention units. The goal of the evaluation is to find the attention unit that has the least impact on the model performance, that is, the target attention unit, as the object of the pruning operation.

[0083] In the target model, the attention module is the core part of processing text information, which contains multiple attention units (such as attention heads), each of which is responsible for processing different dimensions of the text. In order to improve model performance and reduce online inference latency, this application proposes an attention unit pruning method based on an evaluation matrix.

[0084] The size of the attention head of the attention module can be determined, which helps to understand the scope of each attention head processing input information. Next, the evaluation matrix is ​​decomposed along the diagonal direction to obtain the sub-matrices corresponding to each attention unit. This decomposition operation is based on column reorganization, which reorders the weight columns related to each attention head to the front of the matrix, and also reorders the inverse of the Hessian matrix accordingly, thereby calculating the decomposition matrix of the inverse of the reordered Hessian matrix.

[0085] Based on these sub-matrices, the system can evaluate the impact of pruning each attention unit on the overall performance of the model, thereby identifying the target attention unit, that is, the attention unit whose pruning operation has the least impact on the model. This evaluation process ensures that the pruning operation can effectively reduce the computational complexity of the model while maintaining the performance of the model to the greatest extent, providing a scientific basis for subsequent pruning and model improvement.

[0086] By evaluating the decomposition of the evaluation matrix and the pruning evaluation of the attention unit, the target attention unit is determined, providing guidance for the structured pruning and improvement of the model. This not only reduces the computing resource consumption of the model and reduces the online reasoning delay, but also ensures the minimum performance loss after model pruning through sophisticated evaluation and pruning strategies, improving the operating efficiency and performance stability of the model in a resource-constrained environment.

[0087] Through the above steps, the efficiency and response speed of online reasoning of large language models can be significantly improved. When processing new queries that appear for the first time, accurate search results or rewriting suggestions can be quickly provided to improve user experience. At the same time, due to the detailed evaluation and parameter compensation during the pruning process, the performance of the model is basically retained, ensuring that the quality of search results is not affected, and achieving a balance between efficiency and performance.

[0088] Exemplarily, first, the size of the attention head of the attention module is determined, and then the evaluation matrix is ​​decomposed to obtain a submatrix corresponding to each attention head. Based on these submatrices, the system evaluates the impact of pruning each attention head on the model performance, and identifies the attention head (target attention unit) with the least impact on performance, such as the fourth attention head, as the object of the pruning operation. By pruning this target attention unit and performing parameter compensation on the remaining attention units, the system constructs an improved model, significantly reduces the delay of online reasoning, and improves the search response speed. At the same time, the pruning evaluation ensures that the performance loss of the model after pruning is minimized, thereby achieving a dual improvement in model performance and efficiency on resource-constrained mobile devices, greatly improving the user search experience. The above values ​​are only for example and are not specifically limited.

[0089] The above examples demonstrate the specific application of this application in the search scenario. Through refined attention unit pruning evaluation, the model structure is effectively improved, the consumption of computing resources is reduced, the processing speed is accelerated, and the accuracy and quality of the processing results are guaranteed. It is an effective strategy for large language models to operate efficiently in resource-constrained environments.

[0090] In the above embodiment of the present application, the method also includes: in response to the initial weight matrix being the second weight matrix, grouping the second weight matrix by column based on the dynamic parameter value to obtain multiple groups of columns, wherein the weight values ​​in the second weight matrix are used to describe the information transmission intensity of multiple network layers corresponding to the target model, and the dynamic parameter value is used to represent the parameter value that changes dynamically during the pruning of the target model; performing pruning evaluation on the multiple groups of columns based on the evaluation matrix to obtain the target column corresponding to the target group.

[0091] The above-mentioned multiple network layers may be network layers used to implement specific reasoning steps.

[0092] The pruning operation of the above-mentioned multiple network layers is mainly achieved by dynamically adjusting the pruning parameters and using the evaluation matrix to perform pruning evaluation. The information transmission strength can be determined. The weight values ​​in the second weight matrix represent the information transmission strength between multiple network layers, which is the basis for the model to process information and make predictions. The dynamic parameter values ​​are grouped, and the columns of the second weight matrix are dynamically grouped based on the dynamic parameter values ​​to obtain multiple groups of columns. The dynamic parameter value refers to the parameter value that is dynamically adjusted according to the data distribution and model status at different stages during the pruning process, which is used to guide the pruning operation. Pruning evaluation is implemented, and the evaluation matrix is ​​used to perform pruning evaluation on multiple groups of columns. This evaluation process aims to identify the columns that have the least impact on the model performance, that is, the target columns. After obtaining the evaluation results, a target column in a group of columns (target group) is selected as the object of the next pruning operation. The selection of the target column is based on the result of the pruning evaluation, that is, the column with the smallest error.

[0093] It should be noted that the second weight matrix can be grouped in column format based on the dynamic parameter value. After obtaining multiple groups of columns, the multiple groups of columns can be pruned and evaluated through the evaluation matrix to obtain the target columns corresponding to the target group that needs to be pruned.

[0094] The above-mentioned multiple network layers are important components of the target model, responsible for the dimension increase and dimension reduction operations of information. Since multiple network layers are usually very large, direct pruning may be very time-consuming and inefficient. Therefore, this application proposes a column pruning strategy based on dynamic parameter values ​​to improve the structure and performance of multiple network layers.

[0095] The above dynamic parameter values ​​(such as top-N) are used to guide the column grouping and pruning strategy during the pruning process. At each iteration, the system will mark the top-N columns with the smallest errors and use these columns as candidates for evaluation and pruning. This dynamic adjustment method can gradually adjust the size of N from an initial large value (such as 1024) to a very small value (such as 8) as the pruning process progresses, to ensure the efficiency and accuracy of the pruning operation. The above numerical values ​​are for example only and are not specifically limited.

[0096] First, the second weight matrices of multiple network layers are obtained. Based on the information transmission strength of the network layers described by the weight values, the columns of the matrices are grouped using dynamic parameter values ​​to obtain multiple groups of candidate columns. Next, by evaluating the impact of matrix quantization pruning on model performance, multiple groups of columns are evaluated for pruning, the columns with the least impact on performance (target columns) are identified, and the target columns in a group are selected for pruning. This process is iterative until the desired degree of pruning is achieved.

[0097] Through the pruning strategy of this embodiment, the computational complexity of multiple network layers can be effectively reduced, the latency of online reasoning can be reduced, and the pruning process can be ensured to have the least impact on model performance through the guidance of evaluation matrix and dynamic parameter values. This not only improves the running efficiency of the model on resource-constrained devices, but also maintains the accuracy and stability of the model.

[0098] The system prunes and evaluates these grouped columns through the evaluation matrix, identifies the columns (target columns) that have the least impact on model performance, and selects a target column from multiple groups of columns for pruning. This process is iterative until the predetermined number of pruning is reached. Ultimately, the pruned multiple network layers can significantly reduce the latency of online reasoning and improve the response speed of the search system while maintaining accurate search results and rewrite suggestions. On resource-constrained mobile devices, this improvement strategy greatly improves the model's operating efficiency and user experience.

[0099] By dynamically adjusting pruning parameters and guiding the evaluation matrix, efficient pruning of multiple network layers is achieved, which reduces the consumption of computing resources and speeds up processing while ensuring the accuracy and quality of processing results. This is an important means to improve the online reasoning of the target model.

[0100] In the above embodiment of the present application, a pruning operation is performed on the initial weight matrix based on the target column to obtain an intermediate weight matrix, including: a pruning operation is performed on the initial weight matrix based on the target column to obtain a pruned weight matrix, wherein the pruned weight matrix includes pruned columns and unpruned columns; and parameter compensation is performed on the unpruned columns in the pruned weight matrix based on the evaluation matrix to obtain an intermediate weight matrix.

[0101] The target columns mentioned above are analyzed based on the evaluation matrix to determine the columns that have the least impact on the model performance after pruning, namely the target columns.

[0102] The above pruning operation is to perform a pruning operation on the initial weight matrix based on the target column, that is, to reset or remove the weights of these columns to form a pruned weight matrix. The pruned weight matrix contains pruned columns and unpruned columns, where the weight values ​​of the pruned columns are modified or deleted, and the weight values ​​of the unpruned columns remain unchanged.

[0103] The above parameter compensation is to use the evaluation matrix to perform parameter compensation on the unpruned columns in the pruned weight matrix after the pruning operation. This is done by calculating the new weight value of each unpruned column in order to compensate for the performance loss that may be caused by the pruning operation and ensure that the model can maintain its stability and performance after pruning.

[0104] Exemplarily, for the pruning of the attention module, assuming that the goal is to prune the third attention head, first, it is necessary to determine the columns related to the attention head as the target columns. Then, these columns in the initial weight matrix are pruned, and they are set to zero or removed to form a pruned weight matrix. At this time, the columns of the third attention head in the pruned weight matrix are marked as pruned columns, and the columns corresponding to other heads are retained as unpruned columns. The above values ​​are for illustrative purposes only and are not specifically limited.

[0105] To ensure the performance of the pruned model, the next step is to perform parameter compensation on the unpruned columns in the pruned weight matrix. This step uses the evaluation matrix to estimate and adjust the new weight values ​​of the unpruned columns to compensate for the performance loss caused by pruning. This pruning strategy that considers both the weight matrix and the evaluation matrix information can more effectively retain the key performance of the model while reducing the complexity and computational requirements of the model.

[0106] After the pruning operation, the system uses the evaluation matrix to compensate the parameters of the unpruned columns, and compensates for the performance loss that may be caused by pruning by calculating and adjusting new weight values. For example, if the performance of the columns of the first attention head decreases after pruning, the evaluation matrix can provide information to compensate for the unpruned parameters, thereby maintaining the model performance to restore its performance. This compensation process is refined, ensuring that the pruned model can not only respond quickly to online queries, but also maintain high query rewrite performance, minimizing the impact on user experience.

[0107] The stability and performance of the model are maintained through parameter compensation, avoiding a significant drop in performance. This pruning strategy improves the model's computational cost and response speed while ensuring the model's processing capabilities, and is of great value in improving user experience in scenarios such as search systems and intelligent customer service. In the search scenario, this strategy enables the system to quickly respond to user queries and provide accurate search results while maintaining high query rewriting performance, improving the overall user experience and search efficiency.

[0108] In the above embodiment of the present application, a pruning operation is performed on the initial weight matrix based on the target column to obtain a pruned weight matrix, including: rearranging the initial weight matrix based on the target column to obtain a rearranged weight matrix; and pruning the rearranged weight matrix to obtain a pruned weight matrix.

[0109] The above-mentioned column rearrangement is to reorganize the target column and the columns related to it in the weight matrix to the front position of the matrix to form a rearranged weight matrix. This step ensures that the target column and its related columns are easy to identify and operate.

[0110] The above pruning operation is to perform a pruning operation on the target columns in the rearranged weight matrix, that is, to set the weight values ​​of these columns to zero or remove them to obtain a pruned weight matrix.

[0111] The pruned weight matrix above contains the weights of the model after pruning, where the weights of the target columns are modified or removed, while the unpruned columns remain unchanged.

[0112] Exemplarily, the attention module of the large language model has 8 attention heads, each with 128 columns. After evaluation, the 4th attention head is determined to be the pruning target, and the system needs to prune the columns related to the 4th head. First, the 128 columns related to the 4th attention head are rearranged from the original weight matrix to the front of the matrix to form a rearranged weight matrix. Subsequently, the 128 columns in the rearranged weight matrix are pruned, that is, they are set to zero or removed to obtain a pruned weight matrix. The pruned attention module reduces the consumption of computing resources while maintaining the model processing capability, thereby reducing the delay of online reasoning and improving the response speed of the search system and the user experience. The above values ​​are for example only and are not specifically limited.

[0113] The pruning operation of the present application starts with determining the target column, and the target column is placed at the front of the matrix by rearranging the columns to facilitate pruning. Then, the pruning operation is performed on the rearranged weight matrix to obtain the pruned weight matrix. The implementation of this process ensures the efficiency of the pruning operation and the minimum impact on the model performance.

[0114] In the above embodiment of the present application, parameter compensation is performed on the unpruned columns in the pruned weight matrix based on the evaluation matrix to obtain a target weight matrix, including: determining a target row corresponding to the target column in the evaluation matrix; rearranging the evaluation matrix based on the target row and the target column to obtain a rearranged evaluation matrix; determining parameter compensation values ​​of the unpruned columns in the pruned weight matrix based on the rearranged evaluation matrix; and performing parameter compensation on the unpruned columns in the pruned weight matrix based on the parameter compensation values ​​to obtain a target weight matrix.

[0115] The above-mentioned determination target behavior determines the rows associated with the target columns from the evaluation matrix, and these rows are used for subsequent matrix rearrangement and parameter compensation calculation.

[0116] The above rearranged evaluation matrix is ​​formed by rearranging the evaluation matrix in the same column rearrangement as the target weight matrix. This process ensures the corresponding relationship between the evaluation matrix and the weight matrix, which facilitates the calculation of the parameter compensation value.

[0117] The above-mentioned determination of parameter compensation values ​​is to use the rearranged evaluation matrix to determine the parameter compensation values ​​of the unpruned columns in the pruning weight matrix according to the model compression framework. These compensation values ​​are used to adjust the weights of the unpruned columns to compensate for the performance degradation that may be caused by the pruning operation.

[0118] The above-mentioned parameter compensation is to perform parameter compensation on the unpruned columns in the pruning weight matrix based on the calculated parameter compensation values, so as to obtain the target weight matrix, that is, the model weights after pruning and boosting.

[0119] In the process of rearranging the evaluation matrix (H^-1), the row rearrangement is performed synchronously with the column rearrangement of the weight matrix to maintain the correspondence between them. This step makes the structure of the matrix simpler, which is convenient for the subsequent calculation of the parameter compensation value, because the evaluation matrix after matrix decomposition has a simpler triangular structure, which helps to more accurately calculate the compensation value of a single weight.

[0120] Exemplarily, assume that in a certain attention module, the columns related to the 5th attention head have been identified as pruning targets. First, determine the rows related to these columns from the evaluation matrix as target rows. Then, rearrange the rows of the evaluation matrix to ensure that these target rows correspond to the positions of the pruned columns in the weight matrix. Calculate the parameter compensation values ​​of the unpruned columns based on the rearranged evaluation matrix. Finally, perform parameter compensation on the unpruned columns based on these compensation values ​​to obtain the target weight matrix, that is, the weight matrix of the model after pruning and parameter compensation. The above values ​​are for illustrative purposes only and are not specifically limited.

[0121] The parameter compensation process of the present application starts with determining the rows of the evaluation matrix related to the target column, and then rearranges the rows of the evaluation matrix to be consistent with the column rearrangement of the weight matrix. Then, the compensation value of the unpruned column is calculated according to the rearranged evaluation matrix, and parameter compensation is performed to finally obtain the target weight matrix, that is, the improved model weight matrix. Through the parameter compensation method of the present application, the performance degradation that may be caused by the pruning operation can be effectively compensated, ensuring that the pruned model can continue to provide high-precision processing results while maintaining a low computational complexity. This strategy not only improves the operating efficiency of the model, reduces the online reasoning delay, but also maintains the model performance, which is particularly important for resource-constrained deployment environments.

[0122] In the above embodiment of the present application, determining the target row corresponding to the target column from the evaluation matrix includes: constructing a correspondence between the rows of the initial weight matrix and the columns of the evaluation matrix; and determining the target row corresponding to the target column in the evaluation matrix based on the correspondence.

[0123] The above correspondence is to establish a correspondence between the rows of the initial weight matrix and the columns of the evaluation matrix before the pruning operation begins. This correspondence ensures that the modification of the weights in the pruning operation can match the adjustment of the relevant parameters in the evaluation matrix, maintaining the stability and performance of the model after pruning.

[0124] The above target determination behavior is based on the established correspondence, and the rows related to the target columns are determined from the evaluation matrix. The target columns are those that are about to be pruned. The purpose of determining the target rows is to calculate the subsequent parameter compensation to ensure that the pruning operation does not significantly affect the performance of the model.

[0125] During the pruning process, since the pruning operation needs to take into account the information of both the weight matrix and the evaluation matrix, pre-constructing the correspondence between the rows of the initial weight matrix and the columns of the evaluation matrix can ensure that the pruned model can respond quickly while maintaining high rewrite performance.

[0126] For example, if an attention module containing 16 attention heads is being upgraded. If the column of the 8th attention head (target column) is determined to be the pruning target, first, it is necessary to find the column related to the 8th attention head in the initial weight matrix, and then find the corresponding row in the evaluation matrix to build a correspondence between rows and columns. This correspondence ensures the accuracy of the pruning operation. For example, if a column of the 8th head is pruned, the row related to the column in the evaluation matrix will also be used for subsequent parameter compensation calculations to compensate for the performance loss caused by pruning. The above values ​​are for example only and are not specifically limited.

[0127] The process of determining the target row from the evaluation matrix is ​​part of the pruning operation. First, by constructing the correspondence between the rows of the initial weight matrix and the columns of the evaluation matrix, the necessary information mapping is provided for the pruning operation. Next, based on the determined target columns, the corresponding rows are determined from the evaluation matrix, which will be used to calculate the parameter compensation values ​​of the unpruned columns after pruning, ensuring that the pruning operation has the least impact on the model performance. This step of the present application effectively ensures the accuracy and efficiency of the pruning process. By determining the target rows, the parameter compensation values ​​of the unpruned columns can be accurately calculated, thereby reducing the computational complexity and online inference delay without significantly affecting the model performance. This strategy improves the computational cost and response speed of the model while maintaining the model processing capability, which is of great value for improving the user experience in scenarios such as e-commerce search and intelligent customer service.

[0128] In this application, the large model compression framework can be applied to the hierarchical compression process of large model structured pruning. For the large model compression framework, it mainly includes two steps, error estimation and parameter compensation. For error estimation, the goal is to minimize the L2 error output by the first weight matrix. The Hessian matrix can be calculated by inputting the parameter matrix, and then the error caused by the removal of each weight can be calculated by the Hessian matrix, and the weight with the smallest error can be reset to zero. For parameter compensation, after the weight is removed, the compensation value of the remaining unpruned weight can be calculated by the weight pruning formula, and parameter compensation is performed for these weight updates, that is, the pruning of the weight is completed. It should be noted that the weight pruning formula is used to determine the adjustment path of weight pruning in the neural network, which includes the key step of calculating how to compensate the remaining weights after weight pruning to minimize performance loss. The core idea of ​​the formula is to evaluate the impact of weight pruning on network output based on the Hessian matrix, and adjust the network to adapt to the pruned state through parameter compensation, thereby minimizing the performance degradation caused by pruning.

[0129] Since each row of weights is calculated independently, this application first performs pruning that expands the large model compression framework from weight granularity (weight granularity) to column granularity (column granularity), that is, treating a column of weights as an overall evaluation error, and then makes some improvements on this basis to make it suitable for large model structured pruning.

[0130] Figure 3 is a structural diagram of a model processing according to an embodiment of the present application, such as Figure 3 As shown, it contains the original first weight matrix W and the inverse of the Hessian matrix H -1 , where W represents the original first weight matrix of the attention module or network module in the large model, which may contain multiple head units (head0, head1, head2), and each column corresponds to a set of weights in the model. -1 Represents the inverse of the Hessian matrix, which is used to calculate the possible error after pruning. The matrix W can be reordered to rearrange the attention heads for easy pruning. -1 Perform group decomposition to maintain consistency with the W matrix.

[0131] After determining the part of the matrix W to be pruned, the matrix W is reorganized to rearrange the attention heads for pruning. -1 The corresponding rows and columns of the matrix are also rearranged to maintain consistency with the W matrix. The pruned part can be pruned and the first weight matrix after the pruning operation is displayed, where some columns (corresponding to specific attention head units) are marked as 0, indicating that the weights of these columns will be pruned, that is, removed.

[0132] After pruning the matrix W, the columns of weights and H -1 The rows and columns of H are rearranged to the front, and then when pruning columns one by one, all unpruned columns on the right are compensated, and finally the columns of the weight matrix are restored to the original order. -1 Perform matrix decomposition, and the non-zero elements of different matrices correspond to the decomposition results. The last step is compensation, which can update the remaining unpruned weights to minimize the performance loss caused by pruning. Using matrix decomposition, the diagonal elements of the inverse of the Hessian matrix can be calculated in advance to evaluate the pruning error of each attention head unit. The columns of the attention head unit with the smallest error can be rearranged to the front of the matrix W, and H can be rearranged at the same time. -1 The corresponding rows and columns of the matrix. The columns of the rearranged attention head unit are pruned, and the remaining unpruned weights are compensated until all columns of the attention head unit are completely pruned.

[0133] Through the above pruning method, large models can be effectively compressed, while reducing the latency of online inference, and maintaining high model performance through parameter compensation during the pruning process. The entire process achieves the dual goals of model compression and performance retention without relying on large-scale post-training.

[0134] Step 1: For the attention module in the large model, its smallest pruning unit is the attention head unit pruning. Therefore, the head unit error can be calculated using the column error evaluation in two steps, including head unit error calculation and column-by-column head unit pruning.

[0135] Step 1.1, head unit error calculation. The model compression framework is an iterative process. The calculation of the inverse of the Hessian matrix in the iteration is time-consuming and unstable. Generally, the diagonal elements of the inverse of the Hessian matrix are obtained in advance through matrix decomposition. However, matrix decomposition destroys the comparability between columns, making the calculation of the head unit error by column error very unreliable. This application proposes to perform grouped matrix decomposition along the diagonal with the head unit size as the unit, so that the error comparison of different head units becomes more accurate.

[0136] Step 1.2, prune the head unit column by column. After finding the head unit with the smallest error, rearrange the columns corresponding to the first weight matrix to the front through column reorganization, and rearrange the rows and columns corresponding to the inverse of the Hessian matrix accordingly, calculate the decomposition matrix of the inverse of the rearranged Hessian matrix, start from the first column, prune the columns one by one, and perform parameter compensation on the remaining unpruned columns until all columns of the first head unit are pruned. Finally, restore the order of the pruned first weight matrix, thus completing the pruning of a head unit.

[0137] Repeat steps 1.1 and 1.2 to prune multiple head units until the specified number of pruning is completed.

[0138] Step 2, for the Feed-Forward Network (FFN) module in the large model, since the size of the intermediate layer is very large, column-by-column pruning is time-consuming. This application proposes a dynamic unit size (group size) method for column pruning. In each iteration, the top-N columns with the smallest error are marked, rearranged to the front, and then the method in step 1.2 is used to prune the first N columns, and the remaining unpruned weights are compensated. Due to the dynamic unit size, this application uses a larger N (such as 1024) in the initial stage of pruning. Considering that there may be local rearrangement of columns during the iteration process, this application gradually reduces N until it reaches a relatively small value (such as 8), and iterates this process to complete the pruning of the network module.

[0139] Step 3: After all network modules in a model are pruned layer by layer, the pruning of the model is completed.

[0140] Step 4: After pruning, you can selectively fine-tune the model to restore performance.

[0141] The model processing method in this application can directly operate on the weights without intrusive modification of the model structure to insert masks, and the two stages of pruning and training are decoupled, so the actual pruning process is simpler and more universal, and can be quickly migrated in most scenarios. The pruning method of this application has no upper and lower node dependencies, does not need to rely on code library updates, and can directly adapt the new model quickly. In addition, the large model compression framework performs parameter compensation for the remaining unpruned weights during the pruning process, so it is less dependent on post-training and has a stronger ability to retain the performance of large models.

[0142] Compared with other structured pruning methods, the structured pruning scheme proposed in this application compresses large models and retains more performance while maintaining low cost, low resources, low time consumption, and no task external code dependency. This application is the first to apply a large model compression framework to large model structured pruning, which improves accuracy by reducing the error estimation of the attention head by the large model compression framework, and performs parameter compensation for unpruned weights, greatly reducing the performance loss caused by pruning.

[0143] Large language models have powerful semantic understanding capabilities. In e-commerce query rewriting systems, generative rewriting based on large language models shows great performance improvement in the rewriting task of long-tail queries compared to traditional discriminative rewriting. For new queries that appear for the first time, it is necessary to deploy a large model online for real-time inference for rewriting, and online inference will cause a large delay. Based on this scenario, in order to reduce the inference delay of the online model, this application proposes a low-cost structured pruning method, which is the first time that the weight pruning method of neural networks is applied to structured pruning of large models. The performance loss caused by pruning is reduced by parameter compensation, and the model can be compressed without relying on large-scale post-training, thereby improving the inference speed and retaining performance to the greatest extent.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0145] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0147] According to an embodiment of the present application, a model processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here. Figure 4 is a flow chart of a model processing method according to an embodiment of the present application, such as Figure 4 As shown, the method includes:

[0148] Step S402, obtaining an initial weight matrix of the target model and an evaluation matrix corresponding to the initial weight matrix;

[0149] Among them, the initial weight matrix is ​​used to represent the initial weight matrix to be pruned in the target model, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model.

[0150] Step S404, performing pruning evaluation on the initial weight matrix based on the evaluation matrix to obtain a target column;

[0151] Step S406, performing a pruning operation on the initial weight matrix based on the target column to obtain a target weight matrix.

[0152] The above process can reduce the computational cost and inference latency of the model through pruning without significantly affecting the model performance. First, the impact of changes in matrix quantization weights on model performance is evaluated; then, the target columns for pruning are determined based on the evaluation results; finally, pruning is performed on the target columns to generate the target weight matrix. The entire process combines the structural characteristics and performance requirements of the model, and effectively balances model performance and computational efficiency through a refined pruning strategy.

[0153] By implementing the above pruning steps, the model structure is effectively improved, the latency of online reasoning is reduced, and the evaluation matrix is ​​used to guide the pruning operation, which ensures that the pruning operation does not have a significant negative impact on the model performance. This strategy is particularly suitable for scenarios such as e-commerce query rewriting systems, where large models need to process queries in real time and provide rewriting suggestions. Through pruning, the model can reason faster while maintaining high rewriting performance, improving the user experience.

[0154] Through the above steps, the initial weight matrix of the target model and the evaluation matrix corresponding to the initial weight matrix are obtained, wherein the initial weight matrix is ​​used to represent the initial weight matrix to be pruned in the target model, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model; the initial weight matrix is ​​pruned and evaluated based on the evaluation matrix to obtain the target column; the initial weight matrix is ​​pruned based on the target column to obtain the target weight matrix, thereby achieving the purpose of improving the model processing effect; it is easy to notice that the impact of weight changes on model performance can be quantified based on the evaluation matrix, so as to accurately identify the target column where the weights with less impact on model performance are located, and the initial weight matrix of the target model can be pruned according to the target column, thereby reducing the computational complexity and memory usage of the model while reducing the impact of the pruning operation on the model performance, thereby improving the model processing effect on the text to be processed, thereby solving the technical problem of poor model processing effect in related technologies.

[0155] According to an embodiment of the present application, a text processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here. Figure 5 is a flowchart of a text processing method according to an embodiment of the present application. Figure 5 As shown, the method includes:

[0156] Step S502, in response to an input instruction acting on the operation interface, displaying the text to be processed on the operation interface;

[0157] The above steps describe the beginning of the user-system interaction. When the user enters a query text on the operation interface, the system responds to this input instruction and displays the text entered by the user on the interface as the text to be processed. This step is the bridge between user query and system processing. The user interacts directly with the target model through this interface.

[0158] Step S504, responding to the text processing instruction on the operation interface, and displaying the text processing result on the operation interface.

[0159] Among them, the text processing result is obtained by inferring the text to be processed using the target model, the target weight matrix of the target model is obtained by pruning the initial weight matrix of the target model according to the target column, and the target column is obtained by pruning and evaluating the initial weight matrix based on the evaluation matrix. The initial weight matrix is ​​used to represent the initial weight matrix to be pruned in the target model, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model.

[0160] A text processing instruction is usually an operation that the user asks the system to perform, such as requesting to search and rewrite the input text. In response to this instruction, the system uses the target model to reason about the text to be processed, thereby generating a text processing result.

[0161] Through the above steps, in response to the input instruction on the operation interface, the text to be processed is displayed on the operation interface; in response to the text processing instruction on the operation interface, the text processing result is displayed on the operation interface, wherein the text processing result is obtained by reasoning the text to be processed using the target model, the target weight matrix of the target model is obtained by pruning the initial weight matrix of the target model according to the target column, the target column is obtained by pruning and evaluating the initial weight matrix based on the evaluation matrix, the initial weight matrix is ​​used to represent the initial weight matrix to be pruned in the target model, and the evaluation matrix is ​​used to quantify the impact of the weight change in the initial weight matrix on the performance of the target model, thereby achieving the purpose of improving the model processing effect; it is easy to notice that the impact of the weight change on the model performance can be quantified based on the evaluation matrix, so as to accurately identify the target column where the weight has a small impact on the model performance, and the initial weight matrix of the target model can be pruned according to the target column, while reducing the computational complexity and memory usage of the model, reducing the impact of the pruning operation on the model performance, thereby improving the model processing effect on the text to be processed, thereby solving the technical problem of poor model processing effect in related technologies.

[0162] According to an embodiment of the present application, a text processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here. Figure 6 is a flowchart of a text processing method according to an embodiment of the present application. Figure 6 As shown, the method includes:

[0163] Step S602, obtaining the text to be processed by calling the first interface;

[0164] The first interface includes a first parameter, and a parameter value of the first parameter includes text to be processed.

[0165] The above-mentioned first interface can be an interface for data interaction between the cloud server and the client, and the text to be processed can be passed into the interface function as the first parameter of the interface function to achieve the purpose of uploading the text to be processed to the cloud server.

[0166] Step S604, using the target model to infer the text to be processed to obtain a text processing result;

[0167] The target weight matrix of the target model is obtained by pruning the initial weight matrix of the target model according to the target column, and the target column is obtained by pruning and evaluating the initial weight matrix based on the evaluation matrix. The evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model.

[0168] Step S606: output the text processing result by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the text processing result.

[0169] The above-mentioned second interface can be an interface for data interaction between the cloud server and the client. The cloud server can pass the text processing result to the interface function as the second parameter of the interface function to achieve the purpose of sending the text processing result to the client.

[0170] Through the above steps, the text to be processed is obtained by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the text to be processed; the target model is used to infer the text to be processed to obtain a text processing result, wherein the target weight matrix of the target model is obtained by pruning the initial weight matrix of the target model according to the target column, and the target column is obtained by pruning and evaluating the initial weight matrix based on the evaluation matrix, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model; the text processing result is output by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the text processing result, thereby achieving the purpose of improving the model processing effect; it is easy to notice that the impact of weight changes on model performance can be quantified based on the evaluation matrix, so as to accurately identify the target column where the weight has a small impact on the model performance, and the initial weight matrix of the target model can be pruned according to the target column, while reducing the computational complexity and memory usage of the model, reducing the impact of the pruning operation on the model performance, thereby improving the model processing effect on the text to be processed, thereby solving the technical problem of poor model processing effect in related technologies.

[0171] According to an embodiment of the present application, a text processing device for implementing the above text processing method is also provided. Figure 7 is a schematic diagram of a text processing device according to an embodiment of the present application, such as Figure 7 As shown, the device 700 includes: an acquisition module 702 and a reasoning module 704.

[0172] Among them, the acquisition module is used to obtain the text to be processed; the inference module is used to use the target model to infer the text to be processed to obtain the text processing result; among them, the target weight matrix of the target model is obtained by pruning the initial weight matrix of the target model according to the target column, and the target column is obtained by pruning and evaluating the initial weight matrix based on the evaluation matrix, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model.

[0173] It should be noted that the acquisition module 702 and the reasoning module 704 correspond to steps S202 to S204 in the above embodiment, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run as part of the device in the server 10 provided in the above embodiment.

[0174] In the above embodiments of the present application, the device further includes: a pruning module, an execution module, and a determination module.

[0175] Among them, the pruning module is used to perform pruning operations on the initial weight matrix based on the target column to obtain an intermediate weight matrix; the execution module is used to use the intermediate weight matrix as the initial weight matrix, perform pruning evaluation on the initial weight matrix based on the evaluation matrix, obtain the target column, and perform pruning operations on the initial weight matrix based on the target column to obtain the intermediate weight matrix, until the number of iterations reaches the pruning number; the determination module is used to use the intermediate weight matrix as the target weight matrix.

[0176] In the above embodiments of the present application, the device further includes: an evaluation module and a determination module.

[0177] Among them, the determination module is also used to determine multiple attention units corresponding to the first weight matrix in response to the initial weight matrix being the first weight matrix, wherein the weight values ​​of the first weight matrix are used to describe the attention allocation intensity of the multiple attention units; the evaluation module is used to perform pruning evaluation on the multiple attention units based on the evaluation matrix to obtain the target attention unit, wherein the target attention unit is the attention unit that has the least impact on the target model when the pruning operation is performed on the multiple attention units; the determination module is used to determine the target column corresponding to the target attention unit from the first weight matrix.

[0178] In the above embodiment of the present application, the evaluation module is used to decompose the evaluation matrix along the diagonal direction of the evaluation matrix based on the attention head sizes of multiple attention units to obtain multiple sub-matrices, wherein the multiple sub-matrices correspond one-to-one to the multiple attention units; and perform pruning evaluation on the multiple attention units based on the multiple sub-matrices to obtain the target attention unit.

[0179] In the above embodiments of the present application, the device further includes: a grouping module.

[0180] Among them, the grouping module is used to respond to the initial weight matrix being the second weight matrix, and group the second weight matrix by column based on the dynamic parameter value to obtain multiple groups of columns, wherein the weight values ​​in the second weight matrix are used to describe the information transmission intensity of multiple network layers corresponding to the target model, and the dynamic parameter values ​​are used to represent the parameter values ​​that dynamically change during the pruning of the target model; the evaluation module is used to perform pruning evaluation on the multiple groups of columns based on the evaluation matrix to obtain the target column corresponding to the target group.

[0181] In the above embodiments of the present application, the pruning module is used to perform a pruning operation on the initial weight matrix based on the target column to obtain a pruned weight matrix, wherein the pruned weight matrix includes pruned columns and unpruned columns; and perform parameter compensation on the unpruned columns in the pruned weight matrix based on the evaluation matrix to obtain an intermediate weight matrix.

[0182] In the above embodiments of the present application, the pruning module is used to rearrange the initial weight matrix based on the target column to obtain a rearranged weight matrix; and perform a pruning operation on the rearranged weight matrix to obtain a pruned weight matrix.

[0183] In the above embodiments of the present application, the pruning module is used to determine the target row corresponding to the target column in the evaluation matrix from the evaluation matrix; rearrange the evaluation matrix based on the target row and the target column to obtain a rearranged evaluation matrix; determine the parameter compensation value of the unpruned column in the pruning weight matrix based on the rearranged evaluation matrix; perform parameter compensation on the unpruned column in the pruning weight matrix based on the parameter compensation value to obtain a target weight matrix.

[0184] In the above embodiments of the present application, the pruning module is used to construct a correspondence between the rows of the initial weight matrix and the columns of the evaluation matrix; and determine the target row corresponding to the target column in the evaluation matrix based on the correspondence.

[0185] According to an embodiment of the present application, a model processing device for implementing the above-mentioned model processing method is also provided. Figure 8 is a schematic diagram of a model processing device according to an embodiment of the present application, such as Figure 8 As shown, the device 800 includes: an acquisition module 802 , an evaluation module 804 , and a pruning module 806 .

[0186] Among them, the acquisition module is used to obtain the initial weight matrix of the target model and the evaluation matrix corresponding to the initial weight matrix, wherein the initial weight matrix is ​​used to represent the initial weight matrix to be pruned in the target model, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model; the evaluation module is used to prune and evaluate the initial weight matrix based on the evaluation matrix to obtain the target column; the pruning module is used to prune the initial weight matrix based on the target column to obtain the target weight matrix.

[0187] It should be noted that the acquisition module 802, the evaluation module 804, and the pruning module 806 correspond to steps S402 to S406 in the above embodiment, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run as part of the device in the server 10 provided in the above embodiment.

[0188] According to an embodiment of the present application, a text processing device for implementing the above text processing method is also provided. Fig. 9 is a schematic diagram of a text processing device according to an embodiment of the present application, such as Fig. 9 As shown, the device 900 includes: a first display module 902 and a second display module 904 .

[0189] Among them, the first display module is used to respond to the input instructions acting on the operation interface, and display the text to be processed on the operation interface; the second display module is used to respond to the text processing instructions acting on the operation interface, and display the text processing results on the operation interface, wherein the text processing results are obtained by reasoning about the text to be processed using the target model, and the target weight matrix of the target model is obtained by pruning the initial weight matrix of the target model according to the target column, and the target column is obtained by pruning and evaluating the initial weight matrix based on the evaluation matrix. The initial weight matrix is ​​used to represent the initial weight matrix to be pruned in the target model, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model.

[0190] It should be noted that the first display module 902 and the second display module 904 correspond to steps S502 to S504 in the above embodiment, and the two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run as part of the device in the server 10 provided in the above embodiment.

[0191] According to an embodiment of the present application, a text processing device for implementing the above text processing method is also provided. Fig.10 is a schematic diagram of a text processing device according to an embodiment of the present application, such as Fig.10 As shown, the device 1000 includes: an acquisition module 1002 , a reasoning module 1004 , and an output module 1006 .

[0192] Among them, an acquisition module is used to obtain the text to be processed by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the text to be processed; an inference module is used to use a target model to infer the text to be processed to obtain a text processing result, wherein the target weight matrix of the target model is obtained by pruning the initial weight matrix of the target model according to the target column, and the target column is obtained by pruning and evaluating the initial weight matrix based on the evaluation matrix, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model; an output module is used to output the text processing result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the text processing result.

[0193] It should be noted that the acquisition module 1002, the reasoning module 1004, and the output module 1006 correspond to steps S602 to S606 in the above embodiment, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run as part of the device in the server 10 provided in the above embodiment.

[0194] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the above embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the above embodiments.

[0195] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0196] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0197] In this embodiment, the computer terminal can execute the program code in the method.

[0198] Optionally, Fig.11 is a structural block diagram of an electronic device according to an embodiment of the present application. Fig.11 As shown, the electronic device A may include: one or more ( Fig.11 (only one is shown) processor 102, memory 104, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0199] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and text processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0200] The processor can call the information and application programs stored in the memory through the transmission device to execute the following steps: obtain the text to be processed; use the target model to infer the text to be processed to obtain the text processing result; wherein, the target weight matrix of the target model is obtained by pruning the initial weight matrix of the target model according to the target column, and the target column is obtained by pruning and evaluating the initial weight matrix based on the evaluation matrix, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model.

[0201] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain the initial weight matrix of the target model and the evaluation matrix corresponding to the initial weight matrix, wherein the initial weight matrix is ​​used to represent the initial weight matrix to be pruned in the target model, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model; prune and evaluate the initial weight matrix based on the evaluation matrix to obtain a target column; prune the initial weight matrix based on the target column to obtain a target weight matrix.

[0202] Those skilled in the art will understand that Fig.11 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Fig.11 It does not limit the structure of the above electronic device. For example, the electronic device A may also include Fig.11 More or fewer components (such as network interfaces, display devices, etc.) shown in the figure, or having Fig.11 Different configurations are shown.

[0203] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0204] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method provided in the above embodiment.

[0205] Optionally, in this embodiment, the above storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.

[0206] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining a text to be processed; using a target model to infer the text to be processed to obtain a text processing result; wherein a target weight matrix of the target model is obtained by pruning an initial weight matrix of the target model according to a target column, and the target column is obtained by pruning and evaluating the initial weight matrix based on an evaluation matrix, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model.

[0207] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining an initial weight matrix of the target model and an evaluation matrix corresponding to the initial weight matrix, wherein the initial weight matrix is ​​used to represent the initial weight matrix to be pruned in the target model, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model; performing pruning and evaluation on the initial weight matrix based on the evaluation matrix to obtain a target column; performing pruning operations on the initial weight matrix based on the target column to obtain a target weight matrix.

[0208] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and the computer program implements the method provided in the embodiment when executed by a processor.

[0209] The embodiments of the present application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which may be used to store a computer program, and when the computer program is executed by a processor, the method provided in the embodiments is implemented.

[0210] The embodiment of the present application further provides a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the method provided in the above embodiment is implemented.

[0211] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0212] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0213] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0214] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0215] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0216] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A text processing method, characterized in that: include: Get the text to be processed; Using the target model to infer the text to be processed to obtain a text processing result; Among them, the target weight matrix of the target model is obtained by pruning the initial weight matrix of the target model according to the target column, and the target column is obtained by pruning and evaluating the initial weight matrix based on the evaluation matrix, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model.

2. The text processing method according to claim 1, characterized in that: The method further comprises: Performing a pruning operation on the initial weight matrix based on the target column to obtain an intermediate weight matrix; Using the intermediate weight matrix as the initial weight matrix, executing the steps of pruning and evaluating the initial weight matrix based on the evaluation matrix to obtain a target column, and pruning the initial weight matrix based on the target column to obtain an intermediate weight matrix, until the number of iterations reaches the pruning number; The intermediate weight matrix is ​​used as the target weight matrix.

3. The text processing method according to claim 1 or 2, characterized in that: The method further comprises: In response to the initial weight matrix being a first weight matrix, determining a plurality of attention units corresponding to the first weight matrix, wherein the weight values ​​of the first weight matrix are used to describe the attention allocation strengths of the plurality of attention units; Performing pruning evaluation on the multiple attention units based on the evaluation matrix to obtain a target attention unit, wherein the target attention unit is an attention unit that has the least impact on the target model when the pruning operation is performed on the multiple attention units; Determine the target column corresponding to the target attention unit from the first weight matrix.

4. The text processing method according to claim 3, characterized in that: Performing pruning evaluation on the multiple attention units based on the evaluation matrix to obtain a target attention unit includes: Based on the attention head sizes of the multiple attention units, decomposing the evaluation matrix along the diagonal direction of the evaluation matrix to obtain multiple sub-matrices, wherein the multiple sub-matrices correspond to the multiple attention units one by one; The multiple attention units are pruned and evaluated based on the multiple sub-matrices to obtain the target attention unit.

5. The text processing method according to claim 1 or 2, characterized in that: The method further comprises: In response to the initial weight matrix being a second weight matrix, the second weight matrix is ​​grouped by column based on dynamic parameter values ​​to obtain multiple groups of columns, wherein the weight values ​​in the second weight matrix are used to describe the information transmission strength of the target model corresponding to multiple network layers, and the dynamic parameter values ​​are used to represent the parameter values ​​that dynamically change during the pruning of the target model; Pruning evaluation is performed on the multiple groups of columns based on the evaluation matrix to obtain the target columns corresponding to the target groups.

6. The text processing method according to claim 2, characterized in that: The initial weight matrix is ​​pruned based on the target column to obtain an intermediate weight matrix, including: Performing a pruning operation on the initial weight matrix based on the target column to obtain a pruned weight matrix, wherein the pruned weight matrix includes pruned columns and unpruned columns; Parameter compensation is performed on the unpruned columns in the pruning weight matrix based on the evaluation matrix to obtain the intermediate weight matrix.

7. The text processing method according to claim 6, characterized in that: Performing a pruning operation on the initial weight matrix based on the target column to obtain a pruned weight matrix includes: Rearranging the initial weight matrix based on the target column to obtain a rearranged weight matrix; A pruning operation is performed on the rearranged weight matrix to obtain the pruned weight matrix.

8. The text processing method according to claim 6, characterized in that: Performing parameter compensation on the unpruned columns in the pruning weight matrix based on the evaluation matrix to obtain the target weight matrix includes: Determining a target row corresponding to the target column in the evaluation matrix; Rearranging the evaluation matrix based on the target row and the target column to obtain a rearranged evaluation matrix; Determining parameter compensation values ​​of the unpruned columns in the pruning weight matrix based on the rearrangement evaluation matrix; Parameter compensation is performed on the unpruned columns in the pruning weight matrix based on the parameter compensation values ​​to obtain the target weight matrix.

9. The text processing method according to claim 8, characterized in that: Determining a target row corresponding to the target column in the evaluation matrix includes: Constructing a correspondence between the rows of the initial weight matrix and the columns of the evaluation matrix; The target row corresponding to the target column in the evaluation matrix is ​​determined based on the corresponding relationship.

10. A model processing method, characterized in that: include: Obtaining an initial weight matrix of a target model and an evaluation matrix corresponding to the initial weight matrix, wherein the initial weight matrix is ​​used to represent an initial weight matrix to be pruned in the target model, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model; Performing pruning evaluation on the initial weight matrix based on the evaluation matrix to obtain a target column; The initial weight matrix is ​​pruned based on the target column to obtain a target weight matrix.

11. A text processing method, characterized in that: include: In response to an input instruction acting on the operation interface, displaying the text to be processed on the operation interface; In response to a text processing instruction on the operation interface, a text processing result is displayed on the operation interface, wherein the text processing result is obtained by inferring the text to be processed using a target model, the target weight matrix of the target model is obtained by pruning the initial weight matrix of the target model according to a target column, the target column is obtained by pruning and evaluating the initial weight matrix based on an evaluation matrix, the initial weight matrix is ​​used to represent the initial weight matrix to be pruned in the target model, and the evaluation matrix is ​​used to quantify the impact of weight changes in the initial weight matrix on the performance of the target model.

12. A text processing method, characterized in that: include: Acquire the text to be processed by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter includes the text to be processed; The target model is used to infer the text to be processed to obtain a text processing result, wherein the target weight matrix of the target model is obtained by pruning the initial weight matrix of the target model according to the target column, and the target column is obtained by pruning and evaluating the initial weight matrix based on the evaluation matrix, and the evaluation matrix is ​​used to quantify the impact of the weight change in the initial weight matrix on the performance of the target model; The text processing result is output by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter includes the text processing result.

13. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 12 when running.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 12.

15. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images