Large model reasoning acceleration method and device, computer equipment, readable storage medium and program product
By filtering the redundant layers in the large model and generating a list of discarded layers, the problems of high memory usage and slow inference speed caused by excessive large model parameters are solved, and the inference speed of the large model is improved without reducing performance.
Patent Information
- Application Number
- CN202411754025.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-02
- Publication Date
- 2025-03-18
AI Technical Summary
The explosive growth of the number of parameters in the large model has led to large memory usage, high deployment costs and insufficient inference speed, which cannot meet users' needs for high performance and fast inference.
By obtaining the weight file of the target big model, setting discarded parameters, initializing the model, and using similarity-based measurement methods to calculate the input and output similarity scores of each layer, filtering out the redundant layer, generating a discarded layer list, updating the model configuration file, and performing large-scale model inference.
It reduces the amount of parameters of the large model, reduces the memory usage and computing resource consumption, improves the inference speed of the large model, and maintains the performance without degradation.
Smart Images
Figure CN119647598A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a large model reasoning acceleration method, device, computer equipment, computer-readable storage medium and computer program product. Background Art
[0002] With the development of artificial intelligence technology, large models with massive parameters and complex structures have emerged. Such models usually require a lot of data and computing resources when training, but can also handle more complex and diverse tasks. With the improvement of computing power and the availability of large-scale data sets, large models have been widely used in recent years, especially in natural language processing, computer vision, and speech recognition.
[0003] However, in order to achieve better performance, the parameters of large models have increased explosively, and some even reach hundreds of billions of parameters, which requires a large amount of memory, greatly increasing the deployment cost and the demand for computing resources. At the same time, users have higher and higher requirements for the inference speed of large models, resulting in a contradiction between the performance of large models and the inference speed. Summary of the invention
[0004] Based on this, it is necessary to provide a large model reasoning acceleration method, device, computer equipment, computer readable storage medium and computer program product to address the above technical problems.
[0005] In a first aspect, the present application provides a large model reasoning acceleration method, comprising:
[0006] Obtain a weight file of a target large model, and set a discard parameter of the target large model; the discard parameter includes a discard strategy and a discard layer number;
[0007] Initializing the target large model according to the weight file, and setting a calibration data set of the target large model;
[0008] Calculating input-output similarity scores of each layer in the target large model according to a similarity-based metric and the calibration data set;
[0009] Arrange the input-output similarity scores in descending order, and extract a discarding layer list from high to low in the arranged input-output similarity scores according to the discarding strategy and the number of discarding layers;
[0010] According to the discarded layer list, the configuration file of the target large model is saved, and large model inference is performed based on the weight file and the configuration file.
[0011] In one embodiment, initializing the target large model according to the weight file includes:
[0012] The weight file is loaded using a preset target library to obtain a loaded target weight file; according to the target weight file, the target large model is traversed to initialize the similarity sets of all layers in the target large model.
[0013] In one embodiment, the step of calculating the input-output similarity scores of each layer in the target large model according to the similarity-based measurement method and the calibration data set includes:
[0014] Identify the attention layer and the perceptron (MLP, Multilayer PerceptronLayer) layer in the target large model; for the attention layer, calculate the first cosine similarity between the input and output in the attention layer according to the similarity-based measurement method and the calibration data set as the input-output similarity score; for the perceptron layer, calculate the second cosine similarity between the input and output in the perceptron layer according to the similarity-based measurement method and the calibration data set as the input-output similarity score.
[0015] In one embodiment, calculating the first cosine similarity between the input and output in the attention layer according to the similarity-based metric method and the calibration data set includes:
[0016] According to the similarity-based measurement method and the calibration data set, determine the first input of the layer normalization (Layer Norm) layer before the attention layer and the first output of the attention layer; input the first input and the first output into a preset first similarity calculation model to obtain the first cosine similarity output by the first similarity calculation model.
[0017] In one embodiment, calculating the second cosine similarity between the input and output in the perceptron layer according to the similarity-based measurement method and the calibration data set includes:
[0018] According to the similarity-based measurement method and the calibration data set, determine the second input of the layer normalization layer before the perceptron layer and the second output of the perceptron layer; input the second input and the second output into a preset second similarity calculation model to obtain the second cosine similarity output by the second similarity calculation model.
[0019] In one embodiment, the step of saving the configuration file of the target large model according to the discarded layer list includes:
[0020] According to the discarded layer list, determine the to-be-retained layer in the target large model; map the to-be-retained layer, and update the index of the to-be-retained layer according to the mapping result; save the updated index of the to-be-retained layer as the configuration file.
[0021] In a second aspect, the present application also provides a large model reasoning acceleration device, including:
[0022] A weight acquisition module, used to acquire a weight file of a target large model and set a discard parameter of the target large model; the discard parameter includes a discard strategy and a discard layer number;
[0023] A model processing module, used to initialize the target large model according to the weight file and set a calibration data set of the target large model;
[0024] A score calculation module, used to calculate the input-output similarity score of each layer in the target large model according to a similarity-based measurement method and the calibration data set;
[0025] A list interception module, used to arrange the input-output similarity scores in descending order, and intercept the discarding layer list from high to low in the arranged input-output similarity scores according to the discarding strategy and the number of discarding layers;
[0026] The model inference module is used to save the configuration file of the target large model according to the discarded layer list, and perform large model inference based on the weight file and the configuration file.
[0027] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0028] Obtain a weight file of a target large model, and set discarding parameters of the target large model; the discarding parameters include a discarding strategy and a discarding layer number; initialize the target large model according to the weight file, and set a calibration data set for the target large model; calculate the input-output similarity scores of each layer in the target large model according to a similarity-based measurement method and the calibration data set; arrange the input-output similarity scores in descending order, and according to the discarding strategy and the discarding layer number, extract a discarding layer list from high to low in the arranged input-output similarity scores; save the configuration file of the target large model according to the discarding layer list, and perform large model inference based on the weight file and the configuration file.
[0029] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:
[0030] Obtain a weight file of a target large model, and set discarding parameters of the target large model; the discarding parameters include a discarding strategy and a discarding layer number; initialize the target large model according to the weight file, and set a calibration data set for the target large model; calculate the input-output similarity scores of each layer in the target large model according to a similarity-based measurement method and the calibration data set; arrange the input-output similarity scores in descending order, and according to the discarding strategy and the discarding layer number, extract a discarding layer list from high to low in the arranged input-output similarity scores; save the configuration file of the target large model according to the discarding layer list, and perform large model inference based on the weight file and the configuration file.
[0031] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:
[0032] Obtain a weight file of a target large model, and set discarding parameters of the target large model; the discarding parameters include a discarding strategy and a discarding layer number; initialize the target large model according to the weight file, and set a calibration data set for the target large model; calculate the input-output similarity scores of each layer in the target large model according to a similarity-based measurement method and the calibration data set; arrange the input-output similarity scores in descending order, and according to the discarding strategy and the discarding layer number, extract a discarding layer list from high to low in the arranged input-output similarity scores; save the configuration file of the target large model according to the discarding layer list, and perform large model inference based on the weight file and the configuration file.
[0033] The above-mentioned large model reasoning acceleration method, device, computer equipment, computer-readable storage medium and computer program product initialize the target large model according to the weight file of the target large model, and set the calibration data set of the target large model, and then calculate the input-output similarity score of each layer in the target large model according to the similarity-based measurement method and the calibration data set, and then arrange the input-output similarity score in descending order, and according to the discarding strategy and the number of discarded layers, intercept the discarded layer list from high to low in the arranged input-output similarity score, and finally save the configuration file of the target large model according to the discarded layer list, and perform large model reasoning based on the weight file and the configuration file. In summary, the entire technical scheme proposes a similarity-based measurement method, which can measure the similarity between the input and output of each layer in the large model, and screen out redundant layers, that is, layers that can be discarded, through the similarity score. Discarding these layers does not affect the performance of the large model, thereby greatly reducing the number of parameters of the large model, reducing the occupation of the large model for video memory, reducing the deployment cost and the consumption of computing resources, thereby improving the reasoning speed of the large model without reducing the performance of the large model. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0035] Figure 1 A diagram showing an application environment of a large model reasoning acceleration method in an embodiment;
[0036] Figure 2 A schematic diagram of a flow chart of a large model reasoning acceleration method in an embodiment;
[0037] Figure 3 A schematic diagram of discarding an attention layer in one embodiment;
[0038] Figure 4 A schematic diagram of discarding the MLP layer in one embodiment;
[0039] Figure 5 A schematic diagram of a process for calculating a similarity score in one embodiment;
[0040] Figure 6 A schematic diagram of a flow chart of a large model reasoning acceleration method in a specific embodiment;
[0041] Figure 7 It is a structural block diagram of a large model reasoning acceleration device in one embodiment;
[0042] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0044] The large model reasoning acceleration method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal communicates with the server through the network. The data storage system can store the data that the server needs to process. The data storage system can be integrated on the server or placed on the cloud or other network servers.
[0045] Specifically, the large model reasoning acceleration method provided in the embodiment of the present application can be executed by a server.
[0046] Exemplarily, the server obtains the weight file of the target large model and sets the discarding parameters of the target large model; the discarding parameters include the discarding strategy and the number of discarding layers; then, according to the weight file, the target large model is initialized, and the calibration data set of the target large model is set; then, according to the similarity-based measurement method and the calibration data set, the input-output similarity scores of each layer in the target large model are calculated; then, the input-output similarity scores are arranged in descending order, and according to the discarding strategy and the number of discarding layers, a discarding layer list is intercepted from high to low in the arranged input-output similarity scores; finally, according to the discarding layer list, the configuration file of the target large model is saved, and the large model inference is performed based on the weight file and the configuration file.
[0047] In such Figure 1 In the application environment shown, the terminal can be, but is not limited to, various personal computers, laptops, smart phones, and tablet computers. The server can be implemented as an independent server or a server cluster consisting of multiple servers.
[0048] In one embodiment, Figure 2 As shown in the figure, a large model reasoning acceleration method is provided, which is applied to Figure 1 The server in the example is used to illustrate the following steps:
[0049] Step S201, obtain the weight file of the target large model, and set the discarding parameters of the target large model; the discarding parameters include the discarding strategy and the number of discarding layers.
[0050] Among them, the weight file can be the large model weight obtained by the user's own training, or it can be the open source large model weight downloaded from an open source website. It should be noted that the weight file should contain a pt format file.
[0051] Among them, the discarding strategy can be to regard the MLP layer and its preceding Layer Norm layer as a discarding unit. Similarly, this application regards the attention layer and its preceding Layer Norm layer as a discarding unit, such as Figure 3 and Figure 4 As shown, the discarded MLP layer and attention layer mentioned in this application also include the Layer Norm layer immediately preceding it; the number of discarded layers refers to the number of discarded layers for the current large model parameters.
[0052] Specifically, the server obtains the weight file of the target large model, and sets the discarding strategy and the number of discarding layers of the target large model as the discarding parameters of the target large model.
[0053] Step S202, initializing the target large model according to the weight file, and setting the calibration data set of the target large model.
[0054] Specifically, the server uses the most commonly used transformers library to load the weight file of the target large model to initialize the target large model, and sets the calibration dataset of the target large model.
[0055] Step S203, calculating the input-output similarity score of each layer in the target large model according to the similarity-based measurement method and the calibration data set.
[0056] It should be noted that this application uses the One-shot method, that is, one round of calculation can obtain the similarity between the input and output of each layer in the target large model, thereby avoiding the waste of computing resources caused by multiple rounds of iterations.
[0057] Specifically, the server uses the calibration data set and the similarity-based measurement method proposed in this application to calculate the input-output similarity scores of each layer in the target large model.
[0058] Step S204, the input-output similarity scores are arranged in descending order, and a discarding layer list is intercepted from high to low in the arranged input-output similarity scores according to the discarding strategy and the number of discarding layers.
[0059] Specifically, the server calculates the input-output similarity scores of each layer in the target large model through a one-shot method, and then sorts the similarity scores from high to low. A list of discarded layers with the top rankings is intercepted from high to low in the arranged input-output similarity scores. The list contains the index of the layer in the large model, that is, the top number is the number of discarded layers described in step S201.
[0060] Step S205, save the configuration file of the target large model according to the discarded layer list, and perform large model inference based on the weight file and the configuration file.
[0061] Specifically, the server saves the configuration file of the target large model according to the discarded layer list. Once the list of layers to be discarded is known, the layers to be retained are remapped, given new indexes, and saved as configuration files. The large model is then inferred using the obtained configuration files and weight files, and the weight file of the target large model is loaded using the transformers library. During the loading process, the configuration file is automatically identified, and those discarded layers will not be loaded.
[0062] In the above-mentioned large model reasoning acceleration method, the target large model is initialized according to the weight file of the target large model, and the calibration data set of the target large model is set. Then, according to the similarity-based measurement method and the calibration data set, the input-output similarity scores of each layer in the target large model are calculated, and then the input-output similarity scores are arranged in descending order. According to the discarding strategy and the number of discarded layers, the discarded layer list is intercepted from high to low in the arranged input-output similarity scores. Finally, according to the discarded layer list, the configuration file of the target large model is saved, and the large model reasoning is performed based on the weight file and the configuration file. In summary, the entire technical solution proposes a similarity-based measurement method, which can measure the similarity between the input and output of each layer in the large model, and through the similarity score, the redundant layers, that is, the layers that can be discarded, are screened out. Discarding these layers does not affect the performance of the large model, thereby greatly reducing the number of parameters of the large model, reducing the large model's occupancy of video memory, reducing deployment costs and consumption of computing resources, thereby improving the reasoning speed of the large model without reducing the performance of the large model.
[0063] In one embodiment, in the above step S202, the target large model is initialized according to the weight file, which specifically includes the following steps:
[0064] Use the preset target library to load the weight file to obtain the loaded target weight file; according to the target weight file, traverse the target large model and initialize the similarity sets of all layers in the target large model.
[0065] The target library may be a transformer library, which is a powerful library widely used in the field of natural language processing. It provides a comprehensive set of tools and pre-trained models for performing various natural language processing tasks.
[0066] Specifically, the server uses the most commonly used transformers library to load the weight file of the target large model to obtain the loaded target weight file; according to the target weight file, it traverses all layers of the target large model, initializes the similarity sets of all layers in the target large model, and obtains the initialized target large model.
[0067] In one embodiment, if Figure 5 As shown, in the above step S203, the input-output similarity scores of each layer in the target large model are calculated according to the similarity-based measurement method and the calibration data set, which specifically includes the following steps:
[0068] Step S501, identify the attention layer and MLP layer in the target large model.
[0069] Step S502, for the attention layer, according to the similarity-based measurement method and the calibration data set, calculate the first cosine similarity between the input and output in the attention layer as the input-output similarity score.
[0070] Step S503, for the MLP layer, according to the similarity-based measurement method and the calibration data set, calculate the second cosine similarity between the input and output in the attention layer as the input-output similarity score.
[0071] Among them, the attention layer is an important neural network layer in deep learning. The core idea is to allow the model to focus on the most meaningful part of the input data for the current task by assigning different attention weights, rather than processing all inputs equally.
[0072] Among them, the MLP layer is a basic layer structure in a neural network with multiple layers of neurons, which learns complex patterns and features in the input data through nonlinear activation functions.
[0073] Specifically, the server first identifies the attention layer and the MLP layer in the target large model; for the attention layer, according to the similarity-based measurement method and the calibration data set, using the first similarity calculation model, calculates the first cosine similarity between the input and output in the attention layer as the input-output similarity score; for the MLP layer, according to the similarity-based measurement method and the calibration data set, using the second similarity calculation model, calculates the second cosine similarity between the input and output in the attention layer as the input-output similarity score, thereby quickly and accurately calculating the input-output similarity scores of each layer in the target large model.
[0074] In one embodiment, in the above step S503, for the MLP layer, according to the similarity-based measurement method and the calibration data set, calculating the second cosine similarity between the input and output in the attention layer specifically includes the following steps:
[0075] According to a similarity-based measurement method and a calibration data set, a second input of a Layer Norm layer before the MLP layer and a second output of the MLP layer are determined; the second input and the second output are input into a preset second similarity calculation model to obtain a second cosine similarity output by the second similarity calculation model.
[0076] Specifically, Figure 5 As shown, the MLP layer comes after the Layer Norm layer, which normalizes the input. The input of the Layer Norm layer before the MLP layer at the block is , its output can be expressed as , output It contains two parts, one of which is the output of the MLP layer. Therefore, if only the similarity between the input and output of the MLP layer is considered to decide whether to discard it, it will lead to inaccurate measurement. This application regards the MLP layer and the Layer Norm layer in front of it as a discarding unit, that is, the discarded MLP layer mentioned in this application also includes the Layer Norm layer immediately preceding it.
[0077] At this time, the proposed second cosine similarity can be obtained by the following calculation formula:
[0078]
[0079] In the above formula, Represents the cosine similarity between the input and output.
[0080] In one embodiment, in the above step S502, for the attention layer, according to the similarity-based measurement method and the calibration data set, calculating the first cosine similarity between the input and output in the attention layer specifically includes the following steps:
[0081] According to a similarity-based measurement method and a calibration data set, a first input of a Layer Norm layer before an attention layer and a first output of the attention layer are determined; the first input and the first output are input into a preset first similarity calculation model to obtain a first cosine similarity output by the first similarity calculation model.
[0082] Specifically, Figure 4 As shown in , the calculation method of the attention layer is similar to that of the MLP layer. The input of the Layer Norm layer before the attention layer at the block is , its output can be expressed as ,in That is the attention layer mentioned above; similarly, this application regards the attention layer and the Layer Norm layer before it as a discard unit.
[0083] At this time, the proposed first cosine similarity can be obtained by the following calculation formula:
[0084]
[0085] Similarly, in the above formula, Represents the cosine similarity between the input and output.
[0086] In one embodiment, in the above step S205, according to the discarded layer list, saving the configuration file of the target large model specifically includes the following steps:
[0087] According to the discarded layer list, determine the layers to be retained in the target large model; map the layers to be retained, and update the indexes of the layers to be retained according to the mapping results; save the updated indexes of the layers to be retained as configuration files.
[0088] Specifically, the server knows the list of discarded layers, remaps the layers to be retained, assigns them new indexes, saves the updated indexes of the layers to be retained as configuration files, and then performs reasoning using the obtained configuration files and the weight file of the target large model. The weight file of the target large model is loaded using the transformers library. During the loading process, the configuration files are automatically identified, and the discarded layers will not be loaded, thereby improving the reasoning speed. The performance achieved by this method is equivalent to that before the layers were discarded.
[0089] In one embodiment, Figure 6 As shown, a large model reasoning acceleration method in a specific embodiment is provided, which specifically includes the following steps:
[0090] Step S601, obtain the weight file of the target large model, and set the discarding parameters of the target large model; the discarding parameters include the discarding strategy and the number of discarding layers.
[0091] Step S602, use the preset target library to load the weight file to obtain the loaded target weight file; according to the target weight file, traverse the target large model, initialize the similarity sets of all layers in the target large model, and set the calibration data set of the target large model.
[0092] Step S603, identifying the attention layer and MLP layer in the target large model.
[0093] Step S604, for the attention layer, according to the similarity-based measurement method and the calibration data set, determine the first input of the Layer Norm layer before the attention layer and the first output of the attention layer; input the first input and the first output into a preset first similarity calculation model to obtain the first cosine similarity output by the first similarity calculation model as the input-output similarity score.
[0094] Step S605, for the MLP layer, according to the similarity-based measurement method and the calibration data set, determine the second input of the Layer Norm layer before the MLP layer and the second output of the MLP layer; input the second input and the second output into a preset second similarity calculation model to obtain the second cosine similarity output by the second similarity calculation model as the input-output similarity score.
[0095] Step S606, arranging the input-output similarity scores in descending order, and extracting a discarding layer list from high to low in the arranged input-output similarity scores according to the discarding strategy and the number of discarding layers.
[0096] Step S607, determine the layers to be retained in the target large model according to the discarded layer list; map the layers to be retained, and update the indexes of the layers to be retained according to the mapping results; save the updated indexes of the layers to be retained as configuration files, and perform large model inference based on the weight files and the configuration files.
[0097] The beneficial effects brought by the above embodiments are as follows:
[0098] 1) This application proposes a large model reasoning acceleration method that jointly discards the attention layer and the MLP layer. Specifically, it can be applied to the field of large model reasoning acceleration. This application improves the large model reasoning speed while maintaining the performance of the large model. This method is simple and easy to use, and does not affect downstream tasks, such as large model quantization.
[0099] 2) This application proposes a similarity-based measurement method that can measure the similarity between the input and output of each layer of the large model. Through the similarity score, redundant layers can be screened out, that is, layers that can be discarded. Discarding these layers does not affect the performance of the large model. By using this method, the number of parameters of the large model is greatly reduced, the memory usage is reduced, and the consumption of computing resources is reduced, thereby improving the reasoning speed of the large model.
[0100] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0101] Based on the same inventive concept, the embodiment of the present application also provides a large model reasoning acceleration device for implementing the large model reasoning acceleration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more large model reasoning acceleration device embodiments provided below can refer to the limitations of the large model reasoning acceleration method above, and will not be repeated here.
[0102] In an exemplary embodiment, Figure 7 As shown, a large model reasoning acceleration device is provided, comprising:
[0103] The weight acquisition module 701 is used to obtain the weight file of the target large model and set the discarding parameters of the target large model; the discarding parameters include the discarding strategy and the number of discarding layers;
[0104] The model processing module 702 is used to initialize the target large model according to the weight file and set the calibration data set of the target large model;
[0105] A score calculation module 703 is used to calculate the input-output similarity scores of each layer in the target large model according to a similarity-based measurement method and a calibration data set;
[0106] A list interception module 704 is used to sort the input-output similarity scores in descending order, and intercept the discarding layer list from high to low in the sorted input-output similarity scores according to the discarding strategy and the number of discarding layers;
[0107] The model inference module 705 is used to save the configuration file of the target large model according to the discarded layer list, and perform large model inference based on the weight file and the configuration file.
[0108] In one embodiment, the model processing module 702 is also used to load the weight file using a preset target library to obtain a loaded target weight file; according to the target weight file, traverse the target large model to initialize the similarity sets of all layers in the target large model.
[0109] In one embodiment, the score calculation module 703 is also used to identify the attention layer and the MLP layer in the target large model; for the attention layer, according to the similarity-based measurement method and the calibration data set, the first cosine similarity between the input and output in the attention layer is calculated as the input-output similarity score; for the MLP layer, according to the similarity-based measurement method and the calibration data set, the second cosine similarity between the input and output in the attention layer is calculated as the input-output similarity score.
[0110] In one embodiment, the score calculation module 703 is also used to determine the first input of the Layer Norm layer before the attention layer and the first output of the attention layer according to a similarity-based measurement method and a calibration data set; input the first input and the first output into a preset first similarity calculation model to obtain a first cosine similarity output by the first similarity calculation model.
[0111] In one embodiment, the score calculation module 703 is also used to determine the second input of the Layer Norm layer before the MLP layer and the second output of the MLP layer according to a similarity-based measurement method and a calibration data set; input the second input and the second output into a preset second similarity calculation model to obtain a second cosine similarity output by the second similarity calculation model.
[0112] In one embodiment, the model reasoning module 705 is also used to determine the to-be-retained layers in the target large model according to the discarded layer list; map the to-be-retained layers, and update the index of the to-be-retained layers according to the mapping results; and save the updated index of the to-be-retained layers as a configuration file.
[0113] Each module in the above-mentioned large model reasoning acceleration device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0114] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store weight file data of the target large model. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a large model reasoning acceleration method is implemented.
[0115] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0116] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0117] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0118] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0119] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0120] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0121] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0122] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A large model reasoning acceleration method, characterized in that: The method comprises: Obtain a weight file of a target large model, and set a discard parameter of the target large model; the discard parameter includes a discard strategy and a discard layer number; Initializing the target large model according to the weight file, and setting a calibration data set of the target large model; Calculating input-output similarity scores of each layer in the target large model according to a similarity-based metric and the calibration data set; Arrange the input-output similarity scores in descending order, and extract a discarding layer list from high to low in the arranged input-output similarity scores according to the discarding strategy and the number of discarding layers; According to the discarded layer list, the configuration file of the target large model is saved, and large model inference is performed based on the weight file and the configuration file.
2. The method according to claim 1, characterized in that Initializing the target large model according to the weight file includes: Use a preset target library to load the weight file to obtain a loaded target weight file; According to the target weight file, the target large model is traversed to initialize the similarity sets of all layers in the target large model.
3. The method according to claim 2, characterized in that The step of calculating the input-output similarity scores of each layer in the target large model according to the similarity-based measurement method and the calibration data set includes: Identify the attention layer and the perceptron layer in the target large model; For the attention layer, calculating the first cosine similarity between the input and output in the attention layer according to the similarity-based measurement method and the calibration data set as the input-output similarity score; For the perceptron layer, the second cosine similarity between the input and output in the attention layer is calculated according to the similarity-based measurement method and the calibration data set as the input-output similarity score.
4. The method according to claim 3, characterized in that The calculating, according to the similarity-based metric method and the calibration data set, a first cosine similarity between input and output in the attention layer, comprises: Determining a first input of a layer normalization layer before the attention layer and a first output of the attention layer according to the similarity-based metric and the calibration data set; The first input and the first output are input into a preset first similarity calculation model to obtain the first cosine similarity output by the first similarity calculation model.
5. The method according to claim 3, characterized in that: Calculating the second cosine similarity between the input and output in the perceptron layer according to the similarity-based measurement method and the calibration data set includes: Determine a second input of a layer normalization layer before the perceptron layer and a second output of the perceptron layer according to the similarity-based measurement method and the calibration data set; The second input and the second output are input into a preset second similarity calculation model to obtain the second cosine similarity output by the second similarity calculation model.
6. The method according to any one of claims 1 to 5, characterized in that: The step of saving the configuration file of the target large model according to the discarded layer list comprises: Determining the layers to be retained in the target large model according to the discarded layer list; Mapping the layer to be retained, and updating the index of the layer to be retained according to the mapping result; The updated index of the layer to be retained is saved as the configuration file.
7. A large model reasoning acceleration device, characterized in that: The device comprises: A weight acquisition module, used to acquire a weight file of a target large model and set a discard parameter of the target large model; the discard parameter includes a discard strategy and a discard layer number; A model processing module, used to initialize the target large model according to the weight file and set a calibration data set of the target large model; A score calculation module, used to calculate the input-output similarity score of each layer in the target large model according to a similarity-based measurement method and the calibration data set; A list interception module, used to arrange the input-output similarity scores in descending order, and intercept the discarding layer list from high to low in the arranged input-output similarity scores according to the discarding strategy and the number of discarding layers; The model inference module is used to save the configuration file of the target large model according to the discarded layer list, and perform large model inference based on the weight file and the configuration file.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.