Method and related device for optimizing an artificial intelligence large model

Abstract

The application provides an optimization method and related equipment of an artificial intelligence (AI) large model. The method comprises: inputting N training samples into an initial large model to obtain an initial logical value of each training sample; performing N times of iterative optimization on the size corresponding to each attention head according to the initial large model; obtaining a first large model and a first logical value of each training sample in the n-th optimization process; determining a first loss gradient of each training sample according to the initial logical value and the first logical value of each training sample; determining a first gradient projection of the n-th training sample according to the first loss gradient of the N training samples; and optimizing the size corresponding to each attention head in the first large model according to the first gradient projection of the n-th training sample to obtain a second size corresponding to each attention head. The method can reduce the occupation of the storage space of the AI large model in the inference process and reduce the influence on the inference accuracy of the model.

Description

[0001] This application claims priority to Russian Patent Application No. RU2025111139, filed with the Russian Federal Intellectual Property Office on April 29, 2025, entitled "An Optimization Method and Related Apparatus for a Large Artificial Intelligence Model", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and more specifically, to a method for optimizing a large AI model, a computing device, a cluster of computing devices, a computer program product, and a computer-readable storage medium. Background Technology

[0003] Currently, large-scale AI models are being used more and more widely. To achieve higher accuracy inference capabilities, these models require or generate a large amount of data during inference, leading to significant storage demands. Reducing storage space usage by deleting some parameters can easily result in a decrease in inference accuracy.

[0004] Therefore, how to reduce the storage space occupied by large AI models during inference and reduce the impact on the inference accuracy of the models has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides an optimization method for large AI models, a computing device, a computing device cluster, a computer program product, and a computer-readable storage medium, which can reduce the storage space occupied by large AI models during inference and reduce the impact on inference accuracy.

[0006] Firstly, a method for optimizing a large AI model is provided. This method includes: obtaining an initial large model, which comprises L layers, each of which includes at least one attention head, where L is a positive integer; inputting N training samples into the initial large model to obtain an initial logical value for each of the N training samples, where each training sample includes at least one word unit, and N is a positive integer; and performing N iterative optimizations on the size corresponding to each attention head in the initial large model based on the initial large model. Specifically, in the nth optimization process of the N iterative optimizations, n is a positive integer less than or equal to N. This method further includes: determining the size corresponding to each attention head in the initial large model as a first size for each attention head, thus obtaining a first large model; inputting N training samples into the first large model to obtain... Obtain the first logical value of each training sample from the N training samples; determine the first loss gradient of each training sample based on the initial logical value and the first logical value of each training sample from the N training samples; determine the first gradient projection of the nth training sample based on the first loss gradient of the nth training sample and the average of the first loss gradients of the N training samples; optimize the size corresponding to each attention head in the first large model based on the first gradient projection of the nth training sample to obtain the second size corresponding to each attention head, and use the second size corresponding to each attention head to obtain the second large model, the sum of the second sizes corresponding to each attention head is less than the sum of the sizes corresponding to each attention head in the initial large model.

[0007] In this embodiment, during the nth optimization of the size corresponding to each attention head in the large AI model, the first gradient projection is determined using the first loss gradient of the nth training sample and the average of the first loss gradients of the N training samples. This ensures that when optimizing the size corresponding to each attention head based on the first gradient projection, the sum of the optimized sizes of each attention head is less than the sum of the sizes of each attention head in the initial large model, thereby reducing the storage space occupied by the large AI model during inference. Furthermore, since the first gradient projection is determined based on the initial logical values ​​and first logical values ​​of the training samples, it can reduce the impact on the inference accuracy of the large AI model obtained after N iterations of optimization.

[0008] In some embodiments, each attention head in the initial large model is used to determine the logical value of a training sample, which is used to determine the output corresponding to that training sample. The output corresponding to each training sample includes at least one word.

[0009] In some embodiments, the size corresponding to each attention head is the size of the key-value vector generated by each attention head, which is used to determine the logical value of the training sample.

[0010] In some embodiments, the first loss gradient is used to represent how the difference between the initial logical value and the first logical value of the training sample changes with the size change corresponding to each attention head.

[0011] In conjunction with the first aspect, in some implementations, during the first optimization process of N iterations, the first size corresponding to each attention head is the same. The first size is the target size, which is determined based on the total target size and the number of attention heads in the initial large model.

[0012] In this embodiment, during the initial optimization of the size corresponding to the attention head, the size of each attention head can be set as the target size, ensuring that the size of each attention head is smaller than the initial size. During iterative optimization after the initial optimization of the size corresponding to the attention head, the sizes of different attention heads can be the same or different.

[0013] In conjunction with the first aspect, in some implementations, the first gradient projection of the nth training sample is the difference between the first loss gradient of the nth training sample and the average of the first loss gradients of the N training samples.

[0014] In this embodiment, the first gradient projection is determined by using the difference between the first loss gradient of the nth training sample and the average of the first loss gradients of the N training samples. This provides a gradient for the subsequent optimization process. After optimizing the size of each attention head in the large model using this gradient, the sum of the sizes of all attention heads in the optimized large model is equal to the target total size.

[0015] In conjunction with the first aspect, in some implementations, the first size corresponding to each attention head is optimized based on the first gradient projection of the nth training sample to determine the first optimized size corresponding to each attention head; the second size corresponding to each attention head is determined based on the first optimized size, average size, and target size corresponding to each attention head, and the average size corresponding to each attention head is determined based on the historical values ​​of the size corresponding to each attention head.

[0016] In conjunction with the first aspect, in some implementations, the second size corresponding to each attention head is determined based on the difference between the first optimized size corresponding to each attention head and the first value corresponding to each attention head, and the first value corresponding to each attention head is determined based on the difference between the average size corresponding to each attention head and the target size.

[0017] In this embodiment, the first optimized size corresponding to each attention head is adjusted according to the average size and target size of each attention head to determine the second size corresponding to each attention head, so that the sum of the second sizes corresponding to the attention heads in the large model is the target total size, thereby avoiding the unlimited increase of the size corresponding to the attention head during the optimization process.

[0018] In conjunction with the first aspect, in some implementations, when n is less than N, the second largest model is used as the first largest model in the (n+1)th optimization process of N iterations, where n+1 is a positive integer less than or equal to N; or, when n=N, the second largest model is used as the target large model, which is applied to the inference task.

[0019] For example, the application of this target large model to a reasoning task includes: the target large model is used to obtain an output result corresponding to a reasoning sample based on the reasoning sample. The reasoning sample includes at least one word character. The output result corresponding to the reasoning sample includes at least one word character.

[0020] In this embodiment, during the final optimization process of N iterations, the target large model is obtained by optimizing the size of the attention head in the first large model. During the optimization processes before the final optimization, the second large model obtained after each optimization can be used as the first large model in the next optimization process, thereby continuing to optimize the size of the attention head in the first large model to obtain the target large model.

[0021] In conjunction with the first aspect, in some implementations, after completing N iterations of optimization, the method further includes: obtaining the average value of the second size corresponding to each attention head; when the average value of the second size is greater than the target size, reducing the second size corresponding to one or more attention heads in the second largest model so that the average value of the size corresponding to each attention head in the second largest model is equal to the target size; when the average value of the second size is less than the target size, increasing the second size corresponding to one or more attention heads in the second largest model so that the average value of the size corresponding to each attention head in the second largest model is equal to the target size.

[0022] In this embodiment, after the last iteration of optimization, a second large model is obtained. If the sum of the sizes corresponding to the attention heads in the second large model is not equal to the target total size, the sizes corresponding to some attention heads are adjusted to obtain the target large model and achieve a larger compression ratio.

[0023] In conjunction with the first aspect, in some implementations, after completing N iterations of optimization, the method further includes: determining M third dimensions and M attention head sets corresponding to the target layer based on the second dimension corresponding to each attention head of the target layer, wherein the target layer belongs to L layers, the M third dimensions belong to the second dimensions corresponding to each attention head of the target layer, each attention head set in the M attention head sets includes at least one attention head of the target layer, the m-th third dimension in the M third dimensions corresponds to the m-th attention head set in the M attention head sets, M is a positive integer less than the number of attention heads in the target layer, and m is a positive integer less than or equal to M; determining the dimension corresponding to each attention head of the target layer as the third dimension corresponding to the attention head set to which each attention head belongs, thereby obtaining the target large model.

[0024] In this embodiment, since multiple attention heads in the target layer of the target large model belong to M attention head sets, and the third size corresponding to the attention heads in each attention head set is the same, the computing device only needs to call the attention kernel M times in the target layer when using the target large model for inference, thereby improving the inference speed.

[0025] In conjunction with the first aspect, in some implementations, M third dimensions and M attention head sets minimize the difference between the first total size and the second total size. The first total size includes the sum of the dimensions corresponding to each attention head in the target layer when the third dimension of the attention head set to which each attention head belongs is the same as the sum of the dimensions corresponding to each attention head. The second total size includes the sum of the dimensions corresponding to each attention head in the target layer when the second dimension of each attention head is the same as the sum of the dimensions corresponding to each attention head.

[0026] In this embodiment, by determining the difference between the first total size and the second total size, M third sizes and M attention head sets in the target layer can minimize the difference between the sum of the sizes corresponding to the attention heads in the target large model and the target total size, thereby achieving a larger compression ratio.

[0027] In conjunction with the first aspect, in some implementations, the first attention head in the target large model includes a compensation module. The compensation module is used to determine the first compensation data based on the first set of value vectors. The first set of value vectors includes at least one value vector generated by the first attention head. The first output data of the first attention head is determined based on the first attention output and the first compensation data. The first attention output is determined based on the query vector corresponding to the first input data and the first set of key-value vectors. The first key-value vector includes at least one key-value vector generated by the first attention head.

[0028] In conjunction with the first aspect, in some implementations, the first output data is determined based on a weighted sum of the first attention output and the first compensation data.

[0029] In this embodiment, the value vector obtained by the attention head is processed by the compensation module in the attention head to obtain compensation data. The attention output of the attention head is then compensated based on the compensation data, thereby reducing the loss of inference accuracy caused by compressing the size of the attention head and improving the inference accuracy of the target large model.

[0030] In conjunction with the first aspect, in some implementations, the compensation module is used to determine the second state value corresponding to the first compensation data and the first input data based on the first state value and the first value vector set corresponding to the first input data. The first state value corresponding to the first input data is the second state value corresponding to the second input data obtained when the first attention head processes the second input data. The second input data is the input data processed by the first attention head before processing the first input data.

[0031] In conjunction with the first aspect, in some implementations, the compensation module includes a state space model (SSM).

[0032] In this embodiment, SSM captures information about elements in the value vector that are lost due to the size of the compressed attention head, thereby further reducing the impact on the inference accuracy of large models.

[0033] Secondly, a computing device is provided. The computing device includes an acquisition module and a processing module. The acquisition module is used to: acquire an initial large model, the initial large model including L layers, each of the L layers including at least one attention head, where L is a positive integer. The processing module is used to: input N training samples into the initial large model, obtain the initial logical value of each training sample in the N training samples, each training sample including at least one word, where N is a positive integer; and perform N iterative optimizations on the size corresponding to each attention head in the initial large model; wherein, in the nth optimization process of the N iterative optimizations, n is a positive integer less than or equal to N. Specifically, the processing module is used to: determine the size corresponding to each attention head in the initial large model as a first size corresponding to each attention head, obtaining a first large model; and input the N training samples into the first large model, obtaining the first logical value of each training sample in the N training samples. The first loss gradient of each training sample is determined based on the initial logical value and the first logical value of each training sample in the N training samples. The first gradient projection of the nth training sample is determined based on the first loss gradient of the nth training sample and the average of the first loss gradients of the N training samples. Based on the first gradient projection of the nth training sample, the size corresponding to each attention head in the first large model is optimized to obtain the second size corresponding to each attention head. The second size corresponding to each attention head is used to obtain the second large model. The sum of the second sizes corresponding to each attention head is less than the sum of the sizes corresponding to each attention head in the initial large model.

[0034] In some embodiments, each attention head in the initial large model is used to determine the logical value of a training sample, which is used to determine the output corresponding to that training sample. The output corresponding to each training sample includes at least one word.

[0035] In some embodiments, the size corresponding to each attention head is the size of the key-value vector generated by each attention head, which is used to determine the logical value of the training sample.

[0036] In some embodiments, the first loss gradient is used to represent how the difference between the initial logical value and the first logical value of the training sample changes with the size change corresponding to each attention head.

[0037] In conjunction with the second aspect, in some implementations, during the first optimization process of N iterations, the first size corresponding to each attention head is the same. The first size is the target size, which is determined based on the total target size and the number of attention heads in the initial large model.

[0038] In conjunction with the second aspect, in some implementations, the projection of the first gradient of the nth training sample is the difference between the first loss gradient of the nth training sample and the average of the first loss gradients of the N training samples.

[0039] In conjunction with the second aspect, in some implementations, the processing module is further configured to: optimize the first size corresponding to each attention head based on the first gradient projection of the nth training sample, and determine the first optimized size corresponding to each attention head; determine the second size corresponding to each attention head based on the first optimized size, average size and target size corresponding to each attention head, wherein the average size corresponding to each attention head is determined based on the historical values ​​of the size corresponding to each attention head.

[0040] In conjunction with the second aspect, in some implementations, the second size corresponding to each attention head is determined based on the difference between the first optimized size corresponding to each attention head and the first value corresponding to each attention head, and the first value corresponding to each attention head is determined based on the difference between the average size corresponding to each attention head and the target size.

[0041] In conjunction with the second aspect, in some implementations, when n is less than N, the second largest model is used as the first largest model in the (n+1)th optimization process of N iterations, where n+1 is a positive integer less than or equal to N; or, when n=N, the second largest model is used as the target large model, which is applied to the inference task.

[0042] For example, the application of this target large model to a reasoning task includes: the target large model is used to obtain an output result corresponding to a reasoning sample based on the reasoning sample. The reasoning sample includes at least one word character. The output result corresponding to the reasoning sample includes at least one word character.

[0043] In conjunction with the second aspect, in some implementations, after completing N iterations of optimization, the processing module is further used to: obtain the average value of the second size corresponding to each attention head; when the average value of the second size is greater than the target size, decrease the second size corresponding to one or more attention heads in the second largest model so that the average value of the size corresponding to each attention head in the second largest model is equal to the target size; when the average value of the second size is less than the target size, increase the second size corresponding to one or more attention heads in the second largest model so that the average value of the size corresponding to each attention head in the second largest model is equal to the target size.

[0044] In conjunction with the second aspect, in some implementations, after completing N iterations of optimization, the processing module is used to: determine M third dimensions and M attention head sets corresponding to the target layer based on the second dimension corresponding to each attention head of the target layer, wherein the target layer belongs to L layers, the M third dimensions belong to the second dimensions corresponding to each attention head of the target layer, each attention head set in the M attention head sets includes at least one attention head of the target layer, the m-th third dimension in the M third dimensions corresponds to the m-th attention head set in the M attention head sets, M is a positive integer less than the number of attention heads in the target layer, and m is a positive integer less than or equal to M; and determine the dimension corresponding to each attention head of the target layer as the third dimension corresponding to the attention head set to which each attention head belongs, thereby obtaining the target large model.

[0045] In conjunction with the second aspect, in some implementations, M third dimensions and M attention head sets minimize the difference between the first total size and the second total size. The first total size includes the sum of the dimensions corresponding to each attention head in the target layer when the size corresponding to each attention head is the third dimension corresponding to the attention head set to which each attention head belongs. The second total size includes the sum of the dimensions corresponding to each attention head in the target layer when the size corresponding to each attention head is the second dimension corresponding to each attention head.

[0046] In conjunction with the second aspect, in some implementations, the first attention head in the target large model includes a compensation module. The compensation module is used to determine the first compensation data based on the first set of value vectors. The first set of value vectors includes at least one value vector generated by the first attention head. The first output data of the first attention head is determined based on the first attention output and the first compensation data. The first attention output is determined based on the query vector corresponding to the first input data of the first attention head and the first set of key-value vectors. The first key-value vector includes at least one key-value vector generated by the first attention head.

[0047] In conjunction with the second aspect, in some implementations, the first output data is determined based on a weighted sum of the first attention output and the first compensation data.

[0048] In conjunction with the second aspect, in some implementations, the compensation module is used to determine the second state value corresponding to the first compensation data and the first input data based on the first state value and the first value vector set corresponding to the first input data. The first state value corresponding to the first input data is the second state value corresponding to the second input data obtained when the first attention head processes the second input data. The second input data is the input data processed by the first attention head before processing the first input data.

[0049] In conjunction with the second aspect, in some implementations, the compensation module includes SSM.

[0050] It should be understood that the technical effects achieved by the second aspect or some implementations of the second aspect are similar to those achieved by the first aspect or some implementations of the first aspect, and will not be elaborated here.

[0051] Thirdly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method in the first aspect or any possible implementation of the first aspect.

[0052] Fourthly, this application provides a computer program product containing instructions that, when executed by a cluster of computer devices, cause the cluster of computer devices to perform the method described in the first aspect or any possible implementation thereof.

[0053] Fifthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method described in the first aspect or any possible implementation thereof.

[0054] In a sixth aspect, a chip system is provided, the chip system including logic circuitry for coupling with an input / output interface, through which data is transmitted to perform the method described in the first aspect or any possible implementation thereof. Attached Figure Description

[0055] Figure 1 This is a schematic structural diagram of the large model 100.

[0056] Figure 2 This is a schematic diagram of the cloud scenario.

[0057] Figure 3 This is a schematic flowchart of an AI large model optimization method according to an embodiment of this application.

[0058] Figure 4 This is a schematic diagram of projecting the gradient corresponding to the training sample into the size budget subspace according to an embodiment of this application.

[0059] Figure 5 This is a schematic flowchart of an AI large model optimization method according to another embodiment of this application.

[0060] Figure 6 This is a schematic diagram of an optimization method for a large AI model according to another embodiment of this application.

[0061] Figure 7This is a schematic flowchart of a reasoning method for a target large model according to an embodiment of this application.

[0062] Figure 8 This is a schematic structural diagram of a compensation module according to an embodiment of this application.

[0063] Figure 9 This is a schematic diagram of the reasoning process of a target large model according to an embodiment of this application.

[0064] Figure 10 This is a schematic diagram of the training process of a compensation module according to an embodiment of this application.

[0065] Figure 11 This is a schematic structural diagram of a computing device according to an embodiment of the present application.

[0066] Figure 12 This is a schematic structural diagram of a computing device according to an embodiment of this application.

[0067] Figure 13 This is a schematic structural diagram of a computing device cluster according to an embodiment of this application.

[0068] Figure 14 This is a schematic diagram showing the connection between computing devices 1200A and 1200B via a network according to an embodiment of this application. Detailed Implementation

[0069] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0070] This application will present various aspects, embodiments, or features relating to a system comprising multiple devices, components, modules, etc. It should be understood and appreciated that individual systems may include additional devices, components, modules, etc., and / or may not include all the devices, components, modules, etc. discussed in conjunction with the accompanying drawings. Furthermore, combinations of these approaches are also possible.

[0071] Furthermore, in the embodiments of this application, the words "exemplary," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in the embodiments of this application should not be construed as being better or more advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner. The business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0072] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless otherwise specifically emphasized.

[0073] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "OR" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0074] To facilitate understanding of the technical solutions in the embodiments of this application, some terms involved in the embodiments of this application are introduced below.

[0075] 1. Token In the field of AI, a lexical unit is typically the smallest unit obtained after segmenting text. A lexical unit includes at least one of the following: a number, a letter, a symbol, or a character. For example, a lexical unit may include a word, a subword, at least one letter, at least one number, at least one symbol, or at least one character.

[0076] 2. Large AI Model Large AI models typically refer to machine learning models with a large number of parameters and complex structures, used in fields such as natural language processing, computer vision, and speech recognition. A typical example of a large AI model is the LLM (Low-Level Machine). An LLM is a deep learning model trained on a large amount of text data. An LLM can generate natural language text or understand the meaning of language text, thereby generating corresponding output results based on the input inference sample and providing feedback to the user. This inference sample includes at least one initial word, and the corresponding output result includes at least one generated word.

[0077] For example, a possible structural diagram of a large model is as follows: Figure 1 As shown. Figure 1 The large model 100 includes at least one layer, such as layer 110 and / or layer 120. Exemplarily, the large model 100 is an LLM model. Layer 110 and / or layer 120 are transformer layers.

[0078] For example, layer 110 includes a self-attention module 111 and a feed-forward network (FFN) 112. The self-attention module 111 determines at least one query vector and at least one key-value vector based on the input data of layer 110. The input data includes embedding vectors corresponding to the input data of the large model 100, such as embedding vectors corresponding to at least one word in a training or inference sample. Alternatively, the input data includes the output data of the layer preceding layer 110. The layer preceding layer 110 is similar to layer 110, and the way the layer preceding layer obtains its output data is similar to that of layer 110, which will not be described further here. Each key-value vector includes a key vector and a value vector. The self-attention module 111 is also used to perform self-attention computation based on the at least one query vector and the at least one key-value vector to obtain an attention output. The FFN 112 obtains the output data of layer 110 based on the attention output of the self-attention module 111. The FFN 112 includes two consecutive linear projections with a non-linear activation operation between them.

[0079] For example, the self-attention module 111 of layer 110 may include H attention heads, where H is a positive integer.

[0080] For example, when the self-attention module 111 in layer 110 includes an attention head, the steps for obtaining the attention output of the self-attention module 111 are as follows: Assume the input data of layer 110 is X=[X1, X2, ..., X...]. a1 ], where a1 is a positive integer. The attention head in the attention module 111 uses the input data X and the first weight matrix W Q Obtain the query vector Q, Q = W Q •X. This attention head utilizes the input data X and the second weight matrix W. K Obtain the key vector K, K = W K •X. This attention head utilizes the input data X and the third weight matrix W. V Obtain the value vector V, V = W V·X. Here, the first, second, and third weight matrices are the weight matrices in this attention head. · represents the dot product. This attention head also uses the query vector Q and the key vector K to obtain the attention score vector S. core S core =(Q·K T ) / d. Where, K T Let K be the transpose of K, and d be the dimension of the key vector. This attention head also utilizes the attention score vector S. core Obtain the attention weights P, P = softmax(S core ), where softmax() represents the normalized exponential function. This attention head also uses attention weights P and value vector V to obtain the attention output O from the attention module 111, O = P·V.

[0081] For example, when the self-attention module 111 in layer 110 includes multiple attention heads, each of the multiple attention heads is used to calculate an attention output, thereby obtaining multiple attention outputs. The self-attention module 111 is used to concatenate the multiple attention outputs together and fuse them through a linear layer, thereby obtaining the attention output of the self-attention module 111. The specific steps are as follows: Assume that the input data of layer 110 is X=[X1, X2, ..., X... a1 ], where a1 is a positive integer. The self-attention module 111 uses the input data X and the first weight matrix W of the h-th attention head among the H attention heads. Qh Obtain the query vector Q corresponding to the h-th attention head. h Q h = W Qh • X. h is a positive integer less than or equal to H. The self-attention module 111 utilizes the input data X and the second weight matrix W of the h-th attention head. Kh Obtain the key vector K corresponding to the h-th attention head. h K h = W Kh •X. The self-attention module 111 utilizes the input data X and the third weight matrix W of the h-th attention head. Vh Obtain the value vector V corresponding to the h-th attention head. h V h = W Vh ·X. The self-attention module 111 also utilizes the query vector Q corresponding to the h-th attention head. h and bond vector K h Obtain the attention score vector S corresponding to the h-th attention head. h S h = (Q) h ·K h T) / d h Among them, K h T K represents h transpose, d h Let S be the dimension of the key vector corresponding to the h-th attention head. The self-attention module 111 also utilizes the attention score vector S corresponding to the h-th attention head. h Obtain the attention weight P corresponding to the h-th attention head. h P h = softmax(S h The self-attention module 111 also utilizes the attention weight P corresponding to the h-th attention head. h The value vector V corresponding to the h-th attention head h Obtain the attention output O corresponding to the h-th attention head. h O h = P h ·V h , or O h =W O ·(P) h ·V h W O This represents the fourth weight matrix. The attention module 111 concatenates the attention outputs corresponding to each of the H attention heads to obtain the attention output vector O. all = [O1, O2, ..., O H ].

[0082] For example, in O h = P h ·V h At that time, the attention output O of the self-attention module 111 is O = W O ·O all In O h =W O ·(P) h ·V h When ), the attention output of the self-attention module 111 is O. all .

[0083] For example, the size corresponding to each attention head is the size of a single key vector (or value vector) generated by that attention head, or the size of a pair of key-value vectors generated by that attention head, or the total size of all key-value vectors generated by that attention head. The total size of all key-value vectors generated by the attention head = size of each key vector × number of key vectors generated by the attention head + size of each value vector × number of value vectors generated by the attention head. × represents multiplication, and + represents addition. The size of the key vector and the size of the value vector generated by each attention head are the same.

[0084] For example, the size of a vector includes each dimension of the vector, or the size of a vector includes the product of each dimension of the vector, or the size of a vector includes the number of elements in the vector, or the size of the storage space required to store the vector. The size of a pair of key-value vectors includes the sum of the sizes of the key vectors and the value vectors in the key-value vectors. The total size of multiple key-value vectors includes the sum of the sizes of each of the multiple key-value vectors.

[0085] For example, the output data of layer 110 can be used as the input data of the next layer after layer 110. For instance, when the next layer after layer 110 is layer 120, the input data of layer 120 includes the output data of layer 110.

[0086] For example, when layer 120 is the last layer of the large model, the output data of layer 120 is used to determine the output data of the large model. In other words, a generated lexical is determined based on the output data of layer 120. The output data of the large model includes the generated lexical, or at least one lexical adjusted based on the generated lexical.

[0087] For example, the inference steps of the large model include: (1) inputting the inference sample into the large model, which divides the inference sample into at least one initial token; (2) prefilling the at least one initial token to generate a key-value vector for each initial token and obtaining the first generated token; (3) decoding the key-value vector of the at least one initial token and the first generated token to generate a key-value vector for the first generated token and obtaining the second generated token; (4) decoding the key-value vector of the at least one initial token, the key-value vector of the first generated token, and the second generated token to generate a key-value vector for the second generated token and obtaining the third generated token, i.e., repeating step (3) until a preset token is obtained or the number of generated tokens obtained reaches a threshold; (5) outputting the inference result, which is determined based on at least one generated token. The large model includes multiple layers. During pre-filling in the large model, each layer obtains and stores the key-value vector of each initial word in at least one initial word corresponding to that layer. During decoding of the at least one initial word and the first generated word, each layer obtains the query vector and key-value vector of the first generated word corresponding to that layer. Each layer uses the query vector of the first generated word and the key-value vector of each initial word in at least one initial word to obtain its output. Each layer stores the key-value vector of the first generated word. The large model uses the output of each layer to obtain the second generated word. When the large model decodes the at least one initial word, the first generated word, and the second generated word, each layer in the large model obtains the query vector and key-value vector of the second generated word corresponding to that layer; each layer uses the query vector of the second generated word corresponding to that layer, the key-value vector of each initial word in at least one initial word, and the key-value vector of the first generated word to obtain the output result of that layer; each layer stores the key-value vector of the second generated word corresponding to that layer; the large model uses the output result of each layer to obtain the third generated word.

[0088] 3. Initial Lexical Elements After inputting training or inference samples into a large model, the large model performs text segmentation on the training or inference samples to obtain at least one word element included in the training or inference samples. Each word element in this at least one word element is called an "initial word element". That is, the initial word element is not a word element generated by the large model, but a word element included in the samples input into the large model.

[0089] 4. Generate word elements After training or inference samples are input into a large model, the model generates at least one word based on at least one initial word in the input samples. This generated word is called a "generated word." In other words, a generated word is not a word included in the input samples, but rather a word generated by the large model.

[0090] 5. Logical value (logit) After inputting training or inference samples into the large model, the large model uses each self-attention module and FFN in each layer for inference, obtaining the logistic value corresponding to each word in the output of the input sample. The logistic value corresponding to each word is the unnormalized score of the hidden state mapped to the vocabulary size by the fully connected layer in the last layer of the large model. The logistic value corresponding to each word can be transformed into a probability distribution through softmax, and the word can be determined by this probability distribution. The logistic value of the training sample includes the logistic value corresponding to each word in the output of the corresponding training sample. Similarly, the logistic value of the inference sample includes the logistic value corresponding to each word in the inference result of the corresponding inference sample.

[0091] For example, the last layer of the large model adds the attention output of the self-attention module of this layer to the output data of the large model to achieve a residual connection, and performs layer normalization on the result of the addition, so that the result of layer normalization is input into the FFN of this layer. The FFN of this layer obtains the output data of the FFN through fully connected layers and nonlinear activation layers. This layer performs residual connection and normalization again on the result of the above layer normalization and the output data of the FFN (i.e., normalizes the result of the above layer normalization and the result of the addition of the output data of the FFN), to obtain the output data of this layer. The large model uses the output data X of the last layer. out and vocabulary projection matrix W vocab Obtain a logical value L token L token = X out ·W vocab The large model uses softmax to transform the logistic value into a probability distribution. This probability distribution represents the likelihood that each word in the vocabulary is the next generating word, thus determining the next generating word. The logistic value corresponding to this next generating word is L. token .

[0092] 6. Prefill treatment After inputting training or inference samples into the large model, the large model pre-padded at least one initial word unit in the input samples to obtain the first generated word unit. Specifically, the large model utilizes all layers within the large model to pre-padded this at least one initial word unit to obtain the first generated word unit. The implementation of how each layer in the large model processes the input data to obtain the output data is described above.

[0093] During the parallel processing of the at least one initial lexical unit, the large model can also obtain the query vector and key-value vector of each initial lexical unit in each layer of the large model. The key-value vector of each initial lexical unit corresponding to each layer includes: a key vector and a value vector of the initial lexical unit obtained by the layer after inputting the input data corresponding to the initial lexical unit. When the layer is the first layer of the large model, the input data corresponding to the initial lexical unit includes the embedding vector corresponding to the initial lexical unit. When the layer is a layer in the large model other than the first layer, the input data corresponding to the initial lexical unit includes the output data obtained by the previous layer after processing the input data corresponding to the initial lexical unit.

[0094] For example, the key-value vector of each initial lexical unit in the at least one initial lexical unit corresponding to each layer of the large model is stored in a computing device or storage device so that the large model can use these key-value vectors in subsequent inference processes. The computing device includes a computing device for running the large model, and the storage device is connected to the computing device.

[0095] 7. Decoding Processing After the large model pre-fills at least one initial lexical unit to obtain the first generated lexical unit, it can determine a set of lexical units to be input. This set of lexical units to be input includes the at least one initial lexical unit and at least one generated lexical unit generated by the large model. For example, after obtaining the (n1-1)th generated lexical unit, the set of lexical units to be input includes: at least one initial lexical unit, and each generated lexical unit from the first generated lexical unit to the (n1-1)th generated lexical unit. Here, n1 is a positive integer greater than 1.

[0096] The large model decodes the input lexical set to obtain new generated lexical units. For example, if the input lexical set includes at least one initial lexical unit and each of the generated lexical units from the first to the (n1-1)th generated lexical unit, the large model decodes the input lexical set to obtain the n1th generated lexical unit. Specifically, the large model utilizes all layers within it to decode the input lexical set, thereby obtaining the n1th generated lexical unit. The implementation of each layer in the large model processing the input data to obtain the output data is described above.

[0097] During the processing of the input lexicon set, the large model can also obtain the query vector and key-value vector of the (n1-1)th generated lexicon corresponding to each layer of the large model. This (n1-1)th generated lexicon is the most recently obtained generated lexicon by the large model. The query vector of the (n1-1)th generated lexicon corresponding to each layer includes: the query vector of the (n1-1)th generated lexicon obtained by the layer after inputting the input data corresponding to the (n1-1)th generated lexicon. The key-value vector of the (n1-1)th generated lexicon corresponding to each layer includes: the key vector and value vector of the (n1-1)th generated lexicon obtained by the layer after inputting the input data corresponding to the (n1-1)th generated lexicon. When this layer is the first layer of the large model, the input data corresponding to the (n1-1)th generated lexicon includes the embedding vector corresponding to the (n1-1)th generated lexicon. When this layer is a layer other than the first layer in the large model, the input data corresponding to the (n1-1)th generated word includes the output data obtained by the previous layer after processing the input data corresponding to the (n1-1)th generated word. When the (n1-1)th generated word is processed in each layer of the large model, the large model can also calculate the query vector of the (n1-1)th generated word corresponding to the layer and the key-value vector of each generated word in the set of words to be input corresponding to the layer, excluding the (n1-1)th generated word, in each layer to obtain the output data of the layer, and then obtain the (n1)th generated word based on the output data of the layer.

[0098] For example, the key-value vector of the (n1-1)th generated lexical unit corresponding to each layer of the large model is stored in a computing device or storage device so that the large model can use these key-value vectors in subsequent inference processes. The computing device includes a computing device for running the large model, and the storage device is connected to the computing device.

[0099] The methods described in this application embodiment can be applied to various computing devices used to run large models. These computing devices include, for example, containers, virtual machines, servers, and computing equipment. When the computing device is a server, the server includes, for example, a local server or a cloud server. When the server is a cloud server, it belongs to the infrastructure managed by a cloud management platform, which is used to provide cloud services to tenants, such as... Figure 2 As shown.

[0100] Figure 2 This is a schematic diagram of a cloud service scenario used in this application, such as... Figure 2 As shown, this cloud scenario may include: a cloud management platform 210, the Internet 220, and a client 230. For example... Figure 2As shown, the cloud management platform 210 is used to manage the infrastructure that provides multiple cloud services. The infrastructure includes multiple cloud data centers, each containing multiple servers, and each server containing cloud service resources to provide corresponding cloud services to tenants. For example, the cloud service resources could be cloud databases.

[0101] The cloud management platform 210 can be located in a cloud data center. The cloud management platform 210 provides access interfaces (such as user interfaces or application program interfaces, APIs). Tenants can use client 230 to remotely access the access interface to register a cloud account and password on the cloud management platform 210 and log in. After successful authentication of the cloud account and password, the tenant can further select and purchase specific specifications (such as processors, memory, and disks) of computing devices (e.g., containers, virtual machines, servers, computing equipment, etc.) on the cloud management platform 210. After successful purchase, the cloud management platform 210 provides a remote login account and password for the purchased computing device. The tenant can remotely log in to the computing device through client 230 to install and run their applications on the computing device. Therefore, tenants can create, manage, log in to, and operate computing devices in the cloud data center through the cloud management platform 210. These computing devices can also be referred to as Elastic Compute Service (ECS) or Elastic Instances (different cloud service providers may use different names).

[0102] It should be understood that cloud service tenants can be individuals, businesses, schools, hospitals, government agencies, etc.

[0103] The cloud management platform 210 includes, but is not limited to, a user console, compute management services, network management services, storage management services, authentication services, and image management services. The user console provides an interface or API for interaction with tenants. The compute management services manage servers running virtual machines and containers, as well as bare metal servers. The network management services manage network services (such as gateways and firewalls). The storage management services manage storage services (such as data bucket services). The authentication services manage tenant account passwords. The image management services manage virtual machine images. Tenants use client 230 to log in to the cloud management platform 210 via the internet 220 to manage their rented cloud services.

[0104] Large AI models require a large number of key-value vectors during inference, resulting in significant storage space needed to store them. Furthermore, the storage space required by large models increases linearly with the number of terms in the input inference samples and the model size. Reducing storage space usage by deleting unimportant key-value vectors can easily lead to a decrease in inference accuracy. Therefore, this application provides an optimization method for large AI models that can reduce storage space usage during inference and mitigate the impact on inference accuracy.

[0105] Figure 3 This is a schematic flowchart of the AI ​​large model optimization method provided in the embodiments of this application. Figure 3 The methods described are applied to large AI models that can run on computing devices or clusters of computing equipment, such as... Figure 2 Any server managed by the cloud management platform 210. Figure 3 The method includes the following steps.

[0106] 310, Obtain the initial large model.

[0107] The computing device obtains an initial large model comprising L layers, each layer including at least one attention head. L is a positive integer.

[0108] For example, each of the at least one attention head is used to determine the logical value of a training sample, which is used to determine the output result corresponding to the training sample. The training sample includes at least one initial word, and the output result corresponding to the training sample includes at least one generated word.

[0109] For example, the initial large model is a pre-trained large model. That is, after inputting training samples into the initial large model, the output of the initial large model can be regarded as the expected result. For example, the initial large model is an LLM.

[0110] For example, each attention head is used to calculate the attention output to determine the logical value of the training sample; see the description above for the specific implementation.

[0111] For example, at least two of the L layers have the same or different number of attention heads.

[0112] For example, each attention head in the target layer of the initial large model has the same size. This target layer belongs to the L layers. The size of the attention head is either the size of a single key vector (or value vector) generated by the attention head, or the size of a pair of key-value vectors generated by the attention head, or the total size of all key-value vectors generated by the attention head. The total size of all key-value vectors generated by the attention head = size of each key vector × number of key vectors generated by the attention head + size of each value vector × number of value vectors generated by the attention head. × represents multiplication, and + represents addition. The size of a key vector and the size of a value vector generated by an attention head are the same.

[0113] For example, the size of a vector includes each dimension of the vector, or the size of a vector includes the product of each dimension of the vector, or the size of a vector includes the number of elements in the vector, or the size of the storage space required to store the vector. The size of a pair of key-value vectors includes the sum of the sizes of the key vectors and the value vectors in the key-value vectors. The total size of multiple key-value vectors includes the sum of the sizes of each of the multiple key-value vectors.

[0114] 320. Input N training samples into the initial large model to obtain the initial logical value of each training sample among the N training samples.

[0115] The computing device inputs N training samples into the initial large model to obtain the initial logical value for each training sample. N is a positive integer. The initial logical value for each training sample includes the logical value corresponding to each generated lexical in the output result corresponding to that training sample generated by the initial large model. The implementation method for determining the logical value of the training sample is described above and will not be repeated here.

[0116] For example, any existing methods for determining the logical values ​​of training samples can be applied to the embodiments of this application. The embodiments of this application do not limit the specific implementation of determining the logical values ​​of training samples.

[0117] 330. Based on the initial large model, perform N iterations to optimize the size corresponding to each attention head in the initial large model.

[0118] The computing device iteratively optimizes the size of each attention head in the initial large model N times to obtain a target large model. This target large model is used to perform the inference task. In other words, the target large model is used to obtain the output corresponding to the inference sample based on the inference sample. The inference sample includes at least one initial word. The output corresponding to the inference sample includes at least one generated word.

[0119] For example, H in the target large modelall The sum of the sizes corresponding to each attention point is less than H in the initial large model. all The sum of the dimensions corresponding to each attention head. H all This represents the total number of attention heads in the L layers of the initial large model.

[0120] For example, H in the target large model all The sum of the dimensions corresponding to each attention head is the total target size, which is smaller than H in the initial large model. all The sum of the sizes corresponding to each attention head. Or, the H in the target large model. all The difference between the sum of the sizes of each attention point and the total size of the target is less than or equal to a first preset threshold.

[0121] In some embodiments, during the nth optimization process of N iterations, where n is a positive integer less than or equal to N, the computing device executes steps 331-335. 331. Determine the size corresponding to each attention head in the initial large model as the first size corresponding to each attention head, and obtain the first large model.

[0122] The computing device determines the size corresponding to each attention head in the initial large model as a first size for each attention head, thereby obtaining the first large model. In other words, the computing device adjusts the size corresponding to each attention head in the initial large model so that the size corresponding to each attention head is the first size for that attention head, and uses the adjusted initial large model as the first large model. That is, the number of layers in the initial large model is the same as the number of layers in the first large model, and the number of attention heads in each layer of the initial large model is the same as the number of attention heads in the corresponding layer of the first large model.

[0123] For example, the first size corresponding to each attention head includes an integer.

[0124] In some embodiments, H in the first major model all The sum of the sizes corresponding to each attention point is less than H in the initial large model. all The sum of the sizes corresponding to each attention head.

[0125] In some embodiments, the dimensions of each attention head are as described above.

[0126] For example, when the size corresponding to the attention head is the size of a single key vector (or value vector) generated by the attention head, the first size corresponding to the attention head is the size of that single key vector (or value vector) generated by the attention head. When the size corresponding to the attention head is the size of a pair of key-value vectors generated by the attention head, the first size corresponding to the attention head is the size of that pair of key-value vectors generated by the attention head. When the size corresponding to the attention head is the total size of all key-value vectors generated by the attention head, the first size corresponding to the attention head is the total size of all key-value vectors generated by the attention head.

[0127] In some embodiments, during the first optimization process (i.e., when n=1), the computing device adjusts the size corresponding to each attention head in the initial large model to the same target size to obtain the first large model. This target size includes integers. That is, during the first optimization process, the first size corresponding to each attention head in the first large model is the same, and the first size corresponding to each attention head is the target size.

[0128] For example, the computing device determines the target size based on the target total size. The target total size = target size × H all × indicates multiplication. The total size of the objective is smaller than H in the initial large model. all The sum of the sizes corresponding to each attention head.

[0129] For example, the target total size is a preset value, and the specific value of the target total size is not limited in the embodiments of this application.

[0130] For example, in optimization processes following the first optimization process (i.e., when n is greater than 1), at least two attention heads in each layer of the first large model correspond to the same or different first dimensions. Alternatively, at least two attention heads in at least two layers of the first large model correspond to the same or different first dimensions.

[0131] In some embodiments, the computing device adjusts the initial large model according to a first size corresponding to each attention head and a key-value cache (KV cache) compression algorithm to obtain a first large model. That is, the computing device compresses the number of elements in the key-value vector generated by each attention head using a KV cache compression algorithm so that the size corresponding to each attention head is the first size.

[0132] For example, suppose the key vector generated by the first attention head in the initial large model includes 10 columns of elements, and the size of this attention head is 5. The computing device compresses the initial large model using a KV cache algorithm, so that the key vector generated by the first attention head includes 5 columns of elements. These 5 columns of elements belong to the 10 columns, or are obtained based on the 10 columns. This first attention head belongs to H. all One point of attention.

[0133] For example, the embodiments of this application do not limit the position of the compressed elements in the key vector generated by each attention head, as long as the size corresponding to each attention head is the first size corresponding to each attention head. The embodiments of this application do not limit the specific type of KV cache compression algorithm, such as low-rank approximation algorithm, linear / nonlinear dimensionality reduction algorithm, dynamic sparse coding algorithm, etc. The methods in the prior art that can compress and adjust the size corresponding to each attention head in the initial large model can all be applied to the embodiments of this application.

[0134] In some embodiments, the first large model is obtained by actually pruning each attention head in the initial large model. Alternatively, the pruning effect can be simulated using a mask matrix. The size of this mask matrix is ​​the same as the size of the key vectors generated by the corresponding attention heads in the initial large model. Each element in the mask matrix takes the value 1 or 0.

[0135] For example, the first attention head uses a first mask matrix and an initial key vector to obtain a compressed key vector. The size of the first mask matrix is ​​the same as the size of the initial key vector. The elements in the first mask matrix take values ​​of 0 or 1. When the element in the first row and first column of the first mask matrix is ​​0, the element in the first row and first column of the compressed key vector obtained after multiplying the first mask matrix by the corresponding elements in the initial key vector is 0, thus achieving the pruning effect.

[0136] 332. Input N training samples into the first large model to obtain the first logical value of each training sample among the N training samples.

[0137] The computing device inputs N training samples into the first large model to obtain the first logical value for each training sample. The first logical value for each training sample includes the logical value corresponding to each generated word in the output result corresponding to that training sample generated by the first large model. The implementation method for determining the logical value of the training sample is described in step 320, and will not be repeated here.

[0138] 333. Based on the initial logical value and the first logical value of each training sample in the N training samples, determine the first loss gradient of each training sample.

[0139] The computing device determines the first loss gradient for each training sample based on the initial logic value and the first logic value of each of the N training samples. The first loss gradient for each training sample represents how the difference between the initial logic value and the first logic value of the training sample changes with the size change corresponding to each attention head.

[0140] In some embodiments, the computing device determines a first loss function value for each training sample based on a first loss function, an initial logical value for each training sample, and a first logical value. The first loss function represents the difference between the initial logical value and the first logical value for each training sample when the size corresponding to each attention head is a first size. The computing device determines a first loss gradient for each training sample based on the first loss function value for each training sample. That is, the first loss gradient for each training sample is the gradient corresponding to the first loss function value for each training sample. The method for calculating the function gradient is similar to existing technologies and will not be described in detail here for the sake of brevity.

[0141] For example, the specific form of the first loss function is not limited in the embodiments of this application. For example, the first loss function may include the cross-entropy loss function. Any loss function that can be used in the prior art to represent the difference between the initial logical value and the first logical value of each training sample can be applied to the embodiments of this application.

[0142] 334. Based on the first loss gradient of the nth training sample and the average of the first loss gradients of the N training samples, determine the first gradient projection of the nth training sample.

[0143] The computing device determines the first gradient projection of the nth training sample based on the first loss gradient of the nth training sample out of N training samples and the average of the first loss gradients of the N training samples, where n is a positive integer less than or equal to N. This first gradient projection is used to optimize the size corresponding to each attention head in the first large model, so that H in the optimized large model... all The sum of the sizes corresponding to each attention head equals the target total size, or makes the H in the optimized large model equal to the target total size. all The difference between the sum of the sizes corresponding to each attention point and the total size of the target is less than or equal to a first preset threshold. This application embodiment does not limit the specific value of the first preset threshold.

[0144] In some embodiments, the first gradient projection of the nth training sample is the difference between the first loss gradient of the nth training sample and the average of the first loss gradients of each of the N training samples.

[0145] In some embodiments, it is assumed that S includes H of the large model. all The size corresponding to each attention head in the LLM(x) attention head n S) represents the input training sample x into a large model given S. n The obtained logical value, training sample x n For the training sample set X train The nth training sample in the dataset, where n is a positive integer less than or equal to N, and N is the training sample set X. train The total number of training samples included. Assume Q(LLM(x) n ,S)) represents the large model obtaining LLM(x n The model accuracy at time S) is considered, and a higher model accuracy is better. To reduce the storage space occupied by large models during inference and to minimize the impact on the inference accuracy of large models, the problem of optimizing the size of each attention head in a large model can be expressed as:

[0146]

[0147]

[0148] in, Represents the training sample set X train The sum of the model accuracies obtained after inputting each training sample into the large model. This means that the training sample set X train The value of S is the sum of the model accuracies obtained after each training sample is input into the large model, which is the maximum value. That is, S in formula (1) is... ' To make the training sample set X train S is the sum of the model accuracies obtained after each training sample is input into the large model, which is the maximum value. Formula (1) needs to satisfy the constraint of Formula (2). In Formula (2), Indicates the target size. H represents the large model all The average size corresponding to each attention head. The calculation formula is shown in formula (3). In formula (3), S l,h This represents the size corresponding to the h-th attention head in the l-th layer. l h is a positive integer less than or equal to L, and h is less than or equal to H. l H is a positive integer.l Represents the Lth layer l The total number of attention heads in the layer.

[0149] In some embodiments, the training sample set X train The number of tokens is the same in each training sample.

[0150] In some embodiments, since the optimization problem represented by Equation (1) is difficult to solve, and the optimization problem represented by Equation (1) can be transformed into the optimization problem represented by Equation (4), the optimal size corresponding to each attention head in the large model can be determined by solving the optimization problem represented by Equation (4). Equation (4) is shown below:

[0151]

[0152] Among them, t n Indicates training sample x n The corresponding expected output data, LLM(x) n S) represents the training sample x n The corresponding actual output data. For example, t n Including training samples x n The initial logic values ​​obtained after inputting the initial large model, LLM(x) n S) includes training samples x given S. n The first logical value S obtained after inputting the first large model includes H of the first large model. all The size corresponding to each attention head in the attention heads. loss() represents the first loss function, L n (S) indicates that according to t n and LLM(x n The first loss function value is determined by S). This means that the training sample set X train The value of S when the sum of the first loss function values ​​corresponding to each training sample in the formula reaches its minimum. That is, S in formula (4) is... ' To make the training sample set X train S is the sum of the first loss function values ​​corresponding to each training sample in the dataset that is at its minimum.

[0153] In some embodiments, the computing device can solve formula (4) using various adaptive gradient algorithms. The specific type of adaptive gradient algorithm is not limited in the embodiments of this application, such as adaptive gradient (Adagrad), root mean square propagation (RMSProp), adaptive moment estimation (Adam), etc.

[0154] In some embodiments, when the computing device solves formula (4) using the adaptive gradient algorithm, it needs to use S from the nth iteration optimization process to obtain L in each nth iteration optimization process. n (S). Since reducing the size of the attention head in a large model usually leads to a decrease in the inference accuracy of the large model, resulting in an increase in the value of the first loss function, the loss gradient function corresponding to the first loss function will tend to increase S. In order to make The gradient corresponding to the training sample can be projected into a constant-size budget subspace using formula (6), thereby avoiding the unlimited increase of the loss gradient function S.

[0155]

[0156] in, This indicates that, given S, the training sample x... n The corresponding loss gradient, This indicates that, given a sample set S, the training sample set X is... train The average loss gradient corresponding to each training sample in S. That is, Equation (6) is used to represent the average loss gradient of training sample x given S. n The corresponding gradient projection △S n For training sample x n The corresponding loss gradient and the training sample set X train The difference between the average values ​​of the loss gradients corresponding to each training sample.

[0157] For example, a diagram illustrating the projection of the gradients corresponding to training samples onto a constant-size budget subspace is shown below. Figure 4 As shown. In Figure 4 In the diagram, the vertical axis represents the size value corresponding to the first attention head of the first layer in the large model, and the horizontal axis represents the size value corresponding to the second attention head of the first layer in the large model. Figure 4 The slashes in the diagram represent the total size of the target, which is the sum of the size corresponding to the first attention head in the first layer and the size corresponding to the second attention head in the first layer. Figure 4The diagonal line in the diagram includes two points, S1 and S2. The x-coordinate of S1 represents the size of the second attention head in the first layer given S1, and the y-coordinate of S1 represents the size of the first attention head in the first layer given S1. Figure 4 The line segment pointing from S1 to the upper right in the diagram represents the training sample x given S1. n The opposite direction of the corresponding loss gradient and the magnitude of the loss gradient. Since the loss gradient function tends to increase the size of each attention head during optimization iterations, the training sample set X... train The direction of the average value of the loss gradient for each training sample will be perpendicular to the direction of the average value of the loss gradient. Figure 4 The diagonal lines in the diagram are used to ensure that, with a fixed total budget size, the size of each attention head can reach its maximum value. Given S1, based on the training samples x... n The corresponding loss gradient and training sample set X train The difference between the average loss gradients corresponding to each training sample in the dataset can be used to obtain the training sample x. n The corresponding gradient projection △S n . Figure 4 In the diagram, the line segment pointing from S1 to S2 represents the training sample x given S1. n The corresponding gradient projection △S n The opposite direction of the direction and the gradient projection △S n The size of the training sample x given S1. n The corresponding gradient projection △S n The optimized S2 is determined.

[0158] In some embodiments, training sample x n The corresponding gradient projection △S n The influence of each attention head on the loss function in a large model can be preserved, thus identifying the attention heads with the greatest impact on the loss function within the large model. For example, after multiple iterations of optimization, if the size corresponding to an attention head is larger than the sizes corresponding to other attention heads, it indicates that the attention head has a significant impact on the loss function, i.e., it has a significant impact on inference accuracy. Alternatively, if the size corresponding to an attention head is greater than the average size of all attention heads, it also indicates that the attention head has a significant impact on the loss function, i.e., it has a significant impact on inference accuracy. Optimizing the size corresponding to the attention heads by gradient projection of each training sample can achieve a balance between the size of each attention head and the number of attention heads during the optimization process.

[0159] 335. Based on the first gradient projection of the nth training sample, optimize the size corresponding to each attention head in the first large model to obtain the second size corresponding to each attention head.

[0160] The computing device optimizes the size corresponding to each attention head in the first large model based on the first gradient projection of the nth training sample to determine the second size corresponding to each attention head. The second size corresponding to each attention head is used to obtain the second large model. The sum of the second sizes corresponding to each attention head is less than the sum of the sizes corresponding to each attention head in the initial large model.

[0161] In some embodiments, when n is less than N, the computing device uses the second largest model as the first largest model in the (n+1)th optimization process of N iterations, where n+1 is a positive integer less than or equal to N. In other words, when n is less than N, the second size corresponding to each attention head obtained after the nth optimization is used as the first size corresponding to each attention head in the (n+1)th optimization process, thereby obtaining the first largest model in the (n+1)th optimization process.

[0162] For example, when n is less than N, in the nth optimization process, after the computing device obtains the second size corresponding to each attention head, it does not need to actually obtain the second largest model. Instead, in the (n+1)th optimization process, it uses the second size corresponding to each attention head as the first size, thereby obtaining the first largest model in the (n+1)th optimization process. In this case, the second largest model mentioned in the nth optimization process refers to the first largest model in the (n+1)th optimization process.

[0163] In some embodiments, when n=N, the computing device uses the second largest model as the target large model. In other words, after completing N iterations of optimization, the computing device obtains the target large model based on the second size corresponding to each attention head obtained after the Nth optimization.

[0164] For example, after N iterations of optimization, the computing device determines the optimal size for each attention head based on the second size corresponding to each attention head after the Nth optimization. The computing device adjusts the size corresponding to each attention head in the initial large model to the optimal size corresponding to each attention head to obtain the target large model. That is, the number of layers in the initial large model is the same as the number of layers in the target large model, and the number of attention heads in each layer of the initial large model is the same as the number of attention heads in the corresponding layer of the target large model. The implementation of adjusting the size of the initial large model to obtain the target large model is similar to the implementation of obtaining the first large model, and will not be described again here.

[0165] In some embodiments, the second size corresponding to each attention head after the Nth optimization is the optimal size corresponding to each attention head. Alternatively, the second size corresponding to each attention head after the Nth optimization is adjusted to obtain the optimal size corresponding to each attention head.

[0166] In some embodiments, the computing device optimizes the first size corresponding to each attention head based on the first gradient projection of the nth training sample, thereby determining a first optimized size for each attention head. The computing device then determines a second size for each attention head based on the first optimized size, the average size, and the target size. The average size for each attention head is determined based on historical values ​​of the size corresponding to each attention head.

[0167] For example, the computing device determines a first optimized size corresponding to each attention head based on various adaptive gradient algorithms. This application embodiment does not limit the specific adaptive gradient algorithm.

[0168] For example, the computing device will H all The first size corresponding to each attention head and the first gradient projection of the nth training sample are used as inputs to the Adam algorithm, thereby obtaining the output of the Adam algorithm, as shown in Equation (7).

[0169]

[0170] Among them, S n Including H all The first dimension corresponding to each attention head in the attention heads, △S n S represents the first gradient projection of the nth training sample. n+1 Including H all The first optimization size corresponding to each attention head in the *n* attention heads, ADAM() represents the Adam algorithm. That is, the first size corresponding to each attention head is regarded as the value of the parameter to be optimized, and the first gradient projection of the nth training sample is regarded as the gradient used to optimize the parameter.

[0171] For example, in accordance with ADAM(S) n , △S n When the first optimized size corresponding to the partial attention head is a decimal (i.e., not an integer), the computing device rounds the first optimized size corresponding to that partial attention head to make S n+1 The first optimized size corresponding to each attention head is an integer. This application does not limit the specific rounding method used in its embodiments; for example, it may include rounding to the nearest integer, rounding up, rounding down, etc.

[0172] For example, the computing device determines the average size corresponding to each attention head based on the historical values ​​of the size corresponding to each attention head. The historical values ​​of the size corresponding to each attention head include the size corresponding to that attention head determined after each optimization in the historical optimization process.

[0173] For example, when n=1, the average size corresponding to the first attention head is the first optimized size corresponding to the first attention head determined in the first optimization process. This first attention head belongs to the initial large model. When n=2, the average size corresponding to the first attention head includes the average of the second size corresponding to the first attention head determined in the first optimization process and the first optimized size corresponding to the first attention head determined in the second optimization process. Similarly, in the nth optimization, the average size corresponding to the first attention head includes the average of the following values: the second size corresponding to the first attention head determined in each optimization process from the first to the (n-1)th optimization process, and the first optimized size corresponding to the first attention head determined in the nth optimization process.

[0174] For example, the second size corresponding to each attention head is determined based on the difference between the first optimized size corresponding to each attention head and the first value corresponding to each attention head, and the first value corresponding to each attention head is determined based on the difference between the average size corresponding to each attention head and the target size.

[0175] For example, the second size corresponding to each attention head is the difference between the first optimized size corresponding to each attention head and the first value corresponding to each attention head, or it is a value obtained by rounding down the difference. The first value corresponding to each attention head is the difference between the average size corresponding to each attention head and the target size, or it is a value obtained by rounding down the difference.

[0176] For example, H all The calculation method for the second dimension corresponding to each attention head in the attention heads is shown in Equation (8).

[0177]

[0178] In equation (8), S on the left side of the arrow n+1 Including H all The second size corresponding to each attention head in the attention heads, S to the right of the arrow. n+1 Including H all The first optimized size corresponds to each attention head in the attention heads. Including H all The average size corresponding to each attention head in the attention heads. Including H allThe target size corresponds to each of the attention heads. Each attention head has the same target size.

[0179] For example, in H all When the average size corresponding to each attention head in the attention heads is a decimal (i.e., not an integer), the computing device rounds the average size corresponding to each attention head to the nearest integer, so that... The average size corresponding to each attention head is an integer. This application does not limit the specific rounding method used in its embodiments; for example, it may include rounding to the nearest integer, rounding up, rounding down, etc.

[0180] For example, in accordance with If the second dimension corresponding to the obtained partial attention head is a decimal (i.e., not an integer), the computing device rounds the second dimension corresponding to that partial attention head so that S on the left side of the arrow in equation (8) is... n+1 The second size corresponding to each attention head is an integer. This application does not limit the specific rounding method used in its embodiments; for example, it may include rounding to the nearest integer, rounding up, rounding down, etc.

[0181] In some embodiments, when using the adaptive gradient algorithm to determine the first optimized size corresponding to each attention head, the average value of the first optimized size corresponding to each attention head changes because the adaptive algorithm iterates and optimizes in directions that are not completely collinear with the provided gradient direction. This results in the sum of the sizes corresponding to each attention head in the optimized large model not being equal to the target total size. In order to make the sum of the sizes corresponding to each attention head in the optimized large model equal to the target total size, the optimized size of each attention head is re-normalized using the above equation (8) to obtain the second size corresponding to each attention head.

[0182] In some embodiments, after N iterations of optimization, the computing device obtains H. all The average value of the second size corresponding to each attention head, H all The second size corresponding to each attention head is obtained after the Nth optimization. When the average value of the second size is greater than the target size, the computing device reduces H. all The second size corresponding to one or more attention heads in a set of attention heads is such that, after reducing the second size corresponding to some attention heads, H all The average size of the dimensions corresponding to each of the attention heads is equal to the target size. If the average size of the second dimension is less than the target size, the computing device increases H. all The second size corresponding to one or more attention heads in a set of attention heads is such that, after increasing the second size corresponding to some attention heads, H allThe average size of the size corresponding to each attention head in the attention heads is equal to the target size.

[0183] For example, after adjusting the second size corresponding to the partial attention head obtained after the Nth optimization, in H all When the average size of the size corresponding to each attention head in the attention heads is equal to the target size, the H all The size of each attention head in the attention heads is the optimal size for each attention head.

[0184] In some embodiments, at least two attention heads in one layer of the target large model correspond to the same or different sizes. Alternatively, at least two attention heads in multiple layers of the target large model correspond to the same or different sizes. That is, different attention heads in the same layer of the target large model may have different compression rates (CR), or attention heads in different layers of the target large model may have different compression rates (CR).

[0185] In some embodiments, the larger the size of the attention head in the target large model, the greater its impact on inference accuracy. Conversely, the smaller the size of the attention head in the target large model, the smaller its impact on inference accuracy.

[0186] exist Figure 3 In this method, the computing device performs N iterations to optimize the size corresponding to each attention head in the large AI model. During the nth optimization, the computing device uses the first loss gradient of the nth training sample and the average of the first loss gradients of the N training samples to determine the first gradient projection. This ensures that when optimizing the size corresponding to each attention head based on the first gradient projection, the sum of the optimized sizes of each attention head is less than the sum of the sizes of each attention head in the initial large model, thereby reducing the storage space occupied by the large AI model during inference. Furthermore, since the first gradient projection is determined based on the initial logical values ​​and first logical values ​​of the training samples, it can reduce the impact on the inference accuracy of the large AI model obtained after N iterations of optimization.

[0187] Optionally, Figure 3 One possible implementation of the method in [the document] is as follows: Figure 5 As shown. Figure 5 This is a schematic flowchart of the AI ​​large model optimization method provided in the embodiments of this application. Figure 5 The methods described are applied to large AI models that can run on computing devices or clusters of computing equipment, such as... Figure 2 Any server managed by the cloud management platform 210. Figure 5The method includes the following steps.

[0188] 501, obtain the initial logic values ​​for the initial large model and N training samples.

[0189] The computing device obtains an initial large model, as described in step 310. The computing device inputs N training samples into the initial large model to obtain the initial logical value for each training sample, as described in step 320 for the specific implementation.

[0190] 502, determine the nth training sample from N training samples, and optimize the size corresponding to each attention head for the nth time.

[0191] The computing device determines the nth training sample from N training samples, where n is a positive integer less than or equal to N. See [reference needed] for the N training samples. Figure 3 The description in the document states that after the computing device determines the nth training sample, it begins the nth optimization process, which involves optimizing the size corresponding to each attention head for the nth time.

[0192] 503. Based on the first size corresponding to each attention head in the nth optimization process, the first largest model in the nth optimization process is obtained.

[0193] When n=1, the first size corresponding to each attention head in the nth optimization process is the target size. That is, in the first optimization process, the computing device determines the size corresponding to each attention head as the target size to obtain the first largest model in the first optimization process. For the method of determining the target size and obtaining the first largest model, please refer to [link to relevant documentation]. Figure 3 The description in the text will not be repeated here.

[0194] When n is greater than 1, the first size corresponding to each attention head in the nth optimization process is the second size corresponding to each attention head in the (n-1)th optimization process. The method for obtaining the second size corresponding to each attention head is described in the following steps. That is, in the nth optimization process, the computing device determines the size corresponding to each attention head as the second size corresponding to each attention head in the (n-1)th optimization process to obtain the first largest model in the nth optimization process. The method for obtaining this first largest model is described in [link to documentation]. Figure 3 The description in the text will not be repeated here.

[0195] 504. Based on the first largest model in the nth optimization process, determine the first logical value of each of the N training samples.

[0196] In the nth optimization process, the computing device inputs N training samples into the first large model of the nth optimization process to obtain the first logical value of each training sample. See step 332 for the specific implementation.

[0197] 505. Based on the initial logical value of each training sample and the first logical value of each training sample in the nth optimization process, determine the first loss gradient of each training sample in the nth optimization process.

[0198] In the nth optimization process, the computing device determines the first loss function value for the nth optimization process based on the initial logical value of each training sample and the first logical value in the nth optimization process. The computing device then determines the first loss gradient for the nth optimization process based on the first loss function value. The method for determining the first loss gradient is similar to step 333 and will not be repeated here.

[0199] 506. Based on the first loss gradient of the N training samples in the nth optimization process, determine the first gradient projection of the nth training sample in the nth optimization process.

[0200] The computing device determines the average of the first loss gradients of the N training samples in the nth optimization process based on the first loss gradient of each training sample in the N training samples in the nth optimization process. The computing device then determines the first gradient projection of the nth training sample in the nth optimization process based on the first loss gradient of the nth training sample and the average of the first loss gradients of the N training samples. For a detailed implementation, please refer to step 334, which will not be elaborated here.

[0201] For example, the computing device obtains the first gradient projection in the nth optimization process according to the above equation (6).

[0202] 507. Based on the first gradient projection of the nth training sample in the nth optimization process, determine the first optimization size corresponding to each attention head in the nth optimization process.

[0203] The computing device, based on the adaptive gradient algorithm, uses the first size corresponding to each attention head in the nth optimization process as the value of the parameter to be optimized, and uses the projection of the first gradient in the nth optimization process as the gradient for optimizing the parameter, to obtain the output of the adaptive gradient algorithm. This output includes the first optimized size corresponding to each attention head in the nth optimization process. For a detailed implementation, please refer to the description in step 335, which will not be repeated here.

[0204] For example, the computing device obtains the first optimization size corresponding to each attention head in the nth optimization process according to the above equation (7).

[0205] 508. Based on the first optimized size, average size, and target size corresponding to each attention head in the nth optimization process, determine the second size corresponding to each attention head in the nth optimization process. For the specific implementation, please refer to the description in step 335, which will not be repeated here.

[0206] For example, the computing device obtains the second size corresponding to each attention head in the nth optimization process according to the above equation (8).

[0207] 509. Determine if n is equal to N.

[0208] If n is less than N, the computing device executes step 510. If n is equal to N, the computing device executes step 511.

[0209] 510, Update n.

[0210] When n is less than N, the computing device updates the value of n so that the updated n is n+1 of the original value. After updating n, the computing device repeats steps 502-509 above using the updated n.

[0211] 511. Based on the second size corresponding to each attention head during the Nth optimization process, the target large model is obtained.

[0212] When n equals N, the computing device determines the optimal size for each attention head based on the second size corresponding to each attention head during the Nth optimization process. The sum of the optimal sizes for each attention head is the target total size. The computing device adjusts the size corresponding to each attention head in the initial large model to the optimal size corresponding to each attention head to obtain the target large model. The specific implementation is similar to that of obtaining the first large model, and will not be described in detail here.

[0213] When n equals N, the computing device determines the average value of the second size corresponding to each attention head during the Nth optimization process. If the average value of the second size corresponding to each attention head during the Nth optimization process is greater than the target size, the computing device reduces the second size corresponding to one or more attention heads such that, after reducing the second size corresponding to some attention heads, the average value of the size corresponding to each attention head equals the target size. If the average value of the second size corresponding to each attention head during the Nth optimization process is less than the target size, the computing device increases the second size corresponding to one or more attention heads such that, after increasing the second size corresponding to some attention heads, the average value of the size corresponding to each attention head equals the target size.

[0214] For example, after adjusting the second size corresponding to some attention heads obtained after the Nth optimization, when the average size corresponding to each attention head is equal to the target size, the size corresponding to each attention head is the optimal size corresponding to each attention head.

[0215] In some embodiments, when the second size corresponding to some attention heads is a decimal (i.e. not an integer) during the Nth optimization process, the computing device rounds the second size corresponding to each attention head to obtain the rounded second size corresponding to each attention head, as shown in equation (9):

[0216] in, This represents the second size corresponding to the h-th attention head in the l-th layer during the N-th optimization process. round() represents the rounding function. This represents the rounded second size corresponding to the h-th attention head of the l-th layer during the N-th optimization process. The embodiments of this application do not limit the specific rounding method, but may include rounding to the nearest integer, rounding up, rounding down, etc.

[0217] In some embodiments, the computing device uses the rounded second size corresponding to each attention head. The second size corresponding to some attention heads is adjusted (e.g., decreased or increased) so that the average value of the adjusted size corresponding to each attention head is equal to the target size.

[0218] For example, the computing device determines the second size based on the rounded value corresponding to each attention head. Determine the average value of the second size corresponding to each attention head before adjusting the second size. .exist When the target size is equal to the target size, the computing device takes the rounded second size corresponding to each attention head as the optimal size corresponding to each attention head, thereby obtaining the target model.

[0219] For example, in When the size is larger than the target size, the computing device uses the second size before rounding corresponding to each attention head. The second dimension after rounding The difference is used to determine the second value corresponding to each attention head, and this second value is... | represents absolute value. The computing device determines the smallest second value among the second values ​​corresponding to each attention head, and adjusts the second size corresponding to the attention head corresponding to the smallest second value. The adjusted second size = the original second size - 1. Based on the adjusted second size of the attention head, the computing device determines the average value of the second size corresponding to each attention head after the first adjustment. .exist When the size is larger than the target size, the computing device updates the second value corresponding to the attention head based on the adjusted second size corresponding to the attention head that has been adjusted to the second size. This second value is... ,Should The adjusted second size corresponds to the attention head. Based on the updated second value of the attention head, the computing device determines the smallest second value from the second values ​​corresponding to each attention head and repeats the above steps until the average size of the dimensions corresponding to each attention head equals the target size. When the target size is equal, the computing device will The second size corresponding to each attention head is taken as the optimal size for each attention head, thereby obtaining the target model.

[0220] For example, in When the size is smaller than the target size, the computing device uses the second size before rounding corresponding to each attention head. The second dimension after rounding The difference is used to determine the second value corresponding to each attention head, and this second value is... | represents absolute value. The computing device determines the largest second value among the second values ​​corresponding to each attention head, and adjusts the second size corresponding to the attention head corresponding to the largest second value. The adjusted second size = the original second size + 1. Based on the adjusted second size of the attention head, the computing device determines the average value of the second size corresponding to each attention head after the first adjustment. .exist When the size is smaller than the target size, the computing device updates the second value corresponding to the attention head based on the adjusted second size corresponding to the attention head that has been adjusted to the second size. This second value is... ,Should This is the adjusted second size corresponding to the attention head. The computing device, based on the updated second value of the attention head, determines the largest second value from the second values ​​corresponding to each attention head and repeats the above steps until the average size of the dimensions corresponding to each attention head equals the target size. When the target size is equal, the computing device will The second size corresponding to each attention head is taken as the optimal size for each attention head, thereby obtaining the target model.

[0221] For example, the embodiments of this application do not limit the method of determining the attention head that needs to be reduced or increased in the second size. In addition to the above method, it is also possible to select one or more second values ​​that are larger or smaller from the second values ​​corresponding to each attention head, and use the attention head corresponding to the one or more second values ​​as the method of the attention head that needs to be reduced or increased in the second size, so that the sum of the sizes corresponding to each attention head after adjustment is the target total size.

[0222] For example, "the sum of the dimensions corresponding to each attention head is the target total size" and "the average of the dimensions corresponding to each attention head is the target size" have similar meanings and can be used interchangeably.

[0223] exist Figure 5 In this method, when optimizing the size corresponding to the attention head for the nth time, the computing device uses the first loss gradient of the nth training sample and the average of the first loss gradients of the N training samples to determine the first gradient projection. This ensures that when optimizing the size corresponding to each attention head based on the first gradient projection, the sum of the optimized sizes of each attention head is less than the sum of the sizes of each attention head in the initial large model, thereby reducing the storage space occupied by the AI ​​large model during inference. Furthermore, since the first gradient projection is determined based on the initial logical values ​​and first logical values ​​of the training samples, it can reduce the impact on the inference accuracy of the AI ​​large model obtained after N iterations of optimization.

[0224] Optionally, Figure 3 or Figure 5 The method described can be used to compress the size of the attention heads in each layer of the initial large model, or it can be used to compress the size of the attention heads in only some layers of the initial large model. When compressing the size of the attention heads in only some layers of the initial large model, H... all This represents the number of attention heads in this layer. The specific implementation is similar to compressing the size of the attention heads in each layer, and will not be elaborated here.

[0225] Optionally, according to Figure 3 or Figure 5The method described above obtains the optimal size for each attention head, resulting in minimal loss of inference accuracy for large models, but not necessarily faster inference speed. When multiple attention heads correspond to the same size, a single call to the attention kernel (e.g., a flash attention kernel) can process these attention heads in parallel. However, when multiple attention heads correspond to different sizes, different attention kernels are required for computation, increasing the performance overhead of kernel calls and impacting inference speed. Therefore, in Figure 3 or Figure 5 In this method, after obtaining the optimal size corresponding to each attention head, the computing device can execute... Figure 6 The method described in the article obtains a target large model, thereby improving the inference speed of the target large model.

[0226] Figure 6 This is a schematic flowchart of the method for adjusting a large model provided in the embodiments of this application. Figure 6 The methods described are applied to large models that can run on computing devices or clusters of computing equipment, such as... Figure 2 Any server managed by the cloud management platform 210. Figure 6 The method includes the following steps.

[0227] 601. Based on the second dimension corresponding to each attention head of the target layer, determine the M third dimensions and the M attention head sets corresponding to the target layer.

[0228] In some embodiments, prior to step 601, the computing device determines a second size corresponding to each attention head of the target layer. The second size corresponding to each attention head of the target layer is... Figure 3 or Figure 5 The second size corresponding to each attention head of the target layer is obtained after the Nth optimization.

[0229] In some embodiments, "the second size corresponding to each attention head" in step 601 can be replaced by according to Figure 3 or Figure 5 The optimal size for each attention head is obtained using the method described in [the document / reference].

[0230] In some embodiments, the target layer is one of the L layers of the initial large model, where L is a positive integer. This target layer includes H attention heads, where H is a positive integer. See the initial large model for details. Figure 3 or Figure 5 The description in the text.

[0231] In some embodiments, the computing device determines M third dimensions and M sets of attention heads corresponding to the target layer based on the second dimension corresponding to each attention head of the target layer. The M third dimensions belong to a set of dimensions that includes the second dimension corresponding to each attention head of the target layer, where M is a positive integer less than the number of attention heads in the target layer. That is, the M third dimensions are M dimensions selected from the second dimensions corresponding to each attention head of the target layer. Each of the M sets of attention heads includes at least one attention head of the target layer. The m-th third dimension among the M third dimensions corresponds to the m-th attention head set among the M sets of attention heads, where m is a positive integer less than or equal to M.

[0232] For example, M is a preset value, and the specific value of M is not limited in the embodiments of this application.

[0233] In some embodiments, the M third dimensions and the M attention head sets minimize the difference between the first total size and the second total size. The first total size includes the sum of the dimensions corresponding to each attention head when the size corresponding to each attention head in the target layer is the third dimension corresponding to the attention head set to which each attention head belongs. The second total size includes the sum of the dimensions corresponding to each attention head when the size corresponding to each attention head in the target layer is the second dimension corresponding to each attention head after the Nth optimization.

[0234] For example, assuming M=2, the computing device uses the following formulas (10) to (12) to determine the M third dimensions and the M attention head sets.

[0235]

[0236]

[0237]

[0238] In equation (10), P l,h Used to indicate the first l The h-th attention head of the layer is in the first position. l The set of attention heads to which the layer belongs. l h is a positive integer less than or equal to L, and h is less than or equal to H. l H is a positive integer. l For the first l The number of attention heads in the layer. l The layer is the target layer. In P... l,h When = 0, the first l The h-th attention head of layer belongs to the first layer. l The first set of attention heads in layer P. l,hWhen = 1, the h-th attention head of the l-th layer belongs to the l-th layer. l The second set of attention heads in the layer.

[0239] In equation (11), the computing device executes only on the target layer. Figure 6 When using the method in the middle, P includes the first l P corresponding to each attention head of the layer l,h G1 includes the first l A third dimension of the layer, G2 includes the first... l Another third dimension of the layer, E(G1), is used to represent the first... l The sum of the third dimensions corresponding to each attention head in the first attention head set of the layer, E(G2), is used to represent the third dimension. l The sum of the third dimensions corresponding to each attention head in the second set of attention heads in the layer. Indicates the first l The sum of the third dimensions corresponding to each attention head in the second set of attention heads in the layer. Represents the dot product. Indicates the first l The sum of the third dimensions corresponding to each attention head in the first attention head set in the layer, S g (G1, G2, P) represents the nth... l The sum of the dimensions corresponding to each attention head in the layer.

[0240] In equation (11), the computing device performs the operation on each layer of the large model. Figure 6 In the method described, P includes P corresponding to each attention head of each of the L layers. l,h G1 includes a third dimension for each of the L layers, G2 includes another third dimension for each of the L layers, E(G1) represents the sum of the third dimensions corresponding to each attention head in the first attention head set of each of the L layers, and E(G2) represents the sum of the third dimensions corresponding to each attention head in the second attention head set of each of the L layers. This represents the sum of the third dimensions corresponding to each attention head in the second attention head set of each of the L layers. Represents the dot product. S represents the sum of the third dimensions corresponding to each attention head in the first attention head set of each of the L layers. g (G1,G2,P) represents the sum of the sizes of each attention head in each of the L layers, given G1,G2,P.

[0241] In equation (12), the computing device executes only on the target layer. Figure 6 When using the method in S' Including the l The sum of the second dimensions corresponding to each attention head of the layer. This is performed on each layer of the large model by the computing device. Figure 6 When using the method in S ' This includes the sum of the second dimensions corresponding to each attention head in each of the L layers. For S ' With S g The square of the absolute value of the difference between (G1,G2,P). Indicates that The values ​​of G1, G2, and P when the minimum value is reached. That is, G1 in formula (12) ' G2 ' ,P ' In order to make G1, G2, P when the minimum value is obtained. The computing device is based on G1. ' G2 ' ,P ' Determine the first l The layer has M third dimensions and M attention head sets, or according to G1 ' G2 ' ,P ' Determine the M third dimensions and M attention head sets for each of the L layers.

[0242] For example, the value of M corresponding to each of the L layers may be the same or different. At least two of the M attention head sets may have the same or different number of attention heads.

[0243] 602. The size corresponding to each attention head in the target layer is determined as the third size corresponding to the set of attention heads to which each attention head belongs, thus obtaining the target large model.

[0244] In some embodiments, if the set of attention heads to which the first attention head in the target layer belongs is the m-th set of attention heads in the target layer, the computing device determines the size corresponding to the first attention head as the m-th third size of the target layer, thereby obtaining the target large model. The specific implementation is similar to the implementation of obtaining the first large model in step 331, and will not be described again here.

[0245] exist Figure 6 In this method, since multiple attention heads in the target layer of the target large model belong to M attention head sets, and the third size corresponding to the attention heads in each attention head set is the same, the computing device only needs to call the attention kernel M times in the target layer when using the target large model for inference, thereby improving the inference speed.

[0246] Optionally, when using the target large model for inference, the inference accuracy of the target large model is still lower than that of the initial large model because the size of some attention heads is compressed. To reduce the loss of inference accuracy, compensation modules can be added to some or all attention heads of the target model to make the inference process of the target large model as... Figure 7 The method is shown in [the document]. See the compensation module for details. Figure 7 The target large model is the target large model obtained according to the method in the embodiments of this application. Alternatively, the target large model is a compressed large model obtained using a KV cache compression method other than the method provided in the embodiments of this application. In other words, Figure 7 The method in can be compared with Figures 3 to 6 The methods mentioned can be used in combination or individually.

[0247] Figure 7 This is a schematic flowchart of the reasoning method for the target large model provided in the embodiments of this application. Figure 7 The methods described are applied to large models that can run on computing devices or clusters of computing equipment, such as... Figure 2 Any server managed by the cloud management platform 210. Figure 7 The method includes the following steps.

[0248] 701. Input the inference sample into the target large model to obtain the first input data of the first attention head of the target large model.

[0249] In some embodiments, the computing device inputs an inference sample into a target large model, the inference sample including at least one initial lexical unit. The target large model performs pre-filling processing based on the at least one initial lexical unit in the inference sample, obtaining a key-value vector for each initial lexical unit and a first generated lexical unit. The specific implementation of the pre-filling processing is described above. The target large model performs decoding processing based on the key-value vector of the at least one initial lexical unit and the first generated lexical unit, obtaining a key-value vector for the first generated lexical unit and a second generated lexical unit. The target large model performs decoding processing based on the key-value vector of the at least one initial lexical unit, the key-value vector of the first generated lexical unit, and the second generated lexical unit, obtaining a key-value vector for the second generated lexical unit and a third generated lexical unit. The target large model repeats the above process until the generated lexical unit is a preset lexical unit. This preset lexical unit is, for example, an end-of-sentence (EOS) lexical unit. The specific implementation of the decoding processing is described above.

[0250] In some embodiments, the target large model includes L layers, and the first layer in the L layers... l Each layer includes H l A point of attention. L, H lIt is a positive integer. l H is a positive integer less than or equal to L. That is, the target model includes H. all One point of attention, The first point of attention belongs to H. all One point of attention.

[0251] In some embodiments, during the decoding process, the target large model determines the input data of each attention head in each layer of the target large model based on a generated word input to the decoding process, that is, determines the first input data of the first attention head.

[0252] 702. Based on the first input data, obtain the first attention output.

[0253] After determining the first input data of the first attention head, the first attention head determines the query vector, key vector, and value vector corresponding to the first input data. The first attention head then uses the query vector and the first set of key-value vectors corresponding to the first input data to determine its first attention output. This first set of key-value vectors includes at least one key-value vector generated by the first attention head before processing the first input data, as well as the key vector and value vector corresponding to the first input data; that is, the first set of key-value vectors includes at least one key-value vector generated by the first attention head. See the description above for the specific calculation method.

[0254] 703. Obtain the first compensation data based on the first input data.

[0255] In some embodiments, the first attention head further includes a compensation module, which is used to determine first compensation data based on a first set of value vectors. The first set of value vectors includes at least one value vector generated by the first attention head. That is, the first set of value vectors includes at least one value vector generated by the first attention head before processing the first input data and a value vector corresponding to the first input data.

[0256] For example, the compensation module includes a state space model (SSM).

[0257] In some embodiments, the compensation module determines the second state value corresponding to the first compensation data and the first input data based on the first state value corresponding to the first input data and the first value vector set. The first state value corresponding to the first input data is the second state value corresponding to the second input data obtained when the first attention head processes the second input data, and the second input data is the input data processed by the first attention head before processing the first input data.

[0258] For example, during the pre-filling process of the target large model, the compensation module does not need to determine the compensation data corresponding to the input data, but only needs to determine the second state value corresponding to the input data. During the decoding process of the target large model, the compensation module determines the compensation data of the input data and the corresponding second state value.

[0259] For example, during pre-filling processing of the target large model, the compensation module determines the first state value corresponding to the third input data as a preset vector (e.g., a zero vector). This third input data is the input data of the first attention head when the target large model processes at least one initial word in the inference sample. The compensation module determines the second state value corresponding to the third input data according to equation (13):

[0260] Where A represents the trained state transition matrix, and B represents the trained input projection matrix. SS i-1 This represents the first state value corresponding to the current input data (e.g., the third input data), which is also the second state value corresponding to the previous input data. V t This represents the value vector corresponding to the third input data. SS i represents the second state value corresponding to the current input data, which is also the first state value corresponding to the next input data. · represents the dot product. The compensation module uses the second state value corresponding to the third input data as the first state value of the fourth input data, which is the input data of the first attention head when the target large model performs decoding processing based on the first generated word. The compensation module determines the second state value of the fourth input data based on the first state value of the fourth input data and the above formula (13).

[0261] For example, suppose that the input data of the first attention head is the first input data when the target large model decodes based on the first generated word. The compensation module determines the second state value corresponding to the first input data based on the first state value corresponding to the first input data and the above equation (13). The compensation module also determines the first compensation data based on the first value vector set and equation (14).

[0262]

[0263] Where C represents the trained output projection matrix, and D represents the trained skip connection matrix. SS i This represents the second state value corresponding to the first input data. V i R represents a first set of value vectors, which includes the value vector corresponding to each initial word in at least one initial word in the inference sample obtained when the attention head processes the third input data, and the value vector corresponding to the first input data obtained when the attention head processes the first input data. iThis indicates the first compensation data.

[0264] For example, all elements in D are 0 or at least one element is not 0.

[0265] In some embodiments, a structural diagram of the compensation module is shown below. Figure 8 As shown. Figure 8 The supplementary module 800 includes a fully connected (FC) layer 810, an SSM layer 820, and an FC layer 830. The FC layer 810 includes a linear projection layer 811 and a nonlinear activation layer 812. The linear projection layer 811 performs a linear transformation on the value vectors in the first value vector set, reducing the size of the value vectors in the first value vector set from 1×d to 1×d / 2, where d is a positive integer. The nonlinear activation layer 812 performs a nonlinear transformation on the dimensionality-reduced value vectors in the first value vector set to obtain a nonlinearly transformed value vector. The size of the nonlinearly transformed value vector is 1×d / 2. The SSM layer 820 determines a second state value and second compensation data corresponding to the first input data based on the nonlinearly transformed value vector in the first value vector set and the first state value corresponding to the first input data. The second compensation data is a vector, and its size is 1×d / 2. The first state value corresponding to the first input data is a vector, and the size of the first state value corresponding to the first input data is d×sd, where sd represents the product of s and d, and s includes the number of initial tokens in the inference sample. The second state value corresponding to the first input data is a vector, and the size of the second state value corresponding to the first input data is d×sd. The SSM layer 820 is used to perform the calculations of the above equations (12) and (13). The FC layer 830 includes a nonlinear activation layer 831 and a linear projection layer 832. The nonlinear activation layer 831 is used to perform a nonlinear transformation on the second compensation data to obtain the nonlinearly transformed second compensation data. The nonlinearly transformed second compensation data is a vector with a size of 1×d / 2. The linear projection layer 832 is used to perform a linear transformation on the nonlinearly transformed second compensation data, so that the size of the nonlinearly transformed second compensation data is increased from 1×d / 2 to 1×d to obtain the first compensation data. That is, the first compensation data is a vector with a size of 1×d. The embodiments of this application do not limit the specific values ​​of parameters such as transformation formulas and weights in FC layer 810 and FC layer 830.

[0266] 704. Based on the first attention output and the first compensation data, obtain the first output data of the first attention head.

[0267] In some embodiments, the first attention head obtains first output data based on a first attention output and first compensation data. This first output data is determined based on a weighted sum of the first attention output and the first compensation data.

[0268] For example, the first output data is calculated as shown in equation (15) or (16):

[0269] in, R represents i The weights, R i This indicates the first compensation data. express The weight, This represents the first attention output. R i This indicates the first output data. W O This represents the fourth weight matrix. The embodiments of this application specify... and The specific value is not limited.

[0270] In some embodiments, the target large model obtains the output result corresponding to the inference sample based on the first output data of the first attention head.

[0271] For example, suppose the structure of the target large model is as follows: Figure 1 As shown. Assume the layer containing the first attention head is layer 120, and the first attention head is attention head 1211. The target large model obtains the attention output of the self-attention module 121 based on the output data of each attention head in layer 120 (e.g., the output data of attention head 1211 and attention head 1212). The target large model also obtains the output data of FNN 122 based on the attention output of this self-attention module 121, and the output data of FNN 122 is the output data of layer 120. The target large model obtains the output result corresponding to the inference sample based on the output data of layer 120. The implementation method of the target large model obtaining the output result based on the output data of the attention heads is similar to the implementation method in the prior art, and will not be described in detail here.

[0272] exist Figure 7 The proposed method processes the value vector obtained by the attention head through a compensation module within the attention head to obtain compensation data. This compensation data is then used to compensate for the attention output of the attention head, thereby reducing the loss of inference accuracy caused by compressing the size of the attention head and improving the inference accuracy of the target large model. Furthermore, each attention head has its own compensation module, allowing each module to work independently with its corresponding attention head, which avoids excessively increasing the number of parameters in the target large model. For example, the total number of parameters in the compensation module of the target large model accounts for approximately 1% of the total parameters of the target large model.

[0273] In some embodiments, Figure 7 The reasoning process of the target large model in the middle is as follows: Figure 9 As shown. Figure 9 It includes two stages in the reasoning process of the target large model, with the pre-filling stage on the left and the decoding stage on the right. Figure 9 The target large model in the model includes at least one layer, such as one or more of layers 910, 920, and 930. Each layer of the target large model includes a self-attention module and an FNN. The self-attention module in each layer includes at least one attention head. The following explanation uses attention head 921 in layer 920 as an example. It should be understood that each attention head in each layer of the target large model can perform... Figure 9 The reasoning process is shown.

[0274] like Figure 9 As shown, in the pre-filling stage, the target large model obtains the input data X of an attention head 921 in layer 920 based on at least one initial word in the inference sample. p Attention head 921 utilizes input data X p And the first weight matrix W in attention head 921 Q Obtain the query vector Q p Q p = W Q ·X p Attention head 921 based on input data X p And the second weight matrix W in attention head 921 K Obtain the key vector K p K p = W K ·X p The attention head 921 utilizes input data X p And the third weight matrix W in attention head 921 V Obtain the value vector V p V p = W V ·X p The attention calculation module 922 in the attention head 921 utilizes the query vector Q. p Key vector K p Sum vector V p The attention output Y of the attention head 921 is obtained. att p The calculation process in the attention calculation module 922 is described above. The attention head 921 calculates the attention output Y. att p Obtain the attention output Y of the self-attention module in layer 920. p Y p Including W o ·Y att p Wo This is the fourth weight matrix of the self-attention module, where · denotes the dot product. Layer 920 inputs the attention output of this self-attention module into FNN 924 within layer 920 to obtain the output data of layer 920. The target large model determines the first generated word based on the output data of layer 920. See the description above for the specific implementation.

[0275] like Figure 9 As shown, in the pre-filling stage, the compensation module 923 in the attention head 921 also utilizes the value vector V p Obtain input data X p The corresponding second state value SS i The calculation process in compensation module 923 is described above.

[0276] For example, in the pre-filling stage, the attention head 921 is based on the key vector K p Sum vector V p Obtain the key-value vector KV p And store it.

[0277] For example, in the pre-filling stage, Figure 9 Layer 920 in the middle utilizes Figure 6 In the case of compression using the method described above, assuming M=2 for layer 920, meaning layer 920 includes two sets of attention heads, with each attention head having the same size. Assuming attention head 921 belongs to the first set of attention heads in layer 920, then each key vector and value vector generated by attention head 921 belongs to the key-value vector set KV1. i Assuming attention head 921 belongs to the second set of attention heads in layer 920, then each key vector and value vector generated by attention head 921 belongs to the key-value vector set KV2. i .

[0278] For example, in the pre-filling stage, assuming that attention head 921 belongs to the first set of attention heads in layer 920, attention head 921 is based on the query vector Q p Key vector K p Sum vector V p Obtain the metadata collection state1 from at least one of the following: i The metadata collection state1 i Including query vector Q p Key vector K p Sum vector V p Metadata corresponding to at least one item in the query vector Q. Assuming attention head 921 belongs to the second set of attention heads in layer 920, attention head 921 is based on the query vector Q. p Key vector K p Sum vector Vp Obtain the metadata collection state2 from at least one of the following: i The metadata collection state2 i Including query vector Q p Key vector K p Sum vector V p Metadata corresponding to at least one of the items in the dataset.

[0279] like Figure 9 As shown, in the decoding stage, the target large model obtains the input data X of the attention head 921 in layer 920 based on the first generated word generated in the pre-filling stage. i Attention head 921 utilizes input data X i And the first weight matrix W in attention head 921 Q Obtain the query vector Q i Q i = W Q ·X i Attention head 921 based on input data X i And the second weight matrix W in attention head 921 K Obtain the key vector K i K i = W K ·X i The attention head 921 utilizes input data X i And the third weight matrix W in attention head 921 V Obtain the value vector V i V i = W V ·X i The attention calculation module 922 in the attention head 921 utilizes the query vector Q. i Key vector K i Value vector V i And the set of key-value vectors generated by the attention head 921 during the pre-filling stage, to obtain the attention output y of the attention head 921. att i The calculation process in the attention calculation module 922 is described above. The compensation module 923 in the attention head 921 also utilizes the input data X. p The corresponding second state value SS i And the first set of value vectors, to obtain the input data X. i The corresponding second state value SS i+1 And the first compensation data R i The first set of value vectors includes the value vector V. p Sum vector V i The calculation process in compensation module 923 is described above. Attention head 921 outputs y based on the attention output.att i And the first compensation data R i Obtain the attention output y of the self-attention module in layer 920. i y i include Layer 920 inputs the attention output of the self-attention module into FNN 924 within layer 920 to obtain the output data of layer 920. The target large model determines the second generated word based on the output data of layer 920. See the description above for the specific implementation.

[0280] For example, during the decoding phase, if attention head 921 belongs to the first set of attention heads in layer 920, attention calculation module 922 utilizes query vector Q. i Key-value vector set kv1 i And the set of key-value vectors KV1 generated by attention head 921 during the pre-filling stage. i , obtain the attention output y of attention head 921 att i The key-value vector set kv1 i Including key vector K i Value vector V i Attention module 922 obtains the key-value vector set KV1. i+1 The key-value vector set KV1 i+1 Including KV1 i and the set of key-value vectors kv1 i When attention head 921 belongs to the second set of attention heads in layer 920, attention calculation module 922 uses query vector Q. i Key-value vector set kv2 i And the set of key-value vectors KV2 generated by attention head 921 during the pre-filling stage. i , obtain the attention output y of attention head 921 att i The key-value vector set kv2 i Including key vector K i Value vector V i Attention module 922 obtains the key-value vector set KV2. i+1 The key-value vector set KV2 i+1 Including KV2 i and the set of key-value vectors kv2 i .

[0281] For example, in the decoding phase, assuming that attention head 921 belongs to the first set of attention heads in layer 920, attention head 921 is based on the query vector Q i Key vector K iSum vector V i At least one of the following is used to obtain the first metadata. Attention head 921 obtains the first metadata and the metadata set state1 based on the first metadata and the metadata set state1. i Obtain the metadata collection state1 i+1 The first metadata includes the query vector Q. i Key vector K i Sum vector V i Metadata corresponding to at least one item in the query vector Q. Assuming attention head 921 belongs to the second set of attention heads in layer 920, attention head 921 is based on the query vector Q. i Key vector K i Sum vector V i At least one of the following is used to obtain the second metadata. Attention head 921 is based on the second metadata and the metadata set state2. i Obtain the metadata collection state2 i+1 .

[0282] In some embodiments, Figure 7 The training process of the compensation module in the middle is as follows Figure 10 As shown. Figure 10 The training process of the target large model includes two stages in the compensation module: the pre-filling stage on the left and the decoding stage on the right. Figure 10 The target large model structure and Figure 9 The structure of the target large model is similar, and will not be repeated here. The following explanation uses attention head 921 in layer 920 as an example. It should be understood that each attention head in each layer of the target large model can perform... Figure 10 The training process is shown.

[0283] like Figure 10 As shown, in the pre-filling stage, the target large model obtains the input data X of the attention head 921 based on at least one initial word from the training samples. p Attention head 921 based on input data X p Obtain the query vector Q p Key vector K p Value vector V p and query vector Q p1 Key vector K p1 Value vector V p1 (Not shown in the figure). The query vector Q p Key vector K p Value vector V p This refers to the vector generated by attention head 921 when its size is the same as its initial size in the initial large model. The query vector Q is...p1 Key vector K p1 Value vector V p1 This is the vector generated by attention head 921 when the size corresponding to attention head 921 is the optimal size (or third size) corresponding to attention head 921. The optimal size corresponding to attention head 921 is based on... Figure 3 or Figure 5 The dimension obtained by the method in [the text], the third dimension 921 corresponding to this attention head is based on [the method]. Figure 6 The dimensions are obtained using the method described in [the document]. Attention head 921 is based on the query vector Q. p Key vector K p Value vector V p The output data of attention head 921 is obtained, allowing the target large model to obtain the first generated word unit. The method for obtaining the output data of attention head 921 is similar to... Figure 9 The implementation of the pre-filling stage is similar, and will not be repeated here.

[0284] For example, in the pre-filling stage, Figure 10 Layer 920 in the middle utilizes Figure 6 In the case of compression using the method described above, assuming M=2 for layer 920, meaning layer 920 includes two sets of attention heads, with each set containing attention heads of the same size. Assuming attention head 921 belongs to the first set of attention heads in layer 920, then the key vector K generated by attention head 921... p Value vector V p Belongs to the key-value vector set KV1 p Attention head 921 is based on query vector Q. p Key vector K p Sum vector V p Obtain the metadata collection state1 from at least one of the following: p Assuming attention head 921 belongs to the second set of attention heads in layer 920, then the key vector K generated by attention head 921... p Value vector V p Belongs to the key-value vector set KV2 p Attention head 921 is based on query vector Q. p Key vector K p Sum vector V p Obtain the metadata collection state2 from at least one of the following: p .

[0285] like Figure 10 As shown, in the decoding stage, the target large model obtains the input data X of the attention head 921 in layer 920 based on the first generated word. g Attention head 921 based on input data X gObtain the query vector Q g Key vector K g Value vector V g and query vector Q g1 Key vector K g1 Value vector V g1 (Not shown in the figure). The query vector Q g Key vector K g Value vector V g This is the vector generated by attention head 921 when the size corresponding to attention head 921 is the initial size corresponding to attention head 921. This query vector Q g1 Key vector K g1 Value vector V g1 This refers to the vector generated by attention head 921 when the size corresponding to attention head 921 is the optimal size (or the third size) corresponding to attention head 921. Attention head 921 generates the vector based on the query vector Q. g Key vector K g Value vector V g The initial output data of attention head 921 is obtained by processing the set of key-value vectors generated by attention head 921 during the pre-filling stage (the key-value vectors in this set are generated when the size corresponding to attention head 921 is the initial size corresponding to attention head 921). This initial output data of attention head 921 is the attention output of attention head 921, meaning that this initial output data does not require calculation using the compensation data obtained by compensation module 923. The specific implementation method for obtaining the initial output data of attention head 921 is similar to that in the prior art and will not be elaborated here. Based on the initial output data of attention head 921, the target large model obtains the initial logical value corresponding to the second generated word. The implementation method for obtaining the logical value of the word is described above.

[0286] For example, in Figure 10 In the decoding phase, attention head 921 is based on the query vector Q g1 Key vector K g1 Value vector V g1 The attention output of the attention head 921 is obtained by processing the set of key-value vectors generated by the attention head 921 during the pre-filling stage (the key-value vectors in this set are generated when the size corresponding to the attention head 921 is the optimal size (or the third size) corresponding to the attention head 921). The compensation module 923 in the attention head 921 calculates the attention output based on the value vector V. g1 Sum vector V p1 The attention head 921 obtains compensation data based on the attention output and compensation data. The method by which the attention head 921 obtains its output data is similar to... Figure 9The method for obtaining the decoding stage is similar and will not be repeated here. The target large model obtains the third logical value corresponding to the second generated word based on the output data of attention head 921.

[0287] For example, in Figure 10 In the decoding stage, the target large model determines the value of the second loss function based on the second loss function, the initial logical value corresponding to the second generated word, and the third logical value. This second loss function represents the difference between the initial logical value and the third logical value corresponding to the second generated word when the size corresponding to attention head 921 is the same as the initial size corresponding to attention head 921. Based on the value of the second loss function, the target large model determines the gradient corresponding to the second loss function value and uses this gradient to adjust the parameters in the compensation module and / or the weights corresponding to the compensation module (i.e., in equations (15) and (16)). ) to perform optimization in order to obtain optimized parameters and / or optimized weights.

[0288] For example, the specific form of the second loss function is not limited in the embodiments of this application. For example, the second loss function may include the cross-entropy loss function. Any loss function that can be used in the prior art to represent the difference between the initial logic value and the third logic value can be applied to the embodiments of this application.

[0289] For example, the computing device can optimize the parameters and / or weights of the compensation module using the gradient corresponding to the second loss function value, based on various adaptive gradient algorithms. This application does not limit the specific type of adaptive gradient algorithm, such as Adagrad, RMSProp, Adam, etc.

[0290] For example, in Figure 10 In the decoding stage, the query vector Q is obtained. g1 Key vector K g1 Value vector V g1 In this case, actual pruning of the attention head 921 is unnecessary; instead, the pruning effect can be achieved using the mask matrix in the pruning module 925. For example, the attention head 921 uses the first mask matrix and the key vector K... g Obtain the key vector K g1 The size of the first mask matrix is ​​related to the key vector K. g The dimensions are the same. The elements in the first mask matrix take values ​​of 0 or 1. When the element in the first row and first column of the first mask matrix is ​​0, the first mask matrix and the key vector K are identical. g The key vector K is obtained by multiplying the elements corresponding to each element. g1The element in the first row and first column of the matrix is ​​0, thus achieving the pruning effect. By using the mask matrix in the pruning module 925, the training speed of the compensation module can be accelerated.

[0291] In some embodiments, since the method in this application does not require pruning of some lexical units input into the large model, when the number of lexical units in the inference samples of the large model is large, it will not lead to the loss of long sequence relevance, thereby not reducing the inference performance of the model.

[0292] For example, with the same initial large model and the same inference sample set used, the inference accuracy and compression ratio of the method in this application embodiment are shown in Table 1 compared with the prior art.

[0293] Table 1

[0294] In Table 1, large model A is the large model compressed according to the method in "RazorAttention: Efficient KV Cache CompressionThrough Retrieval Heads". Large model B provided in this embodiment is based on... Figure 6 The target large model is obtained by the method described in this application. The large model C provided in this embodiment is... Figure 7 The target large model is shown in Table 1. The second column of Table 1 represents the average sum of the sizes of all attention heads in the large model obtained after inference from multiple inference samples. The third column of Table 1 represents the average inference accuracy of the large model obtained after inference from multiple inference samples. For the initial large model and large model A, the sum of the sizes of all attention heads in the large model obtained after inference from different inference samples may differ. For large models B and C provided in this embodiment, the sum of the sizes of all attention heads in the large model obtained after inference from different inference samples is the same. As can be seen from Table 1, this embodiment can not only compress the initial large model to a compression ratio of 10 times, but also has a relatively small impact on the inference accuracy of the large model.

[0295] Figure 11 This is a schematic structural diagram of the computing device provided in the embodiments of this application. Figure 11 The computing device 1100 includes an acquisition module 1110 and a processing module 1120. Figure 11 The computing device 1100 in the middle is used to run large models. Figure 11 The computing device 1100 in the middle can be used to perform Figures 3 to 10 The method in the middle. Figure 11 The computing device 1100 in the middle is, for example, Figure 2 The server in the middle.

[0296] The computing device 1100 is used for execution Figure 3 When using the method in the middle, the acquisition module 1110 is used to execute step 310, and the processing module 1120 is used to execute steps 320-330 and 331-335.

[0297] The computing device 1100 is used for execution Figure 5 When using the method described above, the acquisition module 1110 is used to execute step 501, and the processing module 1120 is used to execute steps 502-511.

[0298] The computing device 1100 is used for execution Figure 6 When the method is in progress, the processing module 1120 is used to execute steps 601 and 602.

[0299] The computing device 1100 is used for execution Figure 7 When the method is in progress, the processing module 1120 is used to execute steps 701-704.

[0300] Both the acquisition module 1110 and the processing module 1120 can be implemented in software or in hardware. For example, the implementation of the processing module 1120 will be described below. Similarly, the implementation of the acquisition module 1110 can be referenced to that of the processing module 1120.

[0301] As an example of a software functional unit, processing module 1120 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, processing module 1120 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0302] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0303] As an example of a hardware functional unit, the processing module 1120 may include at least one computing device, such as a server. Alternatively, the processing module 1120 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0304] The processing module 1120 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module 1120 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 1120 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0305] Therefore, the modules of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0306] It should be noted that the computing device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the above methods. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computing device can be divided into different functional modules to complete all or part of the functions described above. For example, the acquisition module 1110 can be used to execute any step in the above methods, and the processing module 1120 can be used to execute any step in the above methods. The steps implemented by the acquisition module 1110 and the processing module 1120 can be specified as needed, and all the functions of the above computing device can be realized by implementing different steps in the above methods through the acquisition module 1110 and the processing module 1120 respectively.

[0307] Furthermore, the computing device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments above, which will not be repeated here.

[0308] The method provided in this application can be executed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as processing units, memory, and memory control units; the functions and structure of this hardware will be described in detail later. The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software. Optionally, the computer system can be a handheld device such as a smartphone, or a terminal device such as a personal computer; this application does not particularly limit this, as long as the method provided in this application can be used. The executing entity of the method provided in this application can be a computing device, or a functional module within the computing device capable of calling and executing programs.

[0309] Figure 12 This is a schematic structural block diagram of a computing device 1200 provided in an embodiment of this application. The computing device 1200 may be a server, a computer, or other device with computing capabilities. Figure 12 The computing device 1200 shown includes at least one processor 1210 and a memory 1220.

[0310] It should be understood that this application does not limit the number of processors and memories in the computing device 1200.

[0311] The processor 1210 executes instructions in the memory 1220, causing the computing device 1200 to implement the method provided in this application. Alternatively, the processor 1210 executes instructions in the memory 1220, causing the computing device 1200 to implement the various functional modules provided in this application, thereby implementing the method provided in this application.

[0312] Optionally, the computing device 1200 also includes a communication interface 1230. The communication interface 1230 uses an acquisition module, such as, but not limited to, a network interface card or a transceiver, to enable communication between the computing device 1200 and other devices or communication networks.

[0313] Optionally, the computing device 1200 further includes a system bus 1240, wherein the processor 1210, memory 1220, and communication interface 1230 are respectively connected to the system bus 1240. The processor 1210 can access the memory 1220 through the system bus 1240; for example, the processor 1210 can perform data read / write or code execution in the memory 1220 through the system bus 1240. The system bus 1240 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1240 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 12 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0314] In one possible implementation, the processor 1210 primarily functions to interpret the instructions (or code) of a computer program and process data within the computer software. The instructions of the computer program and the data within the computer software can be stored in the memory 1220 or the cache of the processor 1210.

[0315] Optionally, processor 1210 may be an integrated circuit chip with signal processing capabilities. By way of example and not limitation, processor 1210 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor is a microprocessor, etc. For example, processor 1210 is a central processing unit (CPU).

[0316] The memory 1220 provides runtime space for processes in the computing device 1200. For example, the memory 1220 stores the computer program (specifically, the program code) used to generate the process. After the computer program is run by the processor to generate a process, the processor allocates corresponding storage space for the process in the memory 1220. Furthermore, the aforementioned storage space further includes text segments, initialized data segments, bit initialized data segments, stack segments, heap segments, etc. The memory 1220 stores data generated during the process's execution, such as intermediate data or process data, in the storage space corresponding to the process.

[0317] Optionally, the memory, also known as RAM, is used to temporarily store the data processed by the processor 1210, as well as data exchanged with external storage devices such as hard disks. As long as the computer is running, the processor 1210 will load the data required for processing into RAM for computation, and then transfer the result back out after the computation is complete.

[0318] By way of example and not limitation, memory 1220 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile storage medium may be, for example, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus DRAM (DRDRAM). It should be noted that the memory 1220 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0319] The above-described structure of the computing device 1200 is merely illustrative and is not intended to limit the application. The computing device 1200 in this application includes various hardware components found in existing computer systems. For example, the computing device 1200 may also include other storage devices besides the memory 1220, such as disk storage. Those skilled in the art should understand that the computing device 1200 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computing device 1200 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the computing device 1200 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 12 All the devices shown.

[0320] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a desktop computer, laptop computer, or smartphone, or other terminal device.

[0321] like Figure 13 As shown, the computing device cluster includes at least one computing device 1200. The memory 1220 of one or more computing devices 1200 in the computing device cluster may store the same instructions for performing the methods described above.

[0322] In some possible implementations, the memory 1220 of one or more computing devices 1200 in the computing device cluster may also each store a portion of the instructions for executing the above-described method. In other words, a combination of one or more computing devices 1200 can jointly execute the instructions of the above-described method.

[0323] It should be noted that the memory 1220 in different computing devices 1200 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the aforementioned computing device. That is, the instructions stored in the memory 1220 of different computing devices 1200 can implement the functions of one or more modules within the aforementioned computing device.

[0324] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 14 One possible implementation is shown. For example... Figure 14 As shown, the two computing devices 1200A and 1200B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0325] It should be understood that Figure 14The functions of the computing device 1200A shown can also be performed by multiple computing devices 1200. Similarly, the functions of the computing device 1200B can also be performed by multiple computing devices 1200.

[0326] In this embodiment of the application, a computer program product containing instructions is also provided. The computer program product may be software or program products containing instructions capable of running on a computing device cluster or stored on any available medium. When run by the computing device cluster, it causes the computing device cluster to perform the methods provided above, or causes the computing device cluster to implement the functions of the computing device provided above.

[0327] This application embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., high-density digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method provided above.

[0328] In this application embodiment, a chip system is also provided. The chip system includes logic circuitry for coupling with an input / output interface to transmit data via the input / output interface, thereby executing the methods provided above.

[0329] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0330] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0331] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0332] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0333] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0334] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0335] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An optimization method for a large-scale artificial intelligence model, characterized in that, include: Obtain an initial large model, which comprises L layers, each of the L layers comprising at least one attention head, where L is a positive integer; Input N training samples into the initial large model to obtain the initial logical value of each of the N training samples, wherein the training sample includes at least one word character and N is a positive integer; Based on the initial large model, the size corresponding to each attention head in the initial large model is optimized through N iterations; Wherein, in the nth optimization process of the N iterations, n is a positive integer less than or equal to N, and the method includes: The size corresponding to each attention head in the initial large model is determined as the first size corresponding to each attention head, thus obtaining the first large model; Input the N training samples into the first large model to obtain the first logical value of each of the N training samples; Based on the initial logical value and the first logical value of each of the N training samples, determine the first loss gradient of each training sample; The first gradient projection of the nth training sample is determined based on the first loss gradient of the nth training sample and the average of the first loss gradients of the N training samples. Based on the first gradient projection of the nth training sample, the first size corresponding to each attention head in the first large model is optimized to obtain the second size corresponding to each attention head. The second size corresponding to each attention head is used to obtain the second large model. The sum of the second sizes corresponding to each attention head is less than the sum of the sizes corresponding to each attention head in the initial large model.

2. The method according to claim 1, characterized in that, In the first optimization process of the N iterations, the first size corresponding to each attention head is the same, and the first size is determined according to the total target size and the number of attention heads in the initial large model.

3. The method according to claim 1 or 2, characterized in that, The first gradient projection of the nth training sample is the difference between the first loss gradient of the nth training sample and the average of the first loss gradients of the N training samples.

4. The method according to any one of claims 1 to 3, characterized in that, The step of optimizing the size corresponding to each attention head in the first large model based on the first gradient projection of the nth training sample to obtain the second size corresponding to each attention head includes: Based on the first gradient projection of the nth training sample, the first size corresponding to each attention head is optimized to determine the first optimized size corresponding to each attention head; Based on the first optimized size, average size, and target size corresponding to each attention head, a second size corresponding to each attention head is determined, and the average size corresponding to each attention head is determined based on the historical values ​​of the size corresponding to each attention head.

5. The method according to claim 4, characterized in that, The second size corresponding to each attention head is determined based on the difference between the first optimized size corresponding to each attention head and the first value corresponding to each attention head. The first value corresponding to each attention head is determined based on the difference between the average size corresponding to each attention head and the target size.

6. The method according to any one of claims 1 to 5, characterized in that, When n is less than N, the second largest model is used as the first largest model in the (n+1)th optimization process of the N iterations, where n+1 is a positive integer less than or equal to N; or, When n=N, the second large model is used as the target large model, which is applied to the reasoning task.

7. The method according to any one of claims 1 to 6, characterized in that, After completing the N iterations of optimization, the method further includes: Obtain the average value of the second size corresponding to each attention head; When the average value of the second size is greater than the target size, reduce the second size corresponding to one or more attention heads in the second large model so that the average value of the size corresponding to each attention head in the second large model is equal to the target size; When the average value of the second size is less than the target size, increase the second size corresponding to one or more attention heads in the second large model so that the average value of the size corresponding to each attention head in the second large model is equal to the target size.

8. The method according to any one of claims 1 to 7, characterized in that, After completing the N iterations of optimization, the method further includes: Based on the second size corresponding to each attention head of the target layer, determine M third sizes and M attention head sets corresponding to the target layer. The target layer belongs to the L layers. The M third sizes belong to the second size corresponding to each attention head of the target layer. Each attention head set in the M attention head sets includes at least one attention head of the target layer. The m-th third size in the M third sizes corresponds to the m-th attention head set in the M attention head sets. M is a positive integer less than the number of attention heads in the target layer, and m is a positive integer less than or equal to M. The size corresponding to each attention head in the target layer is determined as the third size corresponding to the set of attention heads to which each attention head belongs, thus obtaining the target large model.

9. The method according to claim 8, characterized in that, The M third dimensions and the M attention head sets minimize the difference between the first total size and the second total size. The first total size includes the sum of the dimensions corresponding to each attention head in the target layer when the size corresponding to each attention head is the third dimension corresponding to the attention head set to which each attention head belongs. The second total size includes the sum of the dimensions corresponding to each attention head in the target layer when the size corresponding to each attention head is the second dimension corresponding to each attention head.

10. The method according to any one of claims 1 to 9, characterized in that, The first attention head in the target large model includes a compensation module. The compensation module is used to determine first compensation data based on a first set of value vectors. The first set of value vectors includes at least one value vector generated by the first attention head. The first output data of the first attention head is determined based on the first attention output and the first compensation data. The first attention output is determined based on the query vector corresponding to the first input data of the first attention head and a first set of key-value vectors. The first key-value vector includes at least one key-value vector generated by the first attention head.

11. The method according to claim 10, characterized in that, The compensation module is used to determine the second state value corresponding to the first compensation data and the first input data based on the first state value corresponding to the first input data and the first value vector set. The first state value corresponding to the first input data is the second state value corresponding to the second input data obtained when the first attention head processes the second input data. The second input data is the input data processed by the first attention head before processing the first input data.

12. The method according to claim 10 or 11, characterized in that, The compensation module includes a state-space model (SSM).

13. The method according to any one of claims 10 to 12, characterized in that, The first output data is determined based on the weighted sum of the first attention output and the first compensation data.

14. A computing device, characterized in that, The computing device includes a module for performing the method as described in any one of claims 1 to 13.

15. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 13.

16. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 13.

17. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 13.

Images