Large language model performance optimization method and device, storage medium and electronic equipment

Abstract

The invention provides a large language model performance optimization method and device, a storage medium and electronic equipment, and relates to the field of natural language processing. The electronic equipment processes the generated text through a large language model to obtain hidden state information transmitted to a main solution terminal; wherein the large language model further comprises a plurality of slave solution terminals parallel to the master solution terminal, and a preset arrangement sequence among the plurality of slave solution terminals represents an arrangement sequence among the decoding results; then, combining the sequence information of each slave solution code head with the hidden state information to obtain to-be-decoded information of each slave solution code head; and finally, according to the candidate word set decoded from each piece of to-be-decoded information, obtaining a subsequent optimal prediction text of the generated text. Therefore, after the sequence information of each slave solution code head is combined with the hidden state information, the respective to-be-decoded information is decoded in parallel through the plurality of slave solution code heads, so that the reasoning speed of the large language model can be remarkably improved while the reasoning precision is kept.

Description

Technical Field

[0001] This application relates to the field of natural language processing, and more particularly, to a method, apparatus, storage medium, and electronic device for optimizing the performance of large language models. Background Art

[0002] Since the advent of ChatGPT, generative large language models have become a hot topic of research and discussion. Most current generative large language models generate text in an autoregressive manner. Specifically, during the generation process, the model predicts the next most likely word based on the generated text sequence, adds it to the sequence, and then repeats this process until the entire text is completed.

[0003] Exemplarily, the following结合 Figure 1 illustrates this process more intuitively. Taking the input text sequence "我去篮" as an example, the text sequence "我去篮" is input into a large language model called the target model. The target model generates the token of the next word "球" (for ease of description, "word" is used as the description unit hereinafter, and it is actually a token) through one forward inference. Then, the newly generated "球" is appended to the previous input, and the resulting text sequence "我去篮球" is input into the target model. The target model generates the next word "场" through the second forward inference, appends the newly generated word "场" to the previous input, and continues to use the resulting text sequence "我去篮球场" as the next input. And so on, to obtain the complete output text of the target model.

[0004] Therefore, this method relies on the probability distribution of the model, that is, calculates the conditional probability of each possible word according to the context and selects the word with the highest probability as the output; however, it is limited to outputting only one word at a time, resulting in low inference efficiency.

[0005] In view of this, several technical solutions for accelerating inference have been proposed during the research process. The most representative one is the multi-word prediction method, that is, generating multiple candidate words at one time and selecting the best predicted text from them. After practice, it is found that the prediction accuracy of multiple candidate words is often not ideal, resulting in large fluctuations in the acceptance rate of candidate words; moreover, when the acceptance rate decreases, the inference acceleration effect will be significantly weakened, and may even offset the efficiency improvement brought by multi-word prediction. And other related technologies proposed during the research process, although they can improve the accuracy of multi-word prediction, are at the cost of sacrificing the inference speed. Therefore, how to significantly improve the inference speed of large language models while maintaining the inference accuracy has become a key problem to be solved urgently. Summary of the Invention

[0006] To overcome at least one deficiency in the prior art, this application provides a method, apparatus, storage medium, and electronic device for optimizing the performance of large language models, specifically including: In a first aspect, the present application provides a method for optimizing the performance of a large language model, wherein the large language model includes a main decoding head, and the method includes: Processing the generated text by means of a large language model to obtain hidden state information transmitted to the main decoding head, wherein the large language model further comprises a plurality of slave decoding heads in parallel with the main decoding head, and a preset arrangement order between the plurality of slave decoding heads represents an arrangement order between decoding results; Combining the sequence information of each of the slave decoding heads with the hidden state information to obtain information to be decoded of each of the slave decoding heads; According to the candidate word set decoded from each piece of information to be decoded, the best predicted text subsequent to the generated text is obtained.

[0007] In a second aspect, the present application provides a large language model performance optimization device, wherein the large language model includes a main decoding head, and the device includes: a text feature module, configured to process the generated text through a large language model to obtain hidden state information transmitted to the main decoding head, wherein the large language model further comprises a plurality of slave decoding heads in parallel with the main decoding head, and a preset arrangement order between the plurality of slave decoding heads represents an arrangement order between decoding results; A feature optimization module, used for combining the sequence information of each of the slave decoding heads with the hidden state information to obtain information to be decoded of each of the slave decoding heads; The feature decoding module is used to obtain the best predicted text subsequent to the generated text based on the candidate word set decoded from each piece of information to be decoded.

[0008] In a third aspect, the present application provides a storage medium storing a computer program, which implements the large language model performance optimization method under the condition of being executed by a processor.

[0009] In a fourth aspect, the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program implements the large language model performance optimization method under the condition that it is executed by the processor.

[0010] Compared with the prior art, this application has the following beneficial effects: In the large language model performance optimization method, device, storage medium and electronic device provided by the present application, the electronic device processes the generated text through the large language model to obtain hidden state information transmitted to the main decoding head; wherein the large language model also includes multiple slave decoding heads in parallel with the main decoding head, and the preset arrangement order between the multiple slave decoding heads represents the arrangement order between the decoding results; then, the sequence information of each slave decoding head is combined with the hidden state information to obtain the information to be decoded of each slave decoding head; finally, according to the candidate word set decoded from each piece of information to be decoded, the best predicted text for the generated text is obtained. In this way, by combining the sequence information of each slave decoding head with the hidden state information, the information to be decoded of each slave decoding head is obtained, and the respective information to be decoded is decoded in parallel by multiple slave decoding heads, so that the reasoning speed of the large language model can be significantly improved while maintaining the reasoning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 A schematic diagram of the principle of the autoregressive text generation method provided in the embodiment of the present application; Figure 2 A schematic diagram of the interaction principle between the draft model and the target model provided in the embodiment of the present application; Figure 3 One of the schematic diagrams of the principle of multiplexing hidden state information provided in an embodiment of the present application; Figure 4 The second schematic diagram of the principle of multiplexing hidden state information provided in the embodiment of the present application; Figure 5 A flowchart of a large language model performance optimization method provided in an embodiment of the present application; Figure 6 A schematic diagram of the principle of combining hidden state information and sequence information provided in an embodiment of the present application; Fig. 7A One of the detailed schematic diagrams of the large language model performance optimization method provided in the embodiment of the present application; Figure 7B A second detailed schematic diagram of the large language model performance optimization method provided in an embodiment of the present application; Figure 8 A tree-shaped combination relationship diagram provided for an embodiment of the present application; Fig. 9The attention mask matrix provided by the embodiment of the present application; Fig.10 A schematic diagram of a shielding block provided in an embodiment of the present application; Fig.11 A schematic diagram of the principle of shielding calculation of a shielding block provided in an embodiment of the present application; Fig. 12A A comparison chart of the acceptance rate based on Qwen2-7B provided in the embodiments of the present application; Fig. 12B A comparison chart of the acceptance rate based on Qwen2-72B provided in the embodiments of the present application; Fig.13 A schematic diagram of the structure of a large language model performance optimization device provided in an embodiment of the present application; Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of the embodiment of the present application (hereinafter referred to as the embodiment) clearer, the technical solution in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is a part of the embodiment of the present application, not all of the embodiments. Generally, the components of the embodiment of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0014] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0015] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0016] In the description of the present application, it should be noted that the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. In addition, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0017] Based on the above statement, as introduced in the background technology, although accelerated reasoning technologies such as multi-word prediction can improve efficiency, the insufficient prediction accuracy leads to fluctuations in acceptance rate and weakened acceleration effect, and technologies to improve accuracy often sacrifice speed. Therefore, how to significantly improve the reasoning speed of large language models while maintaining accuracy has become a key issue at present.

[0018] In order to make the objectives, technical solutions and advantages of the embodiments of the present application easier to understand, the research process of proposing this solution is described in detail below.

[0019] During the research, it was considered that the large model parameters of the target model were the main reason for the slow word-by-word reasoning speed. Therefore, if multiple candidate words could be inferred with a smaller amount of calculation for the target model to screen, the reasoning speed could be improved. Based on this concept, it was proposed to use a model with relatively few model parameters, called the draft model of the target model, to quickly generate multiple candidate words. Then, the target model verifies these candidate words and accepts some of them based on the verification results.

[0020] For example, continue with Figure 1 Take the text sequence "I go to the basket" as an example. Figure 2As shown, the text sequence "I go to the basketball court" is input to the draft model, and the draft model generates 4 candidate words "court fight" after 4 forward reasonings. The 4 candidate words are appended to the text sequence "I go to the basketball court" to obtain the text sequence "I go to the basketball court fight". Then, the text sequence "I go to the basketball court fight" is input to the target model at one time, and the target model generates 5 candidate words "court play basketball" after one forward reasoning. Finally, the 4 candidate words "court fight" generated by the draft model and the 5 candidate words "court play basketball" generated by the target model are compared in turn until different candidate words are compared, and the comparison is stopped, and the same part "court play basketball" is intercepted as the best predicted text for the subsequent text sequence "I go to the basketball court". In this way, the target model only needs to generate 4 words after one forward reasoning. It should be understood that the draft model usually uses a model with a small number of parameters in the same series as the target model, and the reasoning speed is much faster than the target model. When the acceptance rate of the candidate words generated by the draft model is large, the reasoning speed of the target model can be significantly improved.

[0021] However, in practice, it was found that although the above draft model has a small number of parameters, it still takes up a certain amount of computing resources (including GPU resources and memory resources) during reasoning, which limits its promotion and application, especially when computing power is scarce.

[0022] In view of this, the research process further proposed a technology to reuse the hidden state information of the target model to generate multiple candidate words. Before explaining this technology, it is necessary to first understand that the Transformer module is the basic unit of the large language model based on the Transformer architecture. Figure 3 As shown in the figure, multiple slave decoding heads are added in parallel with the main decoding head (Lm_Head) after the last hidden layer of the Transformer module. Each slave decoding head can generate a candidate word set containing at least one candidate word; then, the candidate words in these candidate word sets are verified through a forward reasoning process. Among them, the main decoding head (Lm_Head) is used to convert the hidden state information of the model into actual words in the vocabulary. The role of the slave decoding head is similar to that of the main decoding head, which decodes the same hidden state information in parallel to generate more candidate words, thereby improving the reasoning efficiency and the quality of the candidate text.

[0023] For the above-mentioned multiple sub-decoding heads, it was found in practice that there is a preset arrangement order between the newly added multiple sub-decoding heads, which is used to characterize the arrangement order between the decoding results. However, the sub-decoding head cannot perceive its own sequence position during decoding, which sometimes leads to an unstable acceptance rate of the generated candidate words. When the acceptance rate of the candidate words becomes low, the acceleration effect is not obvious. Specifically, since more sub-decoding heads are added, each sub-decoding head can generate at least one candidate word. As the number of sub-decoding heads and candidate words increases, the generated permutations and combinations also increase. Therefore, more candidate words need to be verified for forward reasoning, but each additional candidate sequence will increase the amount of calculation of the attention mechanism. If the acceptance rate is unstable, it is easy to waste a lot of computing resources, but only a small number of candidate words can actually be accepted, which makes the reasoning speed slower.

[0024] Therefore, inspired by the Long Short-Term Memory (LSTM) model, this model introduces the dependency of time steps to carry the order information between sequences and proposes the following Figure 4 The solution shown. Figure 4 As shown, the predicted word and hidden state information output by the main decoding head are input into the first slave decoding head, and the candidate word set output by the first slave decoding head and the hidden state information are input into the second slave decoding head together to obtain the candidate word set output by the second decoding head. Similarly, multiple candidate word sets can be generated in sequence. Finally, the candidate words in these candidate word sets are verified through a forward reasoning process.

[0025] However, in practice, it was found that since decoding needs to be performed one by one from the decoding heads, and each decoder also needs to perform a certain amount of calculation, the inference speed will be significantly reduced after the number of decoding heads increases to a certain level.

[0026] In summary, how to significantly improve the reasoning speed of large language models while maintaining reasoning accuracy has become a key issue that needs to be solved urgently.

[0027] Based on the discovery of the above technical problems, the following technical solutions are proposed after creative work to solve or improve the above problems. It should be noted that the defects existing in the solutions in the above prior art are the results obtained after practice and careful study. Therefore, the discovery process of the above problems and the solutions proposed in the embodiments of the present application for the above problems below should be regarded as contributions to the present application in the process of invention and creation, and should not be understood as technical contents known to those skilled in the art.

[0028] In view of the above technical problems, this embodiment provides a large language model performance optimization method. Figure 5 As shown, the method includes: S1, processes the generated text through the large language model to obtain the hidden state information transmitted to the main decoding head.

[0029] The large language model further includes a plurality of slave decoding heads in parallel with the main decoding head, and a preset arrangement order among the plurality of slave decoding heads represents an arrangement order among decoding results.

[0030] S2, combining the sequence information of each slave decoding head with the hidden state information to obtain the information to be decoded of each slave decoding head.

[0031] S3, obtaining the best predicted text subsequent to the generated text according to the candidate word set decoded from each piece of information to be decoded.

[0032] In this way, by combining the sequence information of each slave decoding head with the hidden state information, the information to be decoded of each slave decoding head is obtained, and the respective information to be decoded is decoded in parallel by multiple slave decoding heads, thereby significantly improving the inference speed of the large language model while maintaining the inference accuracy.

[0033] It should be understood that for the large language model performance optimization method provided in this embodiment, the electronic device implementing the method may be, but is not limited to, a mobile terminal, a tablet computer, a laptop computer, a desktop computer, a server, etc. The server may be a single server or a server group. The server group may be centralized or distributed (for example, the server may be a distributed system). In some embodiments, the server may be local or remote relative to the user terminal. In some embodiments, the server may be implemented on a cloud platform; as an example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud (Community Cloud), a distributed cloud, an inter-cloud (Inter-Cloud), a multi-cloud (Multi-Cloud), etc., or any combination thereof. In some embodiments, the server may be implemented on an electronic device having one or more components.

[0034] To make the solution provided by this embodiment clearer, the following uses a server as an electronic device to implement the method. Figure 5 Each step of the method shown is described in detail. However, it should be understood that the operations of the flowchart may not be implemented in order, and steps without logical contextual relationship may be reversed or implemented simultaneously. In addition, those skilled in the art may add one or more other operations to the flowchart, or remove one or more operations from the flowchart under the guidance of the content of this application. Figure 5 , the method comprising: S1, processes the generated text through the large language model to obtain the hidden state information transmitted to the main decoding head.

[0035] Among them, the large language model also includes multiple slave decoding heads in parallel with the main decoding head, and the preset arrangement order between the multiple slave decoding heads represents the arrangement order between the decoding results. It can be understood that in the large language model of this embodiment, in addition to the main decoding head, there are multiple parallel slave decoding heads; these slave decoding heads work in parallel with the main decoding head and jointly participate in the decoding process of the model. In addition, different slave decoding heads will generate different candidate words, and the order in which these candidate words are combined into text is determined by the arrangement order of the slave decoding heads. For example, if slave decoding head A is arranged before slave decoding head B, then when candidate word A generated by slave decoding head A is combined with candidate word B generated by slave decoding head B, candidate word A needs to be arranged in front of candidate word B.

[0036] In addition, the generated text refers to the part of the text content that has been generated and output by the large language model. In addition, the hidden state information refers to the intermediate representation extracted during the large language model processing of the generated text, which contains the context information and current state accumulated by the model when processing the text. During the processing of the large language model, the server transmits this hidden state information to the main decoding head, and then it can infer the subsequent predicted words of the generated text.

[0037] like Figure 6 As shown, for the above-mentioned multiple slave decoding heads, they are drawn from the position before the main decoding head in the forward propagation path, so that the multiple slave decoding heads can simultaneously decode the hidden state information transmitted to the main decoding head. In this embodiment, the branch where the main decoding head is located is called the trunk model of the large language model. As an optional implementation, the slave decoding head can adopt a structure similar to that of the main decoding head. For example, the structure of each slave decoding head is single-layer or multi-layer Continue Layer (a linear layer that transforms the latent space to the dictionary dimension). Each layer The structure is ,here is the latent space input, is the activation function, is a linear layer. Therefore, compared to training an additional draft model, only a simple fine-tuning of each slave decoding head is required. During training, only the final hidden state information of the backbone model is used without passing through the main prediction head. Layer deal with.

[0038] Based on the above embodiment, the text generated in step S1 and the hidden state information are introduced. Figure 5 , let's continue Figure 5 Step S2 in the description is as follows: S2, combining the sequence information of each slave decoding head with the hidden state information to obtain the information to be decoded of each slave decoding head.

[0039] For this, see Figure 6 , the server can map the arrangement position of each slave decoding head to the sequence information of each slave decoding head through a sine function or a cosine function. This mapping method draws on the position encoding technology in the Transformer model. Its core principle is to use sine or cosine functions to generate position vectors, which are added to the input word embedding vector. In this way, the model can understand the positional relationship in the input sequence without explicit position information. In this embodiment, a similar method is adopted, but it is applied to the arrangement position of the slave decoding head. That is, the server converts the arrangement position of each slave decoding head into the corresponding sequence information through a sine or cosine function. In this way, the model can understand the relative position of the slave decoding head during the decoding process, so as to perform the decoding operation more efficiently.

[0040] Then, the server further combines the sequence information of each slave decoding head with the hidden state information to obtain the information to be decoded from each slave decoding head. It can be understood that the hidden state information is combined with the sequence information after position encoding to form the information to be decoded. The information to be decoded not only contains context information, but also enables each slave decoding head to perceive its position in the arrangement order. Among them, the sequence information helps the slave decoding head to better understand the position relationship of the candidate words in the sequence, and enhances the order dependence of the model during the decoding process. In this way, the model can select suitable candidate words more accurately, thereby improving the acceptance rate of candidate words and the accuracy of decoding.

[0041] Based on the description of the sequence information from the decoding header and the information to be decoded in step S2 in the above implementation, the following is continued. Figure 5 Step S3 in the following is described: S3, obtaining the best predicted text subsequent to the generated text according to the candidate word set decoded from each piece of information to be decoded.

[0042] Each candidate word set includes at least one candidate word. It can be understood that the server decodes candidate word sets from each piece of information to be decoded, and each candidate word set contains at least one candidate word. These candidate word sets are decoded by multiple decoding heads respectively; then, by evaluating and screening these candidate word sets, at least part of the candidate word sets are selected. Each of these selected candidate word sets will select one of the candidate words and combine them into the best predicted text for the generated text in the order of arrangement.

[0043] For example, the following is combined with Figure 6 This process is explained more intuitively. Figure 62 decoders are shown in the figure. For the generated text "Take one basketball, we", the candidate word set decoded from the first decoder includes "going, very", and the candidate word set decoded from the second decoder includes "to, off, happy"; and the final best predicted text is "going to". It can be seen that "going" is selected from "going, very", and "to" is selected from "to, off, happy".

[0044] against Figure 5 In step S3, various implementation methods have been proposed during the research process. Fig. 7A As shown, step S3 may include: S3-1A, combining the candidate words in each candidate word set in the order of arrangement to obtain multiple candidate texts.

[0045] For example, see Figure 6 , "going, very" and "to, off, happy" can be combined into "going to", "going off", "going happy", "very to", "very off", "very happy" in order of arrangement, a total of 6 candidate texts. It is necessary to find the best predicted text from the 6 candidate texts. However, it should be understood that when the number of candidate words in each candidate text is large, the best predicted text may be a partial fragment of one of the candidate texts, rather than the entire candidate text.

[0046] S3-2A, determining the best predicted text from multiple candidate texts.

[0047] During the research, we found that since the backbone model often contains a large number of parameters, if the backbone model verifies the candidate text once, it will not actually improve the reasoning speed, but will reduce the reasoning speed. The reason is that Figure 6 6 candidate texts can be obtained, which means that the backbone model needs to perform 6 forward inferences, and the candidate texts finally verified only contain 2 candidate words. However, according to the most common reasoning method, 6 forward inferences can get 6 accurate predicted words. In this regard, after further research, it was found that some candidate texts have obvious semantics and grammar. For example, Figure 6Among the 6 candidate texts obtained, "very to" obviously does not conform to the English expression grammar. During the research process, we first thought of accumulating the probabilities of the candidate words in each candidate text to obtain the score of each candidate text. However, after practice, it was found that this method did not take into account the semantic information of the context, resulting in the final retention of the candidate word with the highest probability in each candidate word set, but many times the candidate text composed of the candidate word with the highest probability is not the best text. Therefore, in this embodiment, a model with relatively small parameters can be used to quickly score the semantics and grammar of the above 6 candidate texts, and the one with the highest score can be selected for verification by the backbone model.

[0048] Based on the above concept, the server can splice at least part of the text fragments in the generated text with each candidate text to obtain multiple texts to be scored; score the multiple texts to be scored respectively through a pre-trained text scoring model to obtain a score for each text to be scored; and determine the best predicted text based on the score of each text to be scored.

[0049] It should be understood that this embodiment splices at least part of the text fragments in the generated text with each candidate text, mainly to provide more contextual information to make the scoring more accurate. In addition, for the above-mentioned text scoring model, the BERT model can be used for fine-tuning. As a pre-trained deep learning model, BERT already has the ability to capture bidirectional contextual information of the text, and is very suitable for processing semantic and grammatical tasks. Therefore, it is only necessary to use the pre-trained BERT model and train it for specific tasks on its basis to adapt it to specific scoring requirements.

[0050] In this way, each candidate text is scored by a text scoring model with smaller parameters, so that the score of each candidate text is quickly obtained, and the candidate text with the highest score is selected for further verification by the backbone model, thereby determining the best subsequent predicted text for the generated text.

[0051] As another optional implementation of the above step S3, a tree-like attention mask matrix can be used to perform a forward reasoning on the candidate words in the plurality of candidate word sets to complete the verification. Figure 7B As shown, step S3 may also include: S3-1B, multiply each candidate word set in order of arrangement to obtain multiple subsequences to be verified.

[0052] The number of candidate word sets in each subsequence to be verified is equal to the number of candidate words in the previous subsequence to be verified. Figure 6 The predicted word "are" output by the main decoding head and the two candidate word sets decoded from the decoding head are taken as examples, and combined with Figure 8This process is explained more intuitively. Figure 8 In the tree-shaped combination relationship diagram shown, "are" can be combined with "going, very". Since there is only one "are", the subsequence to be verified obtained by doubling "going, very" includes one candidate word set "going, very". Based on the subsequence to be verified obtained by doubling "going, very", "going, very" can be combined with "to, off, happy" respectively. Therefore, the subsequence to be verified obtained by doubling "to, off, happy" includes two candidate word sets "to, off, happy".

[0053] Based on the description of the subsequence to be verified in the above embodiment, Figure 7B Step S3-2B in the example is as follows: S3-2B, concatenates the generated text and multiple subsequences to be verified into a complete text sequence, and constructs an attention mask matrix based on the combination order between candidate words in the text sequence.

[0054] Among them, the rows and columns of the attention mask matrix correspond one-to-one to the candidate words in the text sequence respectively, and the attention mask matrix includes unrelated elements marked with masks, and the candidate words associated with the unrelated elements do not have an adjacent combination relationship.

[0055] For example, continue with Figure 8 Take the tree-shaped combination relationship diagram shown as an example, and combine Fig. 9 This process is explained more intuitively. Figure 8 The text sequence obtained by the tree-shaped combination relationship diagram shown is "going, very, to, off, happy, to, off, happy". Fig. 9 The corresponding attention mask matrix is ​​shown in , where the 8 rows of the attention mask matrix correspond to the 8 candidate words in the text sequence, and the 8 columns of the attention mask matrix also correspond to the 8 candidate words in the text sequence. Fig. 9 The elements in the upper triangle are marked as irrelevant elements. Different from the conventional attention mask matrix, Fig. 9 In the lower triangle shown, the elements filled with patterns represent that there is an adjacent combination relationship between the candidate words located in the row and column positions of the elements; therefore, the elements not marked with patterns can be marked with masks as unrelated elements.

[0056] Based on the description of the attention mask matrix in step S3-2B in the above embodiment, we will continue to Figure 7B Step S3-3B in the example is as follows: S3-3B, takes the attention mask matrix as the constraint condition of each decoding layer, and processes the feature information of the text sequence through the self-attention mechanism in each decoding layer in turn; and determines the best predicted text based on the decoding results of the last layer in multiple decoding layers.

[0057] In this way, the above-mentioned attention mask matrix ensures that when the decoding layer performs self-attention calculation, each candidate word is only allowed to see the previous candidate word in its combined path, and will not see the information of other paths, thereby avoiding interference between different paths. Through this mechanism, each candidate word can only focus on the candidate word in front of it, ensuring independence and order in the decoding process. It can be understood that by using the attention mask matrix as a constraint condition for each decoding layer, one forward reasoning can verify the candidate words in the entire candidate word set. Therefore, through the above-mentioned attention mask matrix, not only the decoding efficiency is improved, but also it is ensured that each candidate word only relies on the correct path before it during the decoding process, thereby improving the accuracy and reliability of the overall decoding.

[0058] During the research process, it was also found that compared with the conventional attention mask matrix, the attention mask matrix constructed in this embodiment is a sparse matrix, that is, there are a large number of unrelated elements. In the calculation process of the self-attention mechanism, even if the weight scores of the positions of the unrelated elements are calculated, they will not participate in the weighted summation later. It can be understood that the weight scores corresponding to these unrelated elements will be ignored during the calculation process, but will not affect the final weighted summation result. In view of this, this embodiment also provides the following optional implementation method of step S3-3B to reduce the amount of calculation in the reasoning process of the large language model, thereby improving the reasoning speed of the large language model. In order to make the implementation method of S3-3B easier to understand, the structure of the GPU and the optimization method for its structure are explained below.

[0059] First of all, it should be understood that there are two different levels of memory in the GPU, namely SRAM (static random access memory) and HBM (high bandwidth memory). The two are significantly different in performance, capacity and access speed. Among them, SRAM is located on the GPU chip and has extremely high access speed, but small capacity, which is suitable for storing small-scale data that is frequently accessed. HBM is a high-bandwidth off-chip memory with a large capacity but relatively slow access speed, which is suitable for storing large-scale data.

[0060] In addition, the reasoning process of large language models often involves a large number of self-attention mechanism operations, including generating query, key and value matrices, calculating attention scores, and normalizing them through the Softmax function to obtain attention weights, and finally performing weighted summation of the value matrix to generate output. Limited by the space limitations of SRAM, the conventional self-attention mechanism needs to frequently transfer data between SRAM and HBM during the calculation process, and frequent reading and writing of data from HBM will take a lot of time, especially when processing large-scale data.

[0061] Limited by the small capacity of SRAM, it is difficult to directly use SRAM for calculations when the data scale is large. Therefore, the relevant technology adopts a block strategy to divide the matrix to be calculated into blocks that can be stored in SRAM, so that all calculations are gradually completed in SRAM, so as to reduce the number of read and write interactions with HBM and reduce the time required for I / O operations.

[0062] Based on the block operation adopted in the above description, this embodiment provides the following implementation of step S3-3B: S3-3B-1, divide the attention mask matrix into a plurality of scoring sub-blocks of a preset size, and determine a shielding block from the plurality of scoring sub-blocks.

[0063] Each element in the shielded block is an unrelated element. For example, Fig.10 As shown, for Fig. 9 The attention mask matrix shown here assumes the preset size is , then you can Fig. 9 The lower triangular portion shown is divided into 10 sub-blocks, and the 10 sub-blocks include 3 shielding blocks.

[0064] Based on the above implementation of the introduction of the sub-block and the shielding block in step S3-3B-1, step S3-3B further includes: S3-3B-2, for each decoding layer, the query matrix mapped from the text sequence is divided into multiple query sub-blocks according to a preset size.

[0065] The size of each query sub-block satisfies the constraint relationship of matrix operation with the preset size. Fig.11 This process is explained more intuitively. Fig.11 As shown in the figure, the query matrix, key value matrix and attention weight matrix involved in the self-attention mechanism are shown. The dimension of the attention weight matrix is ​​the same as that of the attention mask matrix, and the elements correspond one to one. Assume that the size of each masked block in the attention mask matrix is , then the size of each corresponding block in the attention weight matrix is ​​also If you want to get Fig.11 The size of the attention weight matrix shown is The sub-block can be Fig.11 The query matrix in is divided into query sub-blocks, and adaptively divide the key-value matrix into sub-block.

[0066] Based on the description of the relationship between the query sub-block and the shielded block in the above embodiment, step S3-3B further includes: S3-3B-3, determine at least one target block that has no corresponding relationship with the shielding block from multiple query sub-blocks.

[0067] S3-3B-4, obtain an attention weight matrix based on at least one target block.

[0068] S3-3B-5, obtain the decoding result of the decoding layer according to the attention weight matrix.

[0069] So, just as Fig.11 The correspondence between the shielded blocks and the query sub-blocks is shown. This embodiment only operates on the target blocks in the multiple query sub-blocks that have no correspondence with the shielded blocks, thereby reducing invalid operations and improving the reasoning speed of the large language model.

[0070] In addition, it was found in practice that the construction process of the attention mask matrix is ​​closely related to the candidate word set and its size. With the increase in the candidate word set and the number of candidate words in each candidate word set, the dimension of the attention mask matrix will increase significantly, which directly leads to the large model requiring more computing resources during the reasoning process. However, through in-depth analysis of the candidate word set, it was found that there are a considerable proportion of redundant terms that conflict with the contextual semantics. Since these redundant terms not only increase the computational complexity, but may also affect the reasoning accuracy of the model, therefore, compared to directly using each of the multiple initial candidate words decoded directly from the decoding head as the candidate word set decoded from the information to be decoded, the candidate word set in this embodiment can also be obtained by screening the multiple initial candidates directly decoded. Therefore, in Figure 5During step S3, the server can also decode its own information to be decoded through each decoding head to obtain multiple initial word sets, wherein each initial word set includes multiple initial candidate words; then, at least part of the text fragments in the generated text are spliced ​​with the multiple initial word sets into a sequence to be screened; the sequence to be screened is processed by a pre-trained word screening model to determine the redundant words in each initial word set, wherein the redundant words in each initial word set represent the initial candidate words that conflict with the context semantics; finally, the redundant words in each initial word set are eliminated to obtain the candidate word set decoded from each information to be decoded.

[0071] For example, assuming Figure 6 The candidate word sets "going, very", "to, off, happy" shown are all initial word sets that have not been screened. If the server inputs the sequence to be screened after splicing "Take one basketball, we" with "going, very", "to, off, happy" into the word screening model for processing, and the output result shows that "happy" is a redundant word, it will be removed from "to, off, happy". Subsequently, the server uses the remaining "to, off" and "going, very" as the final candidate word set. In this way, by eliminating redundant words that conflict with the context semantics, not only the quality of the candidate word set is optimized, but also the amount of calculation required in the subsequent reasoning process is significantly reduced, thereby improving the overall efficiency of the model.

[0072] The above-mentioned word screening model can also be obtained by fine-tuning the training model using a model with fewer model parameters. For example, taking BERT as an example, the BERT model is used as the basic model, and a classification layer is added on its basis to output the probability of whether each candidate word is a redundant word. Then, a training data set is constructed, and each sample in the training data set includes a concatenated sequence of a generated text fragment and an initial candidate word set, and redundant terms that conflict with context semantics are marked (1 indicates redundancy, and 0 indicates non-redundancy). Finally, the concatenated text sequence is input into the BERT model to obtain the context representation of each candidate word, and the model is supervised and trained using the labeled data to obtain the above-mentioned word screening model.

[0073] For the large language model performance optimization method provided in this embodiment, a comparative test was also conducted on models with different parameter scales. Fig. 12A as well as Fig. 12BAs shown, based on the two open source models of Qwen2-7B and Qwen2-72B, Qwen2-7B and Qwen2-72B with sequence information introduced are represented in blue, and Qwen2-7B and Qwen2-72B without sequence information are represented in green. In addition, the horizontal axis in the figure represents the selected test data set, and the vertical axis represents the acceptance rate of the candidate words, which is defined as the average number of candidate words accepted per step. The test results show that after the introduction of sequence information, on average, 2.1~2.8 candidate words can be obtained per reasoning, and the acceptance rate has been significantly improved. It should be noted that the large language model performance optimization method provided in this embodiment can not only be used for the large language model of the Qwen2 series, but also can be applied to other large language models, for example, large language models such as llama and baichuan.

[0074] Based on the same inventive concept as the large language model performance optimization method provided in this embodiment, this embodiment also provides a large language model performance optimization device, and the large language model includes a main decoding head. It should be understood that the device includes at least one software function module that can be stored in a memory or solidified in an electronic device in the form of software. The processor in the electronic device is used to execute the executable module stored in the memory. For example, the software function module and computer program included in the device. Please refer to Fig.13 , functionally speaking, the device may include: The text feature module 11 is used to process the generated text through the large language model to obtain hidden state information transmitted to the main decoding head, wherein the large language model also includes a plurality of slave decoding heads in parallel with the main decoding head, and the preset arrangement order between the plurality of slave decoding heads represents the arrangement order between the decoding results; A feature optimization module 12, used to combine the sequence information of each slave decoding head with the hidden state information to obtain the information to be decoded of each slave decoding head; The feature decoding module 13 is used to obtain the best predicted text of the generated text according to the candidate word set decoded from each piece of information to be decoded.

[0075] In this embodiment, the text feature module 11 is used to implement Figure 5 In step S1, the feature optimization module 12 is used to implement Figure 5 In step S2, the feature decoding module 13 is used to implement Figure 5 Therefore, for the detailed description of the above modules, please refer to the specific implementation of the corresponding steps.

[0076] In addition, it should be understood that since the large language model performance optimization method provided in this embodiment has the same inventive concept, the large language model performance optimization device can also implement other steps or sub-steps of the method through the above modules.

[0077] Optionally, the feature decoding module 13 is further specifically used for: Combining the candidate words in each candidate word set in the order of arrangement to obtain multiple candidate texts; Determine the best predicted text from multiple candidate texts.

[0078] Optionally, the feature decoding module 13 is further specifically used for: splicing at least part of the text fragments in the generated text with each candidate text to obtain multiple texts to be scored; Scoring multiple texts to be scored respectively through a pre-trained text scoring model to obtain a score for each text to be scored; According to the score of each text to be scored, the best predicted text is determined.

[0079] Optionally, the branch where the main decoding head is located is called a trunk model of the large language model, and the trunk model includes a plurality of decoding layers connected in series; the feature decoding module 13 is further specifically used for: Each candidate word set is multiplied in sequence according to the arrangement order to obtain multiple subsequences to be verified, wherein the number of candidate word sets in each subsequence to be verified is equal to the number of candidate words in the previous subsequence to be verified; The generated text and multiple subsequences to be verified are concatenated into a complete text sequence, and an attention mask matrix is ​​constructed according to the combination order between the candidate words in the text sequence, wherein the rows and columns of the attention mask matrix correspond to the candidate words in the text sequence one by one, and the attention mask matrix includes unrelated elements marked with masks, and the candidate words associated with the unrelated elements do not have an adjacent combination relationship; The attention mask matrix is ​​used as the constraint condition of each decoding layer, and the feature information of the text sequence is processed in turn through the self-attention mechanism in each decoding layer; and the best predicted text is determined based on the decoding result of the last layer among multiple decoding layers.

[0080] Optionally, the feature decoding module 13 is further specifically used for: The attention mask matrix is ​​divided into a plurality of score sub-blocks of a preset size, and a shielding block is determined from the plurality of score sub-blocks, wherein each element in the shielding block is an unrelated element; For each decoding layer, the query matrix mapped from the text sequence is divided into a plurality of query sub-blocks according to a preset size, wherein the size of each query sub-block satisfies a constraint relationship of matrix operation with the preset size; Determine at least one target block that has no corresponding relationship with the shielding block from the plurality of query sub-blocks; According to at least one target block, an attention weight matrix is ​​obtained; According to the attention weight matrix, the decoding result of the decoding layer is obtained.

[0081] Optionally, before obtaining the best predicted text subsequent to the generated text according to the candidate word set decoded from each piece of information to be decoded, the feature decoding module 13 is further used to: By each slave decoding head decoding its own information to be decoded, a plurality of initial word sets are obtained, wherein each initial word set includes a plurality of initial candidate words; splicing at least part of the text segments in the generated text with a plurality of initial word sets into a sequence to be screened; The sequence to be screened is processed by a pre-trained word screening model to determine redundant words in each initial word set, wherein the redundant words in each initial word set represent initial candidate words that conflict with context semantics; Redundant words in each initial word set are removed to obtain a candidate word set for decoding each piece of information to be decoded.

[0082] Optionally, the feature optimization module 12 is further specifically used for: Mapping the arrangement position of each slave decoding head to the sequence information of each slave decoding head through a sine function or a cosine function; The sequence information of each slave decoding head is combined with the hidden state information to obtain the information to be decoded of each slave decoding head.

[0083] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0084] It should also be understood that if the above implementation is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0085] Therefore, this embodiment also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by the processor, the large language model performance optimization method provided in this embodiment is implemented. Among them, the storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, etc., which can store program codes.

[0086] This embodiment provides an electronic device for implementing a large language model performance optimization method. Fig.14 The electronic device may include a processor 22 and a memory 21. In addition, the memory 21 stores a computer program, and the processor implements the large language model performance optimization method provided in this embodiment by reading and executing the computer program corresponding to the above implementation in the memory 21.

[0087] Continue to see Fig.14 The electronic device further includes a communication unit 23. The memory 21, the processor 22 and the communication unit 23 are electrically connected to each other directly or indirectly through a system bus 24 to achieve data transmission or interaction.

[0088] The memory 21 may be an information recording device based on any electronic, magnetic, optical or other physical principle, used to record execution instructions, data, etc. In some embodiments, the memory 21 may be, but is not limited to, a volatile memory, a non-volatile memory, a storage drive, etc.

[0089] In some embodiments, the volatile memory may be a random access memory (RAM); in some embodiments, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), a flash memory, etc.; in some embodiments, the storage drive may be a disk drive, a solid-state drive, any type of storage disk (such as a CD, a DVD, etc.), or a similar storage medium, or a combination thereof, etc.

[0090] The communication unit 23 is used to send and receive data through a network. In some embodiments, the network may include a wired network, a wireless network, a fiber optic network, a telecommunication network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a wide area network (WAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, or a near field communication (NFC) network, or any combination thereof. In some embodiments, the network may include one or more network access points. For example, the network may include a wired or wireless network access point, such as a base station and / or a network switching node, and one or more components of the service request processing system may be connected to the network through the access point to exchange data and / or information.

[0091] The processor 22 may be an integrated circuit chip with signal processing capability, and the processor may include one or more processing cores (e.g., a single-core processor or a multi-core processor). By way of example only, the processor may include a central processing unit (CPU), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), or a microprocessor, or any combination thereof.

[0092] Understandably, Fig.14The structure shown is for illustration only. The electronic device may also have Fig.14 More or fewer components than shown, or with Fig.14 Different configurations are shown. Fig.14 The components shown may be implemented in hardware, software or a combination thereof.

[0093] It should be understood that the apparatus and method disclosed in the above-mentioned embodiments can also be implemented in other ways. The apparatus embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0094] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for optimizing the performance of a large language model, characterized in that: The large language model includes a main decoding head, and the method includes: Processing the generated text by means of a large language model to obtain hidden state information transmitted to the main decoding head, wherein the large language model further comprises a plurality of slave decoding heads in parallel with the main decoding head, and a preset arrangement order between the plurality of slave decoding heads represents an arrangement order between decoding results; Combining the sequence information of each of the slave decoding heads with the hidden state information to obtain information to be decoded of each of the slave decoding heads; According to the candidate word set decoded from each piece of information to be decoded, the best predicted text subsequent to the generated text is obtained.

2. The large language model performance optimization method according to claim 1, characterized in that: Obtaining the best predicted text subsequent to the generated text according to the candidate word set decoded from each piece of information to be decoded, including: Combining the candidate words in each of the candidate word sets according to the arrangement order to obtain a plurality of candidate texts; The best predicted text is determined from the multiple candidate texts.

3. The large language model performance optimization method according to claim 2, characterized in that: Determining the best predicted text from the multiple candidate texts includes: splicing at least part of the text segments in the generated text with each of the candidate texts to obtain a plurality of texts to be scored; Scoring the multiple texts to be scored respectively by using a pre-trained text scoring model to obtain a score for each of the texts to be scored; The best predicted text is determined according to the score of each of the to-be-scored texts.

4. The large language model performance optimization method according to claim 1, characterized in that: The branch where the main decoding head is located is called the trunk model of the large language model, and the trunk model includes a plurality of decoding layers connected in series; according to the candidate word set decoded from each piece of information to be decoded, the best predicted text subsequent to the generated text is obtained, including: Each of the candidate word sets is multiplied in sequence according to the arrangement order to obtain a plurality of subsequences to be verified, wherein the number of candidate word sets in each subsequence to be verified is equal to the number of candidate words in the previous subsequence to be verified; splicing the generated text and the multiple subsequences to be verified into a complete text sequence, and constructing an attention mask matrix according to the combination order between the candidate words in the text sequence, wherein the rows and columns of the attention mask matrix correspond to the candidate words in the text sequence one by one, respectively, and the attention mask matrix includes unrelated elements marked with masks, and the candidate words associated with the unrelated elements do not have an adjacent combination relationship; The attention mask matrix is ​​used as a constraint condition for each of the decoding layers, and the feature information of the text sequence is processed in turn through the self-attention mechanism in each of the decoding layers; and the optimal predicted text is determined based on the decoding result of the last layer among the multiple decoding layers.

5. The large language model performance optimization method according to claim 4, characterized in that: The attention mask matrix is ​​used as a constraint condition of each decoding layer, and the feature information of the text sequence is processed in turn through the self-attention mechanism in each decoding layer, including: Dividing the attention mask matrix into a plurality of score sub-blocks of a preset size, and determining a shielding block from the plurality of score sub-blocks, wherein each element in the shielding block is the unrelated element; For each of the decoding layers, the query matrix mapped from the text sequence is divided into a plurality of query sub-blocks according to the preset size, wherein the size of each query sub-block satisfies a constraint relationship of matrix operation with the preset size; Determine at least one target block from the plurality of query sub-blocks that has no corresponding relationship with the shielding block; Obtaining an attention weight matrix according to the at least one target block; According to the attention weight matrix, a decoding result of the decoding layer is obtained.

6. The large language model performance optimization method according to claim 1, characterized in that: Before obtaining the best predicted text subsequent to the generated text according to the candidate word set decoded from each piece of information to be decoded, the method further includes: By each of the slave decoding heads decoding its own information to be decoded, a plurality of initial word sets are obtained, wherein each of the initial word sets includes a plurality of initial candidate words; splicing at least part of the text segments in the generated text with the multiple initial word sets into a sequence to be screened; Processing the sequence to be screened by a pre-trained word screening model to determine redundant words in each of the initial word sets, wherein the redundant words in each of the initial word sets represent initial candidate words that conflict with context semantics; Redundant words in each of the initial word sets are removed to obtain a candidate word set decoded from each piece of information to be decoded.

7. The large language model performance optimization method according to claim 1, characterized in that: Combining the sequence information of each of the slave decoding heads with the hidden state information to obtain information to be decoded from each of the slave decoding heads, including: Mapping the arrangement position of each of the slave decoding heads to the sequence information of each of the slave decoding heads through a sine function or a cosine function; The sequence information of each of the slave decoding heads is combined with the hidden state information to obtain the information to be decoded of each of the slave decoding heads.

8. A large language model performance optimization device, characterized in that: The large language model includes a main decoding head, and the device includes: a text feature module, configured to process the generated text through a large language model to obtain hidden state information transmitted to the main decoding head, wherein the large language model further comprises a plurality of slave decoding heads in parallel with the main decoding head, and a preset arrangement order between the plurality of slave decoding heads represents an arrangement order between decoding results; A feature optimization module, used for combining the sequence information of each of the slave decoding heads with the hidden state information to obtain information to be decoded of each of the slave decoding heads; The feature decoding module is used to obtain the best predicted text subsequent to the generated text based on the candidate word set decoded from each piece of information to be decoded.

9. A storage medium, characterized in that: The storage medium stores a computer program, and the computer program, when executed by a processor, implements the large language model performance optimization method described in any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory stores a computer program, and the computer program, when executed by the processor, implements the large language model performance optimization method according to any one of claims 1 to 7.

Images