Inference acceleration method and device for large model, and electronic equipment

Abstract

The invention provides an inference acceleration method and device for a large model, electronic equipment and a readable storage medium, and relates to the technical field of artificial intelligence such as large models, deep learning and natural language processing. The reasoning acceleration method for the large model comprises the steps that after a to-be-processed source text is input into a target large model, a top layer hidden state, used for predicting a next token lexical element, of the target large model is obtained; acquiring action decision information corresponding to the next token according to the top layer hiding state; obtaining a text copying interval corresponding to the next token according to the top layer hidden state in response to the fact that the action decision information is determined to be a copying action; copying the text in the text copying interval in the to-be-processed source text, and taking a copying result as the next token. According to the method, the reasoning speed and reasoning efficiency of the large model can be improved, and redundant calculation of the large model during reasoning can be reduced, so that precious calculation resources are greatly saved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to the fields of artificial intelligence technologies such as large models, deep learning, and natural language processing. It provides a method, apparatus, electronic device, and readable storage medium for accelerating inference in large models. Background Technology

[0002] Large models, especially large language models, have become a core technology driving the development of artificial intelligence and have been widely used in various industries. In practical applications, the generation of large models is not entirely "created from scratch," but rather involves a common "copying" phenomenon, meaning that the content generated by large models often includes text fragments that can be directly copied from the source text.

[0003] However, existing large models cannot recognize these "copyable" text fragments, causing them to still use a token-by-token generation method to "recreate" text fragments that already exist in the source text. This not only leads to a large amount of redundant computation in large models, thus seriously wasting valuable computing resources, but also reduces the speed of large models when performing inference. Summary of the Invention

[0004] According to a first aspect of this disclosure, a method for accelerating inference in a large model is provided, comprising: after inputting source text to be processed into a target large model, obtaining a top-level hidden state of the target large model for predicting the next token lexicon; obtaining action decision information corresponding to the next token based on the top-level hidden state; in response to determining that the action decision information is a copy action, obtaining a text copying interval corresponding to the next token based on the top-level hidden state; copying the text in the source text to be processed that is located within the text copying interval, and using the copying result as the next token.

[0005] According to a second aspect of this disclosure, an inference acceleration device for a large model is provided, comprising: an acquisition unit, configured to acquire, after inputting a source text to be processed into a target large model, a top-level hidden state of the target large model for predicting the next token; a decision unit, configured to acquire action decision information corresponding to the next token based on the top-level hidden state; a processing unit, configured to, in response to determining that the action decision information is a copy action, acquire a text copying interval corresponding to the next token based on the top-level hidden state; and a copying unit, configured to copy the text in the source text to be processed that is located within the text copying interval, and use the copying result as the next token.

[0006] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0007] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described above.

[0008] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described above.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0012] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;

[0013] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure;

[0014] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0015] Figure 5 This is a schematic diagram according to the fifth embodiment of the present disclosure;

[0016] Figure 6 This is a block diagram of an electronic device used to implement the inference acceleration method for large models according to embodiments of the present disclosure. Detailed Implementation

[0017] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and mechanisms are omitted in the following description.

[0018] Figure 1 This is a schematic diagram based on the first embodiment of this disclosure. (See diagram below.) Figure 1 As shown, the inference acceleration method for large models in this embodiment specifically includes the following steps:

[0019] S101. After inputting the source text to be processed into the target large model, obtain the top-level hidden state of the target large model for predicting the next token word.

[0020] S102. Based on the top-level hidden state, obtain the action decision information corresponding to the next token;

[0021] S103. In response to determining that the action decision information is a copy action, obtain the text copy range corresponding to the next token according to the top-level hidden state;

[0022] S104. Copy the text within the text copying range in the source text to be processed, and use the copying result as the next token.

[0023] The inference acceleration method for large models in this embodiment first obtains the action decision information corresponding to the next token based on the top-level hidden state used to predict the next token before the target large model predicts the next token based on the input source text. Then, if the action decision information is determined to be a copy action, the text copying interval corresponding to the next token is obtained based on the top-level hidden state used to predict the next token. Finally, the text copied from the source text corresponding to the text copying interval is used as the next token predicted by the target large model. This embodiment obtains the action decision information and text copying interval based on the top-level hidden state of the next token, enabling the target large model to obtain the next token by copying the text in the source text. This avoids the problems of wasted computing resources and low inference efficiency that occur when the target large model "recreates" the text that already exists in the source text as the next token. It can improve the inference speed and efficiency of large models and reduce redundant calculations during inference, thereby greatly saving valuable computing resources.

[0024] In this embodiment, the target large model can be a large language model (LLM), which refers to a large-scale neural network model based on deep learning technology, specifically used to process and generate natural language text; the target large model in this embodiment can also be a multimodal large model.

[0025] In this embodiment, a token is the basic unit of text generated by the large model through prediction; a token can be a character, a word, or a phrase, or it can be a sub-word (i.e., part of a word).

[0026] In this embodiment, the top-layer hidden state is the vector representation output by the decoder of the last layer (i.e., the top layer) when the target large model (e.g., a large language model) processes the input data, and is used to predict the next token.

[0027] In this embodiment, when executing S101, the source text to be processed is first obtained, then the obtained source text to be processed is input into the target large model, and finally the top-level hidden state output by the target large model for predicting the next token is obtained during the processing of the source text to be processed.

[0028] In this embodiment, the source text to be processed can be news text, report text, etc., and the target large model is used to process the source text to obtain the corresponding summary text. In this embodiment, the source text to be processed can also be a document retrieved based on question-and-answer text, such as financial report documents, knowledge documents, etc., and the target large model is used to process the source text to obtain the corresponding answer text. In this embodiment, the source text to be processed can also be structured data in fields such as sports events, weather forecasts, and medical records, and the target large model is used to process the source text to obtain the corresponding report text. In this embodiment, the source text to be processed can also be code to be completed, and the target large model is used to process the source text to obtain the corresponding complete code.

[0029] In this embodiment, when executing S101, for the first token predicted by the target large model, the top-level hidden state of the token is obtained by the target large model based on the source text to be processed; for tokens not predicted by the target large model, the top-level hidden state of the token is obtained by the target large model based on the source text to be processed and the already predicted tokens.

[0030] In this embodiment, after executing S101 to obtain the top-level hidden state of the target large model for predicting the next token, S102 is executed to obtain the action decision information of the corresponding next token based on the obtained top-level hidden state.

[0031] In existing technologies, after obtaining the top-level hidden state used to predict the next token, the large model usually directly inputs the top-level hidden state into the language model head (LM head) of the large model, so that the language model head can generate the corresponding content based on the top-level hidden state, thereby completing the prediction of the next token.

[0032] However, in real-world applications, the generation task of large models is not entirely "created from scratch," but rather exhibits a common "copying" phenomenon. This means that the generated content of large models contains a large number of text fragments that can be directly copied from the source text to be processed (such as context, dialogue history, etc.). If these "copyable" text fragments cannot be identified, the large model will "recreate" these text fragments that already exist in the source text to be processed, resulting in a large amount of redundant computation, seriously wasting valuable computing resources, and failing to fully utilize contextual information to improve token generation efficiency.

[0033] Therefore, in order to avoid redundant calculations and improve the generation efficiency of tokens, this embodiment executes S102 after obtaining the top-level hidden state of the target large model for predicting the next token, so as to obtain the action decision information of the corresponding next token based on the top-level hidden state.

[0034] In this embodiment, the action decision information obtained in S102 is one of the copy action and the generation action; wherein, the copy action is used to instruct the target large model to obtain the prediction result of the next token through the "copy" operation, and the generation action is used to instruct the target large model to obtain the prediction result of the next token through the "generation" operation.

[0035] Specifically, in this embodiment, when executing S102 to obtain the action decision information of the next token based on the top-level hidden state, the implementation method can be as follows: input the obtained top-level hidden state into the decision prediction head of the target large model, that is, the target large model in this embodiment includes a decision prediction head for obtaining action decision information; and obtain the action decision information of the next token based on the output result of the decision prediction head.

[0036] In other words, this embodiment obtains the action decision information for the next token by using the decision prediction head included in the target large model. The decision prediction head is obtained through pre-training and can output the corresponding action decision information based on the input top-level hidden state. Therefore, this embodiment uses the decision prediction head located in the target large model, which can improve the efficiency and accuracy of obtaining action decision information.

[0037] In this embodiment, after executing S102 to obtain the action decision information for the next token, S103 is executed in response to determining that the action decision information is a copy action, and the text copying range for the next token is obtained according to the top-level hidden state.

[0038] Specifically, in this embodiment, when executing S103 to obtain the text copying interval corresponding to the next token based on the top-level hidden state, the implementation method can be as follows: input the obtained top-level hidden state into the starting prediction head of the target large model, and obtain the copying starting position of the next token based on the output result of the starting prediction head. That is, the target large model in this embodiment includes a starting prediction head for obtaining the copying starting position; input the obtained top-level hidden state into the ending prediction head of the target large model, and obtain the copying ending position of the next token based on the output result of the ending prediction head. That is, the target large model in this embodiment includes an ending prediction head for obtaining the copying ending position; obtain the text copying interval corresponding to the next token based on the obtained copying starting position and copying ending position.

[0039] In other words, this embodiment obtains the text copying interval of the next token by using the start-point prediction head and the end-point prediction head included in the target large model. The start-point prediction head and the end-point prediction head are obtained through pre-training and can output the copying start position and copying end position respectively according to the input top-level hidden state. Therefore, this embodiment uses the start-point prediction head and the end-point prediction head located in the target large model, which can improve the efficiency and accuracy of obtaining the text copying interval.

[0040] In addition, when executing S103, this embodiment may also include the following: in response to determining that the action decision information is a generation action, the obtained top-level hidden state is input into the language model head of the target large model; and the next token is obtained based on the output of the language model head.

[0041] In other words, after determining that the obtained action decision information is a generation action, this embodiment uses the existing token generation method, that is, the language model head generates the next token in real time based on the top-level hidden state of the next token.

[0042] Therefore, this embodiment determines whether to obtain the next token by "copying" or "generating" based on the action decision information corresponding to the next token, effectively avoiding the problem of using the generation method to obtain the next token in all cases, and improving the flexibility and efficiency of obtaining the next token.

[0043] In this embodiment, after executing S103 to obtain the text copying range corresponding to the next token, S104 is executed to copy the text in the source text to be processed that is located within the text copying range, and the copying result is used as the next token.

[0044] In other words, after obtaining the text copying interval of the next token in S103, this embodiment can copy specific text content in the source text to be processed according to the obtained text copying interval, and use the copying result as the next token to be predicted by the target large model. Since text copying has a faster acquisition speed than text generation and does not require redundant calculation, this embodiment can greatly improve the inference speed of the target large model and effectively save the computing resources required by the target large model when performing inference.

[0045] It is understandable that after the target large model in this embodiment obtains the source text to be processed, it can use a tokenizer to convert the source text into a sequence of sub-words. The resulting sequence of sub-words includes multiple sub-words and the position information of each sub-word.

[0046] Therefore, in this embodiment, when executing S104 to copy the text located within the text copying interval in the source text to be processed and using the copying result as the next token, the implementation method can be as follows: obtain the sub-word sequence of the source text to be processed; determine the sub-words located within the text copying interval in the sub-word sequence according to the position information of each sub-word in the sub-word sequence; copy the determined sub-words and use the copying result as the next token.

[0047] In other words, this embodiment copies specific text content from the source text to be processed based on the obtained text copying interval, and uses the copying result as the next token to be predicted by the target large model. This eliminates the need for the target large model to predict the next token through generation, effectively improving the inference speed of the target large model (i.e., the speed of predicting the next token). Furthermore, by copying the next token from the source text to be processed, the problem of "illusion" in the large model can be avoided, thereby improving the accuracy of the obtained token.

[0048] Figure 2 This is a schematic diagram according to the second embodiment of this disclosure. (See diagram below.) Figure 2As shown in the figure, this embodiment illustrates a structural diagram of a target large model; in this embodiment, the target large model is a span pointer large model (SpanPointerLlama) that is extended based on standard large language models (such as open source large language models like Llama).

[0049] The target large model in this embodiment includes a decoder module (including multiple decoder blocks), a language model head, a decision prediction head, a start prediction head, and an end prediction head; wherein, the top-level hidden state is the hidden state output by the last decoder block in the decoder module.

[0050] In this embodiment, after inputting the source text to be processed into the target large model, the decoder module obtains the top-level hidden state of the target large model used to predict the next token. The obtained top-level hidden state is input into the decision prediction head to obtain the action decision information output by the decision prediction head. If the action decision information is a copy action, the top-level hidden state is input into the start-point prediction head and the end-point prediction head. Then, based on the copy start position and copy end position output by the start-point prediction head and the end-point prediction head respectively, the next token is copied from the text to be processed. If the action decision information is a generation action, the top-level hidden state is input into the language model head. Then, the language model head generates the next token based on the top-level hidden state. After the prediction of the next token is completed, the above steps are repeated to predict subsequent tokens.

[0051] The target large model provided in this embodiment can be applied to summary generation systems, retrieval-enhanced question-answering systems, data-driven text generation systems, code generation and completion systems, etc.

[0052] For example, existing technologies may incorrectly cite names of people, places, companies, or other key data when generating a news summary. However, by using the target big model provided in this embodiment, the decision prediction head identifies these entity information and data as key content that needs to be accurately paraphrased, thereby activating the "copy" mode. The "start point / end point prediction head" directly locates and completely copies the information from the original text. For connecting sentences or paragraphs that require summarization, the system switches back to the "generate" mode to produce fluent text. This ensures that the generated summary is not only highly readable but also contains absolutely accurate factual information.

[0053] For example, when answering user queries, existing technologies often use retrieval-enhanced generation systems to retrieve documents related to the query. Traditional generation models, after reading these documents, may reorganize the answer using their own language, thus introducing bias. However, using the target large model provided in this embodiment, the most core sentences can be extracted directly from the retrieved documents through a "copy" mode to construct a complete answer, greatly improving the reliability of the question-answering system when handling structured, data-intensive questions.

[0054] Figure 3 This is a schematic diagram according to the third embodiment of this disclosure. (See diagram below.) Figure 3 As shown in the figure, this embodiment illustrates the training process of the target large model, which specifically includes the following steps:

[0055] S301. Obtain training data, wherein the training data includes sample source text and the corresponding labeled action sequence of sample target text;

[0056] S302. Construct an initial large model including a decoder module, a language model head, a decision prediction head, a start prediction head, and an end prediction head. The decision prediction head is used to output prediction action decision information based on the top-level hidden state output by the decoder module. The start prediction head is used to output the predicted copy start position based on the top-level hidden state. The end prediction head is used to output the predicted copy end position based on the top-level hidden state.

[0057] S303. Input the sample source text into the initial large model, and obtain the prediction action sequence of the corresponding prediction target text based on the output of the initial large model;

[0058] S304. Calculate the target loss function value based on the labeled action sequence and the predicted action sequence, and use the target loss function value to adjust the parameters of the decision prediction head, the starting point prediction head, and the ending point prediction head to obtain the target large model.

[0059] In other words, this embodiment obtains an initial large model by adding a decision prediction head, a start-point prediction head, and an end-point prediction head to an existing large model. Then, based on the labeled action sequence and the prediction action sequence obtained by the initial large model from the sample source text, the parameters of the three newly added prediction heads in the initial large model are adjusted to obtain the target large model. Since this embodiment adds three additional prediction heads to the large model, it integrates the two operations of "generation" and "copying" into a unified, end-to-end trainable neural network framework, enabling it to be quickly adapted to any existing large model with extremely low computational cost.

[0060] In the training data obtained by executing S301 in this embodiment, the target text of the sample corresponding to the action sequence is the target text of the corresponding source text. For example, if the source text is a news article, then the target text is the summary of the news article.

[0061] In this embodiment, the sequence of annotation actions in the training data includes multiple annotation actions. Each annotation action includes annotation action decision information, annotation copy start position and annotation copy end position. Different annotation actions correspond to different tokens (e.g., sub-words) in the sample target text.

[0062] For example, if the target text of the sample includes token1, token2, token3, and token4, then the annotation action sequence corresponding to the target text of the sample includes the annotation action corresponding to token1, the annotation action corresponding to token2, the annotation action corresponding to token3, and the annotation action corresponding to token4; among which, the annotation action corresponding to token1 can be ("copy", i start i end "copy" indicates that the annotation action decision information for token1 is copy, i start with i end These represent the starting and ending positions of the annotation for token1 in the sample source text, respectively. The annotation action for token2 can be ("generate", empty). "generate" indicates that the annotation action decision information for token2 is "generate", and "empty" indicates that token2 is not located in the sample source text.

[0063] In this embodiment, after executing S301 to obtain training data, S302 is executed to construct an initial large model including a decoder module, a language model head, a decision prediction head, a start prediction head, and an end prediction head.

[0064] In this embodiment, the decoder module is used to obtain the top-level hidden state used when predicting each token; the decoder module includes multiple decoder blocks, and the top-level hidden state is the hidden state output by the last decoder block in the decoder module.

[0065] In this embodiment, the decision prediction head is used to output prediction action decision information corresponding to each token based on the top-level hidden state output by the decoder module each time. The prediction action decision information includes one of the copy action and the generation action.

[0066] Specifically, in this embodiment, when the decision prediction head outputs prediction action decision information based on the top-level hidden state, it can first perform a linear transformation on the top-level hidden state to obtain a two-dimensional vector (e.g., logits). gateThen, based on the obtained two-dimensional vector, the probability of copying action and the probability of generating action are obtained. Finally, based on the two probabilities obtained, the prediction action decision information is obtained (for example, the action with the higher probability is used as the prediction action decision information).

[0067] In this embodiment, the starting point prediction head is activated when the prediction action information output by the decision prediction head is "copy action". It is used to output the predicted copy starting point position according to the top-level hidden state output by the decoder module. The predicted starting point position is used to represent the starting point position when copying the corresponding token from the sample source text.

[0068] Specifically, in this embodiment, when the starting point prediction head predicts the copy starting point position based on the top-level hidden state, it can first perform a linear transformation on the top-level hidden state to obtain a two-dimensional vector (e.g., logits). start Then, based on the obtained two-dimensional vector, the probability distribution of different positions in the sample source text as the starting position of the corresponding token is obtained. Finally, the predicted copy starting position is obtained based on the obtained probability distribution (for example, the starting position with the highest probability is used as the predicted copy starting position).

[0069] In this embodiment, the endpoint prediction head is activated when the prediction action information output by the decision prediction head is "copy action". It is used to output the predicted copy endpoint position according to the top-level hidden state output by the decoder module. The predicted endpoint position is used to represent the endpoint position when copying the corresponding token from the sample source text.

[0070] Specifically, in this embodiment, when the endpoint prediction head predicts and replicates the endpoint position based on the output of the top-level hidden state, it can first perform a linear transformation on the top-level hidden state to obtain a two-dimensional vector (e.g., logits). end Then, based on the obtained two-dimensional vector, the probability distribution of different positions in the sample source text as the endpoint positions of the corresponding tokens is obtained. Finally, the predicted replication endpoint position is obtained based on the obtained probability distribution (for example, the endpoint position with the highest probability is used as the predicted replication start position).

[0071] In this embodiment, the language model head is activated when the predicted action information output by the decision prediction head is "generate action", and is used to generate the corresponding token based on the top-level hidden state output by the decoder module.

[0072] In this embodiment, after completing the construction of the initial large model in S302, the sample source text is input into the initial large model in S303. Based on the output of the initial large model, the prediction action sequence of the corresponding prediction target text is obtained.

[0073] In this embodiment, the predicted action sequence obtained from the output of the initial large model includes multiple predicted actions. Each predicted action includes the predicted action decision information output by the decision prediction head, the predicted copy start position output by the start prediction head, and the predicted copy end position output by the end prediction head. Different predicted actions correspond to different tokens (e.g., sub-words) in the predicted target text.

[0074] In this embodiment, the predicted target text is the target text generated by the initial large model based on the input sample source text. Since the initial large model in this embodiment does not modify the decoder module and the language model head, the predicted target text output by the initial large model is consistent with the sample target text of the corresponding sample source text.

[0075] In this embodiment, after executing S303 to obtain the prediction action sequence of the corresponding prediction target text, S304 is executed to calculate the target loss function value based on the labeled action sequence and the prediction action sequence. The target loss function value is used to adjust the parameters of the decision prediction head, the starting prediction head, and the ending prediction head to obtain the target large model.

[0076] Specifically, in this embodiment, when executing S304 to calculate the target loss function value based on the labeled action sequence and the predicted action sequence, the following implementation method can be adopted: Calculate a first loss function value based on the labeled action decision information and the predicted action decision information corresponding to the same token in the action sequence; the first loss function value is used to supervise the decision prediction head; calculate a second loss function value based on the labeled copy start position and the predicted copy start position corresponding to the same token in the action sequence; the second loss function value is used to supervise the start prediction head; calculate a third loss function value based on the labeled copy end position and the predicted copy end position corresponding to the same token in the action sequence; and obtain the target loss function value based on the obtained first loss function value, second loss function value, and third loss function value, for example, by adding the three loss function values ​​together as the target loss function value.

[0077] In other words, this embodiment obtains the target loss function value for adjusting the three prediction heads based on the action decision information, the copy start position, and the copy end position included in the action sequence. This can improve the accuracy of the obtained target loss function value, thereby improving the accuracy when adjusting the parameters of the three prediction heads.

[0078] In addition, when executing S304 in this embodiment, the LoRA (Low-Rank Adaptation) method can be used to fine-tune the initial large model. That is, the main parameters of the initial large model are frozen, and only the three newly added prediction heads and the introduced low-rank decomposition matrix are trained. This can significantly reduce the computational resources and storage costs required to fine-tune the large model, so that the initial large model can be trained on consumer-grade hardware and easily applied to large language models of different sizes.

[0079] Figure 4 This is a schematic diagram according to the fourth embodiment of this disclosure. (See diagram below.) Figure 4 As shown in the figure, this embodiment illustrates the process of acquiring training data, which specifically includes the following steps:

[0080] S401. Obtain the sample source text and its corresponding sample target text, and obtain the source word sequence of the sample source text and the target word sequence of the sample target text respectively;

[0081] S402. Construct an N-gram index corresponding to the sample source text based on the source word sequence;

[0082] S403. Query the N-gram index for each of the multiple target sub-words in the target sub-word sequence, and obtain the annotation action for each target sub-word based on the query results;

[0083] S404. Obtain the annotation action sequence based on the annotation actions of the multiple target sub-words, and obtain the training data based on the annotation action sequence and the sample source text.

[0084] In other words, this embodiment uses existing text pairs (i.e., including source text and its corresponding target text) to automatically acquire training data without any manual annotation cost. It can automatically acquire the labeled action sequence as training data, thereby improving the efficiency of training data acquisition and reducing the cost of training data acquisition.

[0085] In this embodiment, when executing S401, a preset word segmenter (e.g., a word segmenter corresponding to the target large model) can be used to convert the sample source text into a source word sequence and the sample target text into a target word sequence; for example, the source word sequence can be S tok ={S1, S2, ... S M The target subword sequence can be T. tok ={T1, T2, ..., T N}, where M and N are the word lengths of the source text (S) and target text (T), respectively.

[0086] In this embodiment, the N-gram index of the corresponding sample source text constructed by executing S402 includes multiple source word fragments and the start and end positions of each source word fragment in the sample source text.

[0087] In this embodiment, each source word segment in the N-gram index consists of N consecutive source words; it can be understood that each source word segment in this embodiment may also consist of more than N consecutive source words.

[0088] In this embodiment, when performing S403 to query multiple target subwords in the target subword sequence in the N-gram index and obtain the annotation action for each target subword based on the query results, the following implementation method can be adopted: query the N-gram index based on the current target subword; in response to determining that a target source subword fragment matching the current target subword is found in the N-gram index, use the copy action as the annotation action decision information for the current target subword, use the starting position of the target source subword fragment in the sample source text as the annotation copy starting position of the current target subword, and use the ending position of the target subword fragment in the sample source text as the annotation copy ending position of the current target subword.

[0089] When executing S403, this embodiment may also include the following: in response to determining that no target source word fragment matching the current target word is found in the N-gram index, the action to be generated is used as the annotation action decision information for the current target word.

[0090] In other words, this embodiment sequentially queries the target subwords in the target subword sequence in the constructed N-gram index, obtains the annotation action corresponding to each target subword in the target subword sequence based on the query results, and then obtains the annotation action sequence for model training based on the obtained annotation actions, which can improve the efficiency of obtaining the annotation action sequence and reduce the cost of obtaining the annotation action sequence.

[0091] In addition, when executing S403 in this embodiment, a query can be performed in the N-gram index based on the current target word and the target word fragment composed of N-1 target words following the current target word. If a target source word fragment corresponding to the target word fragment exists, the annotation action of the current target word is obtained based on the "copy action" and the position information of the corresponding target source word fragment. Then, after moving forward N units in the target word sequence, the next query is performed.

[0092] In this embodiment, if the target source word fragment corresponding to the target word fragment cannot be found in the N-gram index when executing S403, the target word sequence is moved forward by 1 unit, and the query continues for the target word located after the current target word.

[0093] Figure 5 This is a schematic diagram according to the fifth embodiment of this disclosure. (See diagram below.) Figure 5 As shown, the inference acceleration device 500 for large models in this embodiment includes:

[0094] The acquisition unit 501 is used to acquire the top-level hidden state of the target large model for predicting the next token word after inputting the source text to be processed into the target large model;

[0095] Decision unit 502 is used to obtain action decision information corresponding to the next token based on the top-level hidden state;

[0096] Processing unit 503 is configured to, in response to determining that the action decision information is a copy action, obtain the text copying range corresponding to the next token based on the top-level hidden state;

[0097] The copying unit 504 is used to copy the text located within the text copying interval in the source text to be processed, and use the copying result as the next token.

[0098] The acquisition unit 501 can first acquire the source text to be processed, then input the acquired source text to be processed into the target large model, and finally acquire the top-level hidden state output by the target large model for predicting the next token during the processing of the source text to be processed.

[0099] For the first token predicted by the target large model, the top-level hidden state obtained by the acquisition unit 501 for predicting the token is obtained by the target large model based on the source text to be processed; for tokens not predicted by the target large model, the top-level hidden state obtained by the acquisition unit 501 for predicting the token is obtained by the target large model based on the source text to be processed and the already predicted tokens.

[0100] In this embodiment, after the acquisition unit 501 acquires the top-level hidden state of the target large model for predicting the next token, the decision unit 502 acquires the action decision information of the corresponding next token based on the acquired top-level hidden state.

[0101] To avoid redundant calculations and improve token generation efficiency, in this embodiment, after obtaining the top-level hidden state of the target large model for predicting the next token, the decision unit 502 obtains the action decision information of the corresponding next token based on the top-level hidden state.

[0102] The action decision information acquired by decision unit 502 is one of copy action and generation action; wherein, copy action is used to instruct the target large model to obtain the prediction result of the next token through "copy" operation, and generation action is used to instruct the target large model to obtain the prediction result of the next token through "generation" operation.

[0103] Specifically, when the decision unit 502 obtains the action decision information of the next token based on the top-level hidden state, the implementation method can be as follows: input the obtained top-level hidden state into the decision prediction head of the target large model, that is, the target large model in this embodiment includes a decision prediction head for obtaining action decision information; and obtain the action decision information of the next token based on the output result of the decision prediction head.

[0104] In other words, the decision unit 502 obtains the action decision information for the next token through the decision prediction head included in the target large model. The decision prediction head is obtained through pre-training and can output the corresponding action decision information based on the input top-level hidden state. Therefore, this embodiment uses the decision prediction head located in the target large model, which can improve the efficiency and accuracy of obtaining action decision information.

[0105] In this embodiment, after the decision unit 502 obtains the action decision information corresponding to the next token, the processing unit 503, in response to determining that the action decision information is a copy action, obtains the text copying range corresponding to the next token based on the top-level hidden state.

[0106] Specifically, when processing unit 503 obtains the text copying interval corresponding to the next token based on the top-level hidden state, the following implementation method can be adopted: inputting the obtained top-level hidden state into the starting prediction head of the target large model, and obtaining the copying starting position of the next token based on the output result of the starting prediction head, that is, the target large model in this embodiment includes a starting prediction head for obtaining the copying starting position; inputting the obtained top-level hidden state into the ending prediction head of the target large model, and obtaining the copying ending position of the next token based on the output result of the ending prediction head, that is, the target large model in this embodiment includes an ending prediction head for obtaining the copying ending position; and obtaining the text copying interval corresponding to the next token based on the obtained copying starting position and copying ending position.

[0107] In other words, the processing unit 503 obtains the text copying interval of the next token by using the start prediction head and end prediction head included in the target large model. The start prediction head and end prediction head are obtained through pre-training and can output the copying start position and copying end position respectively according to the input top-level hidden state. Therefore, this embodiment uses the start prediction head and end prediction head located in the target large model, which can improve the efficiency and accuracy of obtaining the text copying interval.

[0108] In addition, the processing unit 503 can also perform the following: in response to determining that the action decision information is to generate an action, it inputs the acquired top-level hidden state into the language model head of the target large model; and obtains the next token based on the output of the language model head.

[0109] In other words, after determining that the acquired action decision information is a generation action, the processing unit 503 uses the existing token generation method, that is, the language model head generates the next token in real time based on the top-level hidden state of the next token.

[0110] Therefore, this embodiment determines whether to obtain the next token by "copying" or "generating" based on the action decision information corresponding to the next token, effectively avoiding the problem of using the generation method to obtain the next token in all cases, and improving the flexibility and efficiency of obtaining the next token.

[0111] In this embodiment, after the processing unit 503 obtains the text copying range corresponding to the next token, the copying unit 504 copies the text in the source text to be processed that is located within the text copying range, and uses the copying result as the next token.

[0112] In other words, after the processing unit 503 obtains the text copying interval corresponding to the next token, the copying unit 504 can copy specific text content in the source text to be processed according to the obtained text copying interval, and use the copying result as the next token to be predicted by the target large model. Since text copying has a faster acquisition speed and does not require redundant calculation compared with text generation, this embodiment can greatly improve the inference speed of the target large model and effectively save the computing resources required by the target large model when performing inference.

[0113] It is understandable that after the target large model in this embodiment obtains the source text to be processed, it can use a tokenizer to convert the source text into a sequence of sub-words. The resulting sequence of sub-words includes multiple sub-words and the position information of each sub-word.

[0114] Therefore, when copying text within the text copying interval in the source text to be processed and using the copying result as the next token, the copying unit 504 can adopt the following implementation method: obtain the sub-word sequence of the source text to be processed; determine the sub-words within the text copying interval in the sub-word sequence based on the position information of each sub-word in the sub-word sequence; copy the determined sub-words and use the copying result as the next token.

[0115] In other words, the copying unit 504 copies specific text content in the source text to be processed based on the acquired text copying range, and uses the copying result as the next token to be predicted by the target large model. This eliminates the need for the target large model to predict the next token through generation, effectively improving the inference speed of the target large model (i.e., the speed of predicting the next token). Furthermore, by copying the next token from the source text to be processed, the problem of "illusion" in the large model can be avoided, thereby improving the accuracy of the obtained token.

[0116] The inference acceleration device 500 for large models in this embodiment may further include a training unit 505, used to train the target large model in the following manner: acquiring training data, which includes sample source text and labeled action sequences of the corresponding sample target text; constructing an initial large model including a decoder module, a language model head, a decision prediction head, a start prediction head, and an end prediction head, wherein the decision prediction head is used to output prediction action decision information based on the top-level hidden state output by the decoder module, the start prediction head is used to output the predicted copy start position based on the top-level hidden state, and the end prediction head is used to output the predicted copy end position based on the top-level hidden state; inputting the sample source text into the initial large model, and obtaining the predicted action sequence of the corresponding predicted target text based on the output of the initial large model; calculating the target loss function value based on the labeled action sequence and the predicted action sequence, and adjusting the parameters of the decision prediction head, the start prediction head, and the end prediction head using the target loss function value to obtain the target large model.

[0117] In other words, the training unit 505 obtains an initial large model by adding a decision prediction head, a start prediction head, and an end prediction head to the existing large model. Then, based on the labeled action sequence and the predicted action sequence obtained by the initial large model from the sample source text, the parameters of the three newly added prediction heads in the initial large model are adjusted to obtain the target large model. Since this embodiment adds three additional prediction heads to the large model, this embodiment integrates the two operations of "generation" and "copying" into a unified, end-to-end trainable neural network framework, which enables it to be quickly adapted to any existing large model with extremely low computational cost.

[0118] In the training data acquired by training unit 505, the target text of the sample corresponding to the action sequence is the target text of the corresponding source text. For example, if the source text is a news article, then the target text is the summary of the news article.

[0119] In this embodiment, the sequence of annotation actions in the training data includes multiple annotation actions. Each annotation action includes annotation action decision information, annotation copy start position and annotation copy end position. Different annotation actions correspond to different tokens (e.g., sub-words) in the sample target text.

[0120] In this embodiment, the predicted action sequence obtained from the output of the initial large model includes multiple predicted actions. Each predicted action includes the predicted action decision information output by the decision prediction head, the predicted copy start position output by the start prediction head, and the predicted copy end position output by the end prediction head. Different predicted actions correspond to different tokens (e.g., sub-words) in the predicted target text.

[0121] In this embodiment, the predicted target text is the target text generated by the initial large model based on the input sample source text. Since the initial large model in this embodiment does not modify the decoder module and the language model head, the predicted target text output by the initial large model is consistent with the sample target text of the corresponding sample source text.

[0122] Specifically, when the training unit 505 calculates the target loss function value based on the labeled action sequence and the predicted action sequence, it can be implemented as follows: A first loss function value is calculated based on the labeled action decision information and the predicted action decision information corresponding to the same token in the action sequence; this first loss function value is used to supervise the decision prediction head. A second loss function value is calculated based on the labeled copy start position and the predicted copy start position corresponding to the same token in the action sequence; this second loss function value is used to supervise the start prediction head. A third loss function value is calculated based on the labeled copy end position and the predicted copy end position corresponding to the same token in the action sequence. The target loss function value is obtained based on the obtained first, second, and third loss function values, for example, by adding the three loss function values ​​together as the target loss function value.

[0123] In other words, the training unit 505 obtains the target loss function value for adjusting the three prediction heads based on the action decision information, the copy start position, and the copy end position included in the action sequence. This can improve the accuracy of the obtained target loss function value, thereby improving the accuracy when adjusting the parameters of the three prediction heads.

[0124] In addition, the training unit 505 can also use the LoRA (Low-Rank Adaptation) method to fine-tune the initial large model. That is, freeze the main parameters of the initial large model and train only the three newly added prediction heads and the introduced low-rank decomposition matrix. This can significantly reduce the computational resources and storage costs required to fine-tune the large model, so that the initial large model can be trained on consumer-grade hardware and easily applied to large language models of different sizes.

[0125] When acquiring training data, the training unit 505 can adopt the following approach: acquire the sample source text and its corresponding sample target text, and obtain the source word sequence of the sample source text and the target word sequence of the sample target text respectively; construct the N-gram index of the corresponding sample source text based on the source word sequence; query the N-gram index for multiple target words in the target word sequence respectively, and obtain the annotation action for each target word based on the query results; obtain the annotation action sequence based on the annotation actions of multiple target words, and acquire the training data based on the annotation action sequence and the sample source text.

[0126] In other words, the training unit 505 uses existing text pairs (i.e., including source text and its corresponding target text) to automatically acquire training data without any manual annotation cost. It can automatically acquire the labeled action sequence as training data, thereby improving the efficiency of training data acquisition and reducing the cost of training data acquisition.

[0127] The N-gram index of the corresponding sample source text constructed by training unit 505 includes multiple source word fragments and the start and end positions of each source word fragment in the sample source text.

[0128] When training unit 505 queries multiple target words in the target word sequence in the N-gram index and obtains the annotation action for each target word based on the query results, the following implementation method can be adopted: query the N-gram index based on the current target word; in response to determining that a target source word fragment matching the current target word is found in the N-gram index, the copy action is used as the annotation action decision information for the current target word, the starting position of the target source word fragment in the sample source text is used as the annotation copy starting position of the current target word, and the ending position of the target word fragment in the sample source text is used as the annotation copy ending position of the current target word.

[0129] Training unit 505 can also perform the following: in response to determining that no target source word fragment matching the current target word is found in the N-gram index, it generates an action as the annotation action decision information for the current target word.

[0130] In other words, the training unit 505 sequentially queries the target words in the target word sequence in the constructed N-gram index, obtains the labeled action for each target word in the target word sequence based on the query results, and then obtains the labeled action sequence for model training based on the obtained labeled action. This can improve the efficiency of obtaining the labeled action sequence and reduce the cost of obtaining the labeled action sequence.

[0131] In addition, the training unit 505 can query the N-gram index based on the current target word and the target word fragment consisting of N-1 target words following the current target word. If a target source word fragment corresponding to the target word fragment exists, the annotation action of the current target word is obtained based on the "copy action" and the position information of the corresponding target source word fragment. Then, after moving forward N units in the target word sequence, the next query is performed.

[0132] If training unit 505 fails to find a target source word fragment corresponding to the target word fragment in the N-gram index, it moves forward 1 unit in the target word sequence and continues to query the target word following the current target word.

[0133] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0134] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0135] like Figure 6 The diagram shown is a block diagram of an electronic device for an inference acceleration method for large models according to embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0136] like Figure 6As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0137] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0138] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as inference acceleration methods for large models. For example, in some embodiments, the inference acceleration methods for large models may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 608.

[0139] In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by computing unit 601, one or more steps of the inference acceleration method for large models described above may be performed. Alternatively, in other embodiments, computing unit 601 may be configured to execute the inference acceleration method for large models by any other suitable means (e.g., by means of firmware).

[0140] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable inference acceleration device for large models, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for showing information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0144] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0145] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0146] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0147] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for inference acceleration of a large model, comprising: obtaining a top-level hidden state of a target large model for predicting a next token after inputting a source text to be processed into the target large model; obtaining action decision information corresponding to the next token according to the top-level hidden state; in response to determining that the action decision information is a copy action, obtaining a text copy interval corresponding to the next token according to the top-level hidden state; copying text in the text copy interval in the source text to be processed, and taking the copied result as the next token.

2. The method of claim 1, wherein, The obtaining of the action decision information corresponding to the next token according to the top-level hidden state comprises: inputting the top-level hidden state into a decision prediction head of the target large model; obtaining the action decision information corresponding to the next token according to an output result of the decision prediction head.

3. The method of claim 1, wherein, The obtaining of the text copy interval corresponding to the next token according to the top-level hidden state in response to determining that the action decision information is a copy action comprises: inputting the top-level hidden state into a start point prediction head of the target large model, and obtaining a copy start point position corresponding to the next token according to an output result of the start point prediction head; inputting the top-level hidden state into an end point prediction head of the target large model, and obtaining a copy end point position corresponding to the next token according to an output result of the end point prediction head; obtaining the text copy interval corresponding to the next token according to the copy start point position and the copy end point position.

4. The method of claim 1, further comprising: in response to determining that the action decision information is a generation action, inputting the top-level hidden state into a language model head of the target large model; obtaining the next token according to an output result of the language model head.

5. The method of claim 1, wherein, The copying of the text in the text copy interval in the source text to be processed, and taking the copied result as the next token comprises: obtaining a subword sequence of the source text to be processed; determining subwords in the subword sequence located in the text copy interval according to position information of each subword in the subword sequence; copying the determined subwords, and taking the copied result as the next token.

6. The method of claim 1, further comprising: obtaining training data, wherein the training data comprises sample source texts and annotated action sequences corresponding to sample target texts; constructing an initial large model comprising a decoder module, a language model head, a decision prediction head, a start point prediction head, and an end point prediction head, wherein the decision prediction head is configured to output predicted action decision information according to a top-level hidden state output by the decoder module, the start point prediction head is configured to output a predicted copy start point position according to the top-level hidden state, and the end point prediction head is configured to output a predicted copy end point position according to the top-level hidden state. inputting the sample source text into the initial large model, and obtaining a predicted action sequence corresponding to a predicted target text according to an output result of the initial large model; calculating a target loss function value according to the labeled action sequence and the predicted action sequence, and adjusting parameters of the decision prediction head, the start point prediction head, and the end point prediction head using the target loss function value to obtain the target large model.

7. The method of claim 6, wherein, The obtaining training data comprises: obtaining the sample source text and its corresponding sample target text, and obtaining a source subword sequence of the sample source text and a target subword sequence of the sample target text respectively; constructing an N-gram index corresponding to the sample source text according to the source subword sequence; querying a plurality of target subwords in the target subword sequence in the N-gram index respectively, and obtaining a labeled action corresponding to each target subword according to a query result; obtaining the labeled action sequence according to the labeled actions of the plurality of target subwords, and obtaining the training data according to the labeled action sequence and the sample source text.

8. The method of claim 7, wherein, The querying a plurality of target subwords in the target subword sequence in the N-gram index respectively, and obtaining a labeled action corresponding to each target subword according to a query result comprises: querying the N-gram index according to a current target subword; in response to determining that a target source subword segment matching the current target subword is queried in the N-gram index, deciding to copy an action as the labeled action decision information of the current target subword, taking a start point position of the target source subword segment in the sample source text as a labeled copy start point position of the current target subword, and taking an end point position of the target subword segment in the sample source text as a labeled copy end point position of the current target subword.

9. The method of claim 8, further comprising, in response to determining that a target source subword segment matching the current target subword is not queried in the N-gram index, deciding to generate an action as the labeled action decision information of the current target subword.

10. The method of claim 6, wherein, The calculating a target loss function value according to the labeled action sequence and the predicted action sequence comprises: calculating a first loss function value according to labeled action decision information and predicted action decision information corresponding to the same token in the action sequence; calculating a second loss function value according to a labeled copy start point position and a predicted copy start point position corresponding to the same token in the action sequence; calculating a third loss function value according to a labeled copy end point position and a predicted copy end point position corresponding to the same token in the action sequence; obtaining the target loss function value according to the first loss function value, the second loss function value, and the third loss function value.

11. An inference acceleration device for a large model, comprising: an obtaining unit configured to obtain a top-level hidden state used by the target large model to predict a next token after inputting a to-be-processed source text into the target large model; a decision unit, configured to acquire action decision information corresponding to the next token according to the top-level hidden state; a processing unit, configured to acquire a text copy interval corresponding to the next token according to the top-level hidden state in response to determining that the action decision information is a copy action; a copying unit, configured to copy text in the to-be-processed source text located in the text copy interval, and take a copy result as the next token. 12.An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-10.

13. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to make the computer execute the method of any one of claims 1-10. 14.A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-10.

Images