A language model inference optimization method and device

Abstract

The application provides a language model inference optimization method and device. The inference optimization method comprises: in response to obtaining at least one request information requesting to use a preset language model to infer input information, dividing the first request information into multiple pieces of request information according to a preset length; batch-merging the multiple pieces of request information and second request information, performing zero-redundancy full-inference and mixed-inference on the multiple pieces of request information and the second request information by using the language model, and obtaining multiple first inference results, wherein the second request information is at least one request information obtained at the same time as the first request information or obtained in a full-inference or mixed-inference process on the multiple pieces of request information; and performing incremental inference on the multiple first inference results by using the language model, and obtaining an inference result output by the language model when inferring the input information. Through the above method, the occupation of the video memory during language model inference is reduced, and the inference efficiency of the language model is improved.

Description

[0001] This application is a divisional application of the Chinese patent application with the application number 202411076211.3, the application date of August 7, 2024, and the application title of "Inference Optimization Method of Language Model", the content of which is incorporated into the parent application. TECHNICAL FIELD

[0002] The present application relates to the technical field of large language models, in particular to an inference optimization method and device of a language model. BACKGROUND

[0003] The basic structure of a large language model is generally the decoder part of the Transformer language model. The Transformer language model is an autoregressive language model that predicts the next text token based on the previous text in the prediction process, and uses the newly obtained text token as input to predict the next text token. This continues until a token termination indicating the end of the representation is generated. In inference, it is divided into two stages. The first stage is the full inference (or first inference, pre-filling) stage for the previous prompt of the user input. The second stage is the incremental inference stage for the model prediction result.

[0004] Currently, for a super-long input of a long text request, the language model may need several seconds for full inference. The super-long input cannot be incrementally inferred simultaneously with the short input, resulting in a long waiting time for incremental inference, which reduces the efficiency of the language model in computing and inferring. SUMMARY

[0005] Therefore, the purpose of the present application is to provide an inference optimization method and device of a language model. Based on the idea of merging processing, the at least one request information is subjected to segmented full inference and mixed inference, which realizes a smaller occupation of the video memory when the language model infers the long text information, improves the throughput of the language model inference service in actual application, optimizes the video memory occupation and model inference performance in the inference process through zero redundancy processing for full inference and mixed inference, and improves the efficiency of the language model in computing and inferring.

[0006] The present application provides an inference optimization method of a language model, which comprises:

[0007] In response to obtaining at least one request information requesting to infer input information using a preset language model, dividing the first request information into multiple segments of request information according to a preset length;

[0008] batching and merging the multi-segment request information and the second request information, performing zero-redundancy full reasoning and mixed reasoning on the multi-segment request information and the second request information by using the language model to obtain a plurality of first reasoning results, wherein the second request information is at least one request information obtained simultaneously with the first request information or obtained in a process of performing full reasoning or mixed reasoning on the multi-segment request information;

[0009] performing incremental reasoning on the plurality of first reasoning results by using the language model to obtain a reasoning result output by the language model in the process of reasoning on the input information.

[0010] Further, the batching and merging the multi-segment request information and the second request information, performing zero-redundancy full reasoning and mixed reasoning on the multi-segment request information and the second request information by using the language model to obtain a plurality of first reasoning results comprises:

[0011] performing full reasoning or full reasoning and mixed reasoning on the first request segment information in the multi-segment request information by using the language model to obtain historical result information output by the full reasoning or the full reasoning and the mixed reasoning;

[0012] in response to obtaining the second request information requesting to perform reasoning on the input information by using the preset language model, merging the second request segment information in the multi-segment request information and the second request information to obtain first merging information;

[0013] performing zero-redundancy mixed reasoning on the historical result information and the first merging information by using the language model to obtain a plurality of first reasoning results.

[0014] Further, the performing zero-redundancy mixed reasoning on the historical result information and the first merging information by using the language model to obtain a plurality of first reasoning results comprises:

[0015] performing label embedding and layer normalization processing on the first merging information and the historical result information respectively to obtain first merging input information;

[0016] performing vector linear transformation on the first merging input information by using a self-attention layer of the language model to obtain first vector input information;

[0017] performing zero-redundancy full operation on the first vector input information by using the self-attention layer to obtain a first output result;

[0018] performing full feedforward processing on the first output result by using a feedforward layer of the language model to obtain a first intermediate result;

[0019] Based on N first merging input information, first vector input information, first output result and first intermediate result, a plurality of first inference results are obtained, and N is a natural number greater than 1.

[0020] Further, based on the first vector input information, the zero-redundancy full-amount operation is performed by using the self-attention layer to obtain the first output result, including:

[0021] The first vector input information is dispersedly input into a plurality of independent operators in the self-attention layer according to a preset division strategy;

[0022] The plurality of independent operators respectively perform full-amount operations on the dispersedly input first vector input information to obtain a plurality of first operation results, and the plurality of first operation results are merged and written into the preset video memory corresponding to the language model;

[0023] The plurality of first operation results merged and written into the preset video memory are split to obtain the first output result.

[0024] Further, the plurality of first inference results obtained by performing the zero-redundancy full-amount inference and the mixed inference on the plurality of pieces of request information and the second request information by using the language model, further include:

[0025] In response to obtaining third request information requesting to perform inference on input information by using a preset language model, the first inference result and the third request information are merged to obtain second merging information;

[0026] Based on the second merging information, zero-redundancy mixed inference is performed by using the language model to obtain a plurality of first inference results.

[0027] Further, the plurality of first inference results obtained by performing the zero-redundancy full-amount inference and the mixed inference on the plurality of pieces of request information and the second request information by using the language model, include:

[0028] The first request segment information in the plurality of pieces of request information is inferred by using the language model to obtain first historical result information output by the full-amount inference or the full-amount inference and the mixed inference;

[0029] In response to obtaining second request information requesting to perform inference on input information by using a preset language model, the second request segment information in the plurality of pieces of request information and the second request information are merged to obtain first merging information;

[0030] Based on the first historical result information and the first merged information, zero-redundancy hybrid reasoning is performed by using the language model to obtain at least one first reasoning result and second historical result information.

[0031] In response to obtaining third request information requesting to perform reasoning on input information by using a preset language model, the first reasoning result, third request segment information in the multiple pieces of request information, and the third request information are merged to obtain third merged information.

[0032] Based on the second historical result information and the third merged information, zero-redundancy hybrid reasoning is performed by using the language model to obtain multiple first reasoning results.

[0033] Further, the performing incremental reasoning on the multiple first reasoning results by using the language model to obtain a reasoning result output by the language model performing reasoning on the input information comprises:

[0034] The multiple first reasoning results are merged to obtain merged information.

[0035] Zero-redundancy incremental reasoning is performed on the merged information by using the language model to obtain a merged reasoning result output by the language model performing reasoning on the input information.

[0036] Embodiments of the present application further provide an inference optimization device of a language model, and the inference optimization device comprises:

[0037] The segmentation processing module is configured to divide first request information into multiple pieces of request information according to a preset length in response to obtaining at least one request information requesting to perform reasoning on input information by using a preset language model.

[0038] The hybrid reasoning module is configured to perform batch merging on the multiple pieces of request information and second request information, perform zero-redundancy full-amount reasoning and hybrid reasoning on the multiple pieces of request information and the second request information by using the language model, and obtain multiple first reasoning results, wherein the second request information is at least one request information obtained simultaneously with the first request information or obtained in a process of performing full-amount reasoning or hybrid reasoning on the multiple pieces of request information.

[0039] The incremental reasoning module is configured to perform incremental reasoning on the multiple first reasoning results by using the language model to obtain a reasoning result output by the language model performing reasoning on the input information.

[0040] The embodiment of the application further provides an electronic device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor communicates with the memory through the bus, and the machine readable instructions are executed by the processor to perform the steps of the inference optimization method of the language model as described above.

[0041] The embodiment of the application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to perform the steps of the inference optimization method of the language model as described above.

[0042] The inference optimization method and device of the language model provided by the embodiment of the application, the inference optimization method comprises: in response to obtaining at least one request information requesting to use a preset language model to perform inference on input information, dividing first request information into multiple pieces of request information according to a preset length; performing batch merging on the multiple pieces of request information and second request information, performing zero-redundancy full-inference and hybrid inference on the multiple pieces of request information and the second request information by using the language model, to obtain multiple first inference results, wherein the second request information is at least one request information obtained at the same time as the first request information or obtained in a full-inference or hybrid-inference process on the multiple pieces of request information; performing incremental inference on the multiple first inference results by using the language model, to obtain an inference result output by the language model when performing inference on the input information.

[0043] Compared with the method for processing long-text requests with super-long inputs in the prior art, the idea of merging processing is used to perform segmented full-inference and hybrid inference on at least one request information, so that the memory of the language model is less occupied when the language model performs inference on input long-text information, the throughput of the language model inference service is improved in actual application, and the memory occupation and model inference performance in the inference process are optimized through zero-redundancy processing in the full-inference and hybrid-inference, so that the efficiency of the language model in performing calculation inference is improved.

[0044] In order to make the above objectives, characteristics and advantages of the present application more apparent, clear and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are referred to, and the detailed description is as follows. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, and it should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0046] Figure 1 A hierarchical abstract diagram of a reasoning framework of a language model provided by an embodiment of the present application;

[0047] Figure 2 A flowchart of a reasoning optimization method of a language model provided by an embodiment of the present application;

[0048] Figure 3 A segmented schematic diagram of full reasoning of a language model provided by an embodiment of the present application;

[0049] Figure 4 A reasoning schematic diagram of zero-redundancy processing of a language model provided by an embodiment of the present application;

[0050] Figure 5 A structural schematic diagram of a reasoning optimization device of a language model provided by an embodiment of the present application;

[0051] Figure 6 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, every other embodiment obtained by a person skilled in the art without creative work belongs to the scope of protection of the present application.

[0053] Currently, the classic architecture of the Transformer model in the language model includes three main components: the self-attention (Masked Multi-Head Attention) part, which is used to establish the dependency relationship between positions in the input sequence; the residual connection and normalization operation part (Add&Norm); the feed forward neural network part (Feed Forward) and the multilayer perceptron (MLP) part. Through the cooperation of these components, the Transformer model can effectively capture the key information in the input sequence and achieve strong sequence modeling capability.

[0054] It is found through research that the intermediate memory part with a large proportion in the inference process based on the Transformer model structure mainly includes the input and output parts of the model, the output part of the self-attention layer, and the temporary memory part required in the dimension increasing and then dimension reducing process of the feedforward layer; among them, the memory proportion is in linear relationship with the length of the text token (token) processed by the language model, and the size of the intermediate memory part of the language model in the inference process has an important influence on processing long text input. In large model inference, storage resources have become an important competitive factor for high-performance computing. In the case of processing super-long input, the occupation of memory resources is very tense, which limits the process of long text inference of the language model.

[0055] Here, when the language model based on the Transformer model structure receives a super-long input of a long text request, the language model may need a few seconds when performing full inference, and the super-long input cannot be incrementally inferred at the same time as the short input, which makes the waiting time of the incremental inference too long, reducing the efficiency of the language model when performing calculation and inference. In addition, in large model inference, storage resources have become an important competitive factor for high-performance computing. In the case of processing super-long input, the occupation of memory resources is very tense, which limits the process of long text inference of the language model.

[0056] To solve the problems of at least one of the above aspects, the embodiments of the present application provide a language model inference optimization method.

[0057] In the embodiments of the present application, when the pre-set language model infers the input long text information, the overall service framework of the language model includes but is not limited to application business layer, inference service processor and inference engine, etc. Among them, the application business layer includes various application businesses, for example, realizing application businesses such as dialogue chat, intelligent assistant and creation; the inference service processor is responsible for processing multiple concurrent requests, for example, requests such as initialization, calling inference engine and sampling inference result; the inference engine can be classified according to the characteristics of inference, for example, full inference, incremental inference and mixed inference.

[0058] Figure 1 A layered abstraction diagram of a language model inference framework provided by the embodiments of the present application.

[0059] As Figure 1As shown in the figure, in this example, three abstraction layers are included, the first abstraction layer (the uppermost layer) refers to model abstraction, the model abstraction includes but is not limited to full reasoning process of execution context, incremental reasoning process of execution autoregression, and mixed reasoning including full reasoning and incremental reasoning; the second abstraction layer (the middle layer) refers to model layer abstraction for performing operations of all same layers, the model layer abstraction mainly includes but is not limited to Self Attention (self-attention layer) and FeedForward (feedforward layer); the third abstraction layer (the lowermost layer) refers to operator abstraction, the operator abstraction includes but is not limited to FMHA (multi-head self-attention), linear transformation (Linear), addition and normalization (Add&Norm), and softmax, etc.

[0060] Firstly, the full reasoning, mixed reasoning and incremental reasoning involved in the embodiments of the present application are introduced:

[0061] Full reasoning refers to that when processing input text data information, the language model will perform a comprehensive analysis and processing through the entire data set stored by the model to generate a complete output or result, wherein the language model will consider all information and context related to the input text data information in the processing process to ensure the accuracy and comprehensiveness of the output.

[0062] As an example of full reasoning, when a user inputs a complex question to a chat robot based on a language model, the chat robot can perform full reasoning in the following way: first, the chat robot analyzes the grammatical structure of the question and understands the content the user wants to ask; then, the chat robot retrieves its knowledge base to find information and answers related to the question; finally, the chat robot integrates these information to generate a complete, coherent and accurate answer and presents it to the user.

[0063] Mixed reasoning refers to a strategy of flexibly selecting and combining full reasoning and incremental reasoning according to different parts or stages of a problem when solving the problem.

[0064] Incremental reasoning is suitable for scenarios where the input data changes little or the reasoning task needs to be repeatedly executed. When the input data only changes locally, only the changed part is re-reasoned, rather than repeatedly reasoning the entire data set or input. Incremental reasoning can significantly reduce redundant calculations, reduce computational load and power consumption, thereby improving the execution speed and efficiency of reasoning.

[0065] As an example of incremental reasoning, in a video target detection task, there is a lot of similar information between consecutive frames of the video. At this time, only the areas with large inter-frame differences can be incrementally reasoned, and in this way, repeated processing of the entire frame image can be reduced.

[0066] Referring to Figure 2 , Figure 2 A flowchart of a reasoning optimization method of a language model provided by an embodiment of the present application.

[0067] As Figure 2 indicated in the foregoing description, the reasoning optimization method provided by the embodiment of the present application specifically includes:

[0068] Step S100, in response to obtaining at least one request information requesting to utilize a preset language model to perform reasoning on input information, dividing the first request information into multiple pieces of request information according to a preset length.

[0069] In the embodiment of the present application, when inputting information to the preset language model and requesting to perform reasoning on the input information by the language model, at least one request information can be obtained based on the information input to the language model.

[0070] Here, the input information includes but is not limited to long text information, short text information, context information, metadata, feature representation information, etc.; the at least one request information refers to one or more information determined based on the information input to the language model, and the request information is used to instruct the language model to perform reasoning on the input information.

[0071] In addition, the first request information is the request information that needs to be split in the at least one request information; and the preset length can be a division length specifically set according to the performance of the preset language model and the input long text information.

[0072] In the step, in the specific implementation, first, at least one request information is obtained from an external request to perform reasoning on input information input to a preset language model; then, in response to the at least one request information being obtained, a preset length corresponding to the external request is obtained; and finally, a first request information in the at least one request information is divided into multiple pieces of request information according to the preset length.

[0073] For example, assuming that the length of the first request information is 4000 and the preset length is 2000, at this time, the first request information with the length of 4000 can be divided into two pieces of request information according to the preset length of 2000, and the length of each piece of request information is 2000.

[0074] S200, performing batch merging on the multiple pieces of request information and second request information, performing zero-redundancy full reasoning and mixed reasoning on the multiple pieces of request information and the second request information by utilizing the language model, and obtaining multiple first reasoning results, wherein the second request information is at least one request information obtained simultaneously with the first request information or obtained in a process of performing full reasoning or mixed reasoning on the multiple pieces of request information.

[0075] In the embodiments of the present application, in order to improve the processing efficiency of the language model for the input information and reduce the resource occupation, the request information can be processed by merging the multiple request information and the second request information.

[0076] Further, considering that the multiple request information divided from the first request information has multiple division cases, one case is that the multiple request information is divided into N, and another case is that the multiple request information is divided into N+1. N is a natural number greater than 1.

[0077] Therefore, the merging processing manner of the multiple request information and the second request information can include one case that the multiple request information is divided into N and another case that the multiple request information is divided into N+1.

[0078] The following will describe the two cases in the merging processing manner in step S200 by specific examples.

[0079] The first case in the merging processing manner, for example, assuming that the multiple request information is divided into 2, step S200 can include:

[0080] In step S211, the language model is used to perform full reasoning or full reasoning and mixed reasoning on the first request segment information in the multiple request information, to obtain historical result information output by the full reasoning or the full reasoning and the mixed reasoning.

[0081] In the embodiments of the present application, according to whether the obtained request information contains historical information and the input length of the request information, the specific reasoning manner of the language model for the request information is determined.

[0082] Specifically, when the request information does not contain historical information, full reasoning is performed on the request information; when the request information contains historical information and the input length of the request information is 1, incremental reasoning is performed on the request information; when the request information contains historical information and the input length of the request information is not 1, mixed reasoning is performed on the request information.

[0083] In this step, based on whether the first request segment information in the divided multiple request information contains historical result information and the input length of the first request segment information, full reasoning or full reasoning and mixed reasoning is determined for the first request segment; and then, historical result information output by the full reasoning or the full reasoning and the mixed reasoning is obtained.

[0084] For example, assuming that the A request information does not contain historical information and the input length is 2000, full reasoning is performed on the A request information.

[0085] For example, assume that B request information and C request information are obtained, the B request information contains historical information and the input length is 1, and incremental inference is performed on the B request information; the C request information contains historical information and the input length is not 1, and mixed inference is performed on the C request information.

[0086] Here, the first request segment information is not the first segment in a physical sense, but information that needs to be inferred before the second request information is received after the first request information is segmented.

[0087] Specifically, in the process of performing full-inference and full-inference with historical information, the segment information of full-inference and the segment information of full-inference with historical information (which can be referred to as mixed inference) are both determined as the first request segment information.

[0088] As an example of full-inference on the first request segment information, the inference on the first request segment information in the multi-segment request information is performed by the statement "model.run_gpt(input_tokens=split_1_2000)", where the length (tokens) of the first request information is 2000. Further, the full-inference on the first request segment information is performed by the statement "input_lengths=

[2000] ;resource_indices=[0];infer_categories=[0];history_lengths=[0];is_first_finished_list=[0]", where the first request segment information uses the resource with index 0, and the length of the historical result information is 0.

[0089] Figure 3 A segmentation diagram of full-inference of a language model provided by an embodiment of the present application.

[0090] As shown in Figure 3 In full-inference of request information by a language model, assume that the first request information with an input length of 8000 is segmented into four request information each with a length of 2000, and the full-inference with an input length of 8000 is performed once by adding full-inference with historical information, which becomes four times of full-inference with historical information; where the length of the historical information of the first full-inference is 0. In this way, since the size of the intermediate memory is linearly related to the length of the processed text token, the segmentation processing of full-inference can optimize the peak memory occupation.

[0091] In the full-inference of request information by a language model, assume that the first request information with an input length of 8000 is segmented into four request information each with a length of 2000, and the full-inference with an input length of 8000 is performed once by adding full-inference with historical information, which becomes four times of full-inference with historical information; where the length of the historical information of the first full-inference is 0. In this way, since the size of the intermediate memory is linearly related to the length of the processed text token, the segmentation processing of full-inference can optimize the peak memory occupation. Figure 3In the example shown in the figure, the full-volume inference by means of information segmentation reduces the occupation of the intermediate memory of the system by three quarters, which, compared with the common method of memory optimization by host device offloading, avoids the overhead and complex heterogeneous scheduling implementation problems in data copying between devices.

[0092] In step S212, in response to obtaining the second request information for inferring the input information by means of the preset language model, the second request segment information in the multiple request information and the second request information are merged to obtain first merged information.

[0093] In the embodiment of the present application, in the process of inferring the first request segment information, the second request information is also obtained, and the second request segment information and the second request information can be merged and processed to reduce the resource occupation of the language model inference by means of zero-redundancy inference.

[0094] In an example of the present application, the second request information is obtained by the statement "request_2=GetLastRequest()", and the second request segment information and the second request information are merged by the statement "input_tokens=Merge(split_2_2000,request_2)" to obtain the first merged information.

[0095] For example, it is assumed that in the process of inferring the G1 request segment information in the G request information, the H request information is obtained, at which time the G1 request segment information and the H request information need to be merged into the first merged information for zero-redundancy inference of the first merged information.

[0096] In step S213, zero-redundancy mixed inference is performed by means of the language model based on the historical result information and the first merged information to obtain multiple first inference results.

[0097] In the embodiment of the present application, the language model runs inference in N layers in the inference process, and the N layers include multiple hidden layers, which are multi-level abstractions of input features to linearly divide the data of the input layer into different types of data. N is a natural number greater than 1.

[0098] In an embodiment of the present application, in specific implementation, the multiple first inference results are obtained by performing zero-redundancy mixed inference on the vectors included in the hidden layer in step S213, which specifically includes:

[0099] In S2131, the first merged information and the historical result information are respectively subjected to token embedding and layer normalization processing to obtain first merged input information.

[0100] In this step, in specific implementation, firstly, the first merged information and the historical result information are subjected to model mark embedding; then, data distribution characteristics included in the model mark embedded information are calculated, and the data distribution characteristics are adjusted to a specified range for normalization processing; finally, a preset scale and a shift are introduced into the normalized data to obtain the first merged input information.

[0101] S2132, vector linear transformation is performed on the first merged input information by using a self-attention layer of the language model to obtain first vector input information.

[0102] In this step, the first merged input information is transformed by using a self-attention layer of the language model through three different linear transformation modes. Specifically, the first merged input information is multiplied by three different weight matrices to obtain Q, K, and V vectors, and the three vectors are determined as the obtained first vector input information.

[0103] The three different weight matrices can correspond to trainable parameter matrices WQ, WK, and WV, respectively.

[0104] S2133, based on the first vector input information, zero-redundancy full-amount operation is performed by using the self-attention layer to obtain a first output result.

[0105] In an embodiment of the present application, in specific implementation, step S2133 can include:

[0106] S21331, the first vector input information is dispersedly input into a plurality of independent operators in the self-attention layer according to a preset division strategy.

[0107] In this step, based on the number of information in the first vector input information, the first vector input information is sequentially and dispersedly input into a plurality of independent operators in the self-attention layer.

[0108] For example, assuming that the first vector input information includes information with lengths of 1000, 1, 1000, and 1, the first vector input information is input into four independent operators respectively for subsequent operator operation.

[0109] S2332, the plurality of independent operators respectively perform full-amount operation on the dispersedly input first vector input information respectively to obtain a plurality of first operation results, and the plurality of first operation results are merged and written into a preset video memory corresponding to the language model.

[0110] In the embodiment of the present application, the language model generates a plurality of first operation results at one time after processing the entire input sequence, and writes the plurality of first operation results into the preset video memory corresponding to the language model.

[0111] In step S2333, the plurality of first operation results written into the preset video memory are split to obtain a first output result.

[0112] In this step, the plurality of first operation results are split based on the input token information after the request processing to obtain a first output result.

[0113] In step S2134, the first output result is fully fed forward processed by using the feedforward layer of the language model to obtain a first intermediate result.

[0114] In the embodiment of the present application, the first output result is fully fed forward processed based on the processing data capacity of the feedforward layer in the language model to obtain a first intermediate result, so as to perform operation on the first intermediate result by other processing included in the language model.

[0115] In step S2135, based on N first merged input information, first vector input information, first output result and first intermediate result, a plurality of first inference results are obtained, and N is a natural number greater than 1.

[0116] In the embodiment of the present application, the hidden layer vector includes but is not limited to the first merged input information, the first vector input information, the first output result and the first intermediate result.

[0117] Here, it should be understood that after the full feedforward processing of the first output result by the feedforward layer, the inference result is not obtained, and therefore, after the model runs all layers, the language model obtains a plurality of first inference results by other processing on the hidden layer vector.

[0118] Figure 4 A language model zero-redundancy processing inference schematic diagram provided in the embodiment of the present application.

[0119] As an example of zero-redundancy inference, as shown in Figure 4 the first merged information includes four input information, the lengths of which are 1000, 1, 1000 and 1 in turn, and the shape size of the first merged information input is 2002; after the embedding and layer normalization processing of the first merged information, since the above processing is for the text token label of each input information, the intermediate video memory shape size obtained is [2002, H] (H represents the width of the hidden layer).

[0120] Further, in the inference process of the self-attention layer, since the attn operator for calculating the attention score is for each input information in the first merged information, after the vector linear transformation (QKVlinear) is performed on the first merged input information, the first vector input information is dispersed into 4 independent operators (attn1-4) for execution, so as to internally split and write the obtained plurality of first operation results in the same block of the preset GPU memory; then, the first output result is subjected to full-amount feed forward processing by using the feed forward layer of the language model, to obtain a first intermediate result; finally, based on the first merged input information, the first vector input information, the first output result and the first intermediate result, a plurality of first inference results are obtained through the processing of other layers of the model.

[0121] Through the above-mentioned zero redundancy processing at the input end of the language model, the separate merging operation is realized in the operator operation part, and the occupation of the temporary GPU memory by the language model during inference is reduced.

[0122] Optionally, in addition to the full-amount inference and mixed inference of the language model on the plurality of pieces of request information and the second request information to obtain a plurality of first inference results as described in the above steps, steps S214 to S215 are further included, specifically, steps S214 to S215 are used to illustrate an inference method after the obtained second request information is inferred and a third request information is further obtained, and specifically includes:

[0123] Step S214, in response to obtaining the third request information requesting to infer the input information by using the preset language model, merging the first inference result and the third request information to obtain second merged information.

[0124] In the embodiment of the present application, for the obtained third request information, the first inference result and the third request information can be merged to obtain the second merged information, so as to perform zero redundancy mixed inference on the second merged information.

[0125] Step S215, based on the second merged information, performing zero redundancy mixed inference by using the language model to obtain a plurality of first inference results.

[0126] The description of step S215 can refer to the description of steps S2131 to S2135, and the same technical effects can be achieved, and thus will not be described herein.

[0127] As an example of the first case in the manner of the merging processing, assuming that the G request information is to be divided into 2, first, the G request information obtained first is segmented to obtain G1 request segment information and G2 request segment information; then, the G1 request segment information is reasoned to obtain the historical result information Y, and the H request information is obtained in the process; thereafter, the G2 request segment information and the H request information are merged into first merged information, and the historical result information Y and the first merged information are input into the language model to perform zero-redundancy mixed reasoning to obtain a plurality of first reasoning results X1; at this time, the I request information is obtained, the first reasoning results X1 and the I request information are merged to obtain second merged information; finally, the second merged information is input into the language model to perform zero-redundancy mixed reasoning to obtain a plurality of first reasoning results X2.

[0128] The second case in the manner of the merging processing, for example, assuming that the multi-segment request information is divided into 3, the language model is used to perform full reasoning and mixed reasoning on the multi-segment request information and second request information to obtain a plurality of first reasoning results, which can specifically include:

[0129] Step S221, full reasoning or full reasoning and mixed reasoning on the first request segment information in the multi-segment request information is performed by using the language model to obtain first historical result information output by the full reasoning or the full reasoning and the mixed reasoning.

[0130] Step S222, in response to obtaining the second request information requesting to perform reasoning on the input information by using the preset language model, the second request segment information in the multi-segment request information and the second request information are merged to obtain first merged information.

[0131] Step S223, based on the first historical result information and the first merged information, zero-redundancy mixed reasoning is performed by using the language model to obtain at least one first reasoning result and second historical result information.

[0132] Wherein, the description of steps S221 to S223 can refer to the description of steps S211 to S213, and the same technical effects can be achieved, and details are not repeated.

[0133] Step S224, in response to obtaining the third request information requesting to perform reasoning on the input information by using the preset language model, the first reasoning result, the third request segment information in the multi-segment request information and the third request information are merged to obtain third merged information.

[0134] In the embodiment of the present application, for the obtained third request information, the first inference result, the third request segment information and the third request information can be merged to obtain third merged information, and the third merged information and the second historical result information are subjected to zero-redundancy mixed inference.

[0135] In step S225, based on the second historical result information and the third merged information, zero-redundancy mixed inference is performed using the language model to obtain a plurality of first inference results.

[0136] The description of step S225 can refer to the description of steps S2131 to S2135, and the same technical effects can be achieved, and thus will not be described here.

[0137] As an example of the second case in the batch parallel processing mode, assuming that the G request information is divided into 2, first, the G request information obtained first is segmented to obtain G1 request segment information, G2 request segment information and G3 request segment information; then, the G1 request segment information is inferred to obtain the first historical result information Y1, and in the process, the H request information is obtained, and the G2 request segment information and the H request information need to be merged into the first merged information; then, the first historical result information Y1 and the first merged information are input into the language model to perform zero-redundancy mixed inference to obtain at least one first inference result X1 and the second historical result information Y2; then, the I request information is obtained, the first inference result X1, the D3 request segment information and the I request information are merged into the third merged information; finally, the second historical result information Y2 and the third merged information are input into the language model to perform zero-redundancy mixed inference to obtain a plurality of first inference results X2.

[0138] Referring back to Figure 2 , in step 300, the plurality of first inference results are subjected to incremental inference using the language model to obtain an inference result output by the language model for the input information.

[0139] As an example of incremental inference on merged information, the merged information is subjected to incremental inference by the statement "logits=model.run_gpt(input_tokens=input_tokens, input_lengths=[1,1,1], resource_indices=[0,1,2], infer_categories=[1,1,1], history_lengths=[4001,2001,1000], is_first_finished_list=[1,1,1])".

[0140] Wherein, input_tokens represents a list of input text tokens, with a shape of [total_length], which needs to be padded when used; input_lengths represents a list of input lengths, with a shape of [max_batch_size]; resource_indices represents a list of positions of computing resources, with a shape of [max_batch_size] and a value range of [0, max_batch_size-1]; infer_categories represents a list of input inference categories, with a shape of [max_batch_size], where a parameter of 0 represents full-inference, and a parameter of 1 represents incremental inference; history_lengths represents an input of the number of tokens that have been inferred, with a shape of [max_batch_size]; is_first_finished_list represents whether the input has been completed in the current full-inference, i.e., whether it is the last piece of input, with a shape of [max_batch_size], where a parameter of -1 represents an invalid value (used for incremental inference); a parameter of 0 represents that the inference has not ended; and a parameter of 1 represents that the inference has ended.

[0141] Here, incremental inference is a full-inference with historical information, and the input length is 1, i.e., full-inference with historical information naturally supports incremental inference; since the difficulty of inference in the input batch process with a length of 1 lies in IO bandwidth, the short input with a length of 1 is combined with long input for inference, and the inference delay caused by the short input can be ignored.

[0142] In an embodiment of the present application, in specific implementation, step S300 can include:

[0143] S310, merging a plurality of first inference results to obtain merged information.

[0144] S320, performing zero-redundancy incremental inference on the merged information using the language model to obtain a merged inference result output by the language model for inference on the input information.

[0145] In this way, through the mixed inference process of segmented and merged batch processing, continuous requests can be efficiently processed, the advantages of batch processing can be fully utilized while ensuring real-time performance, the batch processing size of incremental inference can be dynamically increased, and the system can effectively utilize computing resources to improve the throughput and efficiency of inference.

[0146] The inference optimization method of the language model provided in the embodiments of the present application is based on the idea of merging processing, performs segmented full-inference and mixed-inference on at least one request information, realizes smaller occupation of the video memory when the language model performs inference on the input long-text information, improves the throughput of the language model inference service in actual application, and optimizes the video memory occupation and model inference performance in the inference process through zero-redundancy processing for full-inference and mixed-inference, and improves the efficiency of the language model when performing calculation inference.

[0147] Referring to Figure 5 , Figure 5 is a structural schematic diagram of an inference optimization device of a language model provided in the embodiments of the present application. As shown in Figure 5 , the inference optimization device 500 includes:

[0148] The segmentation processing module 510 is configured to divide first request information into multiple pieces of request information according to a preset length in response to obtaining at least one request information requesting to perform inference on input information by using a preset language model.

[0149] The mixed-inference module 520 is configured to perform batch merging on the multiple pieces of request information and second request information, perform zero-redundancy full-inference and mixed-inference on the multiple pieces of request information and the second request information by using the language model, and obtain multiple first inference results, wherein the second request information is at least one request information obtained simultaneously with the first request information or obtained in a full-inference or mixed-inference process on the multiple pieces of request information.

[0150] The incremental-inference module 530 is configured to perform incremental-inference on the multiple first inference results by using the language model, and obtain an inference result output by the language model when performing inference on the input information.

[0151] The inference optimization device of the language model provided in the embodiments of the present application is based on the idea of merging processing, performs segmented full-inference and mixed-inference on at least one request information, realizes smaller occupation of the video memory when the language model performs inference on the input long-text information, improves the throughput of the language model inference service in actual application, and optimizes the video memory occupation and model inference performance in the inference process through zero-redundancy processing for full-inference and mixed-inference, and improves the efficiency of the language model when performing calculation inference.

[0152] Referring to Figure 6 , Figure 6 is a structural schematic diagram of an electronic device provided in the embodiments of the present application. As shown in Figure 6 , the electronic device 600 includes a processor 610, a memory 620 and a bus 630.

[0153] The memory 620 stores machine readable instructions executable by the processor 610, when the electronic device 600 is running, the processor 610 and the memory 620 communicate through the bus 630, the machine readable instructions are executed by the processor 610, can execute the method as described above Figure 2 The steps of the inference optimization method of the language model in the method embodiment are shown, and the specific implementation can be referred to the method embodiment, which will not be repeated here.

[0154] The embodiment of the application further provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is run by the processor, and the computer program can execute the method as described above Figure 2 The steps of the inference optimization method of the language model in the method embodiment are shown, and the specific implementation can be referred to the method embodiment, which will not be repeated here.

[0155] The skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

[0156] In several embodiments provided in the application, it should be understood that the disclosed system, device and method can be implemented by other ways. The device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, and for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.

[0157] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment.

[0158] In addition, each functional unit in each embodiment of the application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0159] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application essentially or the parts of the prior art that make contributions or parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0160] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A reasoning optimization method for a language model, characterized in that, The inference optimization method includes: In response to receiving at least one request message that requests reasoning about the input information using a preset language model, the first request message is divided into multiple request messages according to a preset length. The multiple request information segments and the second request information are merged in batches. The language model is used to perform zero-redundancy full-scale reasoning and hybrid reasoning on the information obtained after merging the multiple request information segments and the second request information in batches to obtain multiple first reasoning results. The second request information is at least one request information obtained at the same time as the first request information or during the process of full-scale reasoning or hybrid reasoning of the multiple request information segments. The process involves merging the multiple request information segments and the second request information in batches, and then using the language model to perform zero-redundancy full-scale reasoning and hybrid reasoning on the information obtained after merging the multiple request information segments and the second request information in batches, resulting in multiple first reasoning results, including: The language model is used to perform full reasoning, or full reasoning and hybrid reasoning, on the first request fragment information in the multiple request information segments to obtain historical result information output by the full reasoning, or full reasoning and hybrid reasoning. In response to receiving the second request information requesting reasoning of the input information using a preset language model, the second request fragment information and the second request information in the multiple request information segments are merged to obtain the first merged information; Based on the historical result information and the first merged information, the language model is used to perform zero-redundancy hybrid reasoning to obtain multiple first reasoning results; Incremental reasoning is performed on multiple first reasoning results using the language model to obtain the reasoning result output by the language model in reasoning about the input information.

2. The method according to claim 1, characterized in that, Based on the historical result information and the first merged information, the language model is used to perform zero-redundancy hybrid reasoning to obtain multiple first reasoning results, including: The first merged information and the historical result information are respectively subjected to label embedding and layer normalization processing to obtain the first merged input information; The first merged input information is transformed into a vector linear transformation using the self-attention layer of the language model to obtain the first vector input information; Based on the first vector input information, the self-attention layer is used to perform a full computation with zero redundancy to obtain the first output result; The first output result is fully fed forward using the feedforward layer of the language model to obtain the first intermediate result. Based on N first merged input information, first vector input information, first output result and first intermediate result, multiple first inference results are obtained, where N is a natural number greater than 1.

3. The method according to claim 2, characterized in that, The step of performing a full computation with zero redundancy using the self-attention layer based on the first vector input information to obtain the first output result includes: The first vector input information is distributed into multiple independent operators in the self-attention layer according to a preset partitioning strategy; The multiple independent operators are used to perform full operations on the first vector input information that is input in a dispersed manner to obtain multiple first operation results, and the multiple first operation results are merged and written into the preset display memory corresponding to the language model; The multiple first operation results that are merged and written into the preset video memory are split to obtain the first output result.

4. The method according to claim 1, characterized in that, The step of merging the multiple request information segments and the second request information in batches, and using the language model to perform zero-redundancy full-scale reasoning and hybrid reasoning on the information obtained after merging the multiple request information segments and the second request information in batches to obtain multiple first reasoning results, also includes: In response to receiving a third request message requesting reasoning on the input information using a preset language model, the first reasoning result and the third request message are merged to obtain a second merged message. Based on the second merged information, the language model is used to perform zero-redundancy hybrid reasoning to obtain multiple first reasoning results.

5. The method according to claim 1, characterized in that, The process involves merging the multiple request information segments and the second request information in batches, and then using the language model to perform zero-redundancy full-scale reasoning and hybrid reasoning on the information obtained after the batch merging of the multiple request information segments and the second request information, resulting in multiple first reasoning results, including: The language model is used to perform full reasoning, or full reasoning and hybrid reasoning, on the first request fragment information in the multiple request information segments to obtain the first historical result information output by the full reasoning, or the full reasoning and hybrid reasoning. In response to receiving the second request information requesting reasoning of the input information using a preset language model, the second request fragment information and the second request information in the multiple request information segments are merged to obtain the first merged information; Based on the first historical result information and the first merged information, the language model is used to perform zero-redundancy hybrid reasoning to obtain at least one first reasoning result and second historical result information; In response to receiving a third request information requesting reasoning on the input information using a preset language model, the first reasoning result, the third request fragment information in the multiple request information segments, and the third request information are merged to obtain third merged information; Based on the second historical result information and the third merged information, the language model is used to perform zero-redundancy hybrid reasoning to obtain multiple first reasoning results.

6. The method according to claim 1, characterized in that, The step of incrementally reasoning on multiple first inference results using the language model to obtain the inference result output by the language model inferring the input information includes: The multiple first inference results are merged to obtain merged information; The language model is used to perform zero-redundancy incremental reasoning on the merged information to obtain the merged reasoning result output by the language model in reasoning on the input information.

7. A reasoning optimization device for a language model, characterized in that, The reasoning optimization device for the language model includes: The segmentation module is used to, in response to receiving at least one request information that requests to reason about the input information using a preset language model, divide the first request information into multiple request information segments according to a preset length. The hybrid reasoning module is used to merge the multiple request information segments and the second request information in batches, and use the language model to perform zero-redundancy full reasoning and hybrid reasoning on the information obtained after merging the multiple request information segments and the second request information in batches to obtain multiple first reasoning results. The second request information segment is at least one request information segment obtained at the same time as the first request information segment or during the full reasoning or hybrid reasoning process of the multiple request information segments. When the hybrid reasoning module is used to merge the multiple request information segments and the second request information in batches, and to perform zero-redundancy full reasoning and hybrid reasoning on the information obtained after merging the multiple request information segments and the second request information in batches using the language model to obtain multiple first reasoning results, the hybrid reasoning module is used to: The language model is used to perform full reasoning, or full reasoning and hybrid reasoning, on the first request fragment information in the multiple request information segments to obtain historical result information output by the full reasoning, or full reasoning and hybrid reasoning. In response to receiving the second request information requesting reasoning of the input information using a preset language model, the second request fragment information and the second request information in the multiple request information segments are merged to obtain the first merged information; Based on the historical result information and the first merged information, the language model is used to perform zero-redundancy hybrid reasoning to obtain multiple first reasoning results; The incremental reasoning module is used to perform incremental reasoning on multiple first reasoning results using the language model to obtain the reasoning result output by the language model in reasoning on the input information.

8. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the inference optimization method of the language model as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the inference optimization method for the language model as described in any one of claims 1 to 6.

Images