Language model reasoning optimization method
Through segmented full-scale inference and mixed inference, combined with zero redundancy processing, the memory usage and efficiency problems of large language models when processing long text input are solved, and the throughput and model performance of inference services are improved.
Patent Information
- Application Number
- CN202411076211.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-07
AI Technical Summary
When large language models process long text input, the full inference time is long, resulting in too long waiting time for incremental inference and tight use of video memory resources, limiting the efficiency and performance of the language model.
The segmented full-quantity inference and mixed inference method are adopted to divide the input information into multi-segment request information and use zero-redundancy processing technology to optimize the memory usage and model inference performance, and improve the computational inference efficiency.
The language model reduces the video memory usage during long text information inference, and improves the throughput of inference services and the computational inference efficiency of the model.
Smart Images

Figure CN119090006A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of large language models, and in particular to a method for optimizing reasoning of a language model. Background Art
[0002] The infrastructure of a large language model is generally the decoder part of the Transformer language model. The Transformer language model is an autoregressive language model. During the prediction process, it predicts the next text token based on the previous context, and uses the newly obtained text token as input to predict the next text token. This iteration continues until a token representing the end marker is generated. The reasoning is divided into two stages. The first stage is the full reasoning (or first reasoning, pre-filling) stage for the previous context prompts input by the user, and the second stage is the incremental reasoning stage for the model prediction results.
[0003] Currently, for very long inputs of long text requests, the language model may take several seconds to perform full inference. The very long input cannot be incrementally inferred at the same time as the short input, which makes the waiting time for incremental inference too long, reducing the efficiency of the language model when performing computational inference. In addition, in large-scale model inference, storage resources have become an important competitive factor for high-performance computing. When processing very long inputs, the occupancy of video memory resources is very tight, which limits the progress of the language model in performing long text inference. Summary of the invention
[0004] In view of this, the purpose of this application is to provide a method for optimizing the reasoning of a language model. Based on the idea of continuous batch processing, segmented full reasoning and hybrid reasoning are performed on at least one request information, so as to achieve a smaller occupation of video memory when the language model performs reasoning on the input long text information, improve the throughput of the language model reasoning service in practical applications, and optimize the video memory occupancy and model reasoning performance in the reasoning process through zero redundancy processing in full reasoning and hybrid reasoning, thereby improving the efficiency of the language model when performing computational reasoning.
[0005] The present application embodiment provides a method for optimizing the reasoning of a language model, the method comprising:
[0006] In response to obtaining at least one request information requesting to use a preset language model to infer input information, dividing the first request information into a plurality of request information segments according to a preset length;
[0007] Using the language model to perform full reasoning and mixed reasoning on the multiple segments of request information and the second request information, to obtain multiple first reasoning results, wherein the second request information is at least one request information obtained at the same time as the first request information or obtained in the process of performing full reasoning or mixed reasoning on the multiple segments of request information;
[0008] Incremental reasoning is performed on the plurality of first reasoning results using the language model to obtain a reasoning result output by the language model when reasoning on the input information.
[0009] Furthermore, the language model is used to perform full reasoning and mixed reasoning on the multiple request information segments and the second request information to obtain multiple first reasoning results, including:
[0010] Using the language model, perform full reasoning, or full reasoning and mixed reasoning, on a first request segment information among the multiple request information segments to obtain historical result information output by the full reasoning, or the full reasoning and mixed reasoning;
[0011] In response to obtaining the second request information requesting to use a preset language model to infer the input information, taking the second request fragment information in the multiple request information segments and the second request information as the same batch request to obtain first request batch information;
[0012] Based on the historical result information and the first request batch information, hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
[0013] Further, the using the language model to perform full reasoning and mixed reasoning on the multiple request information segments and the second request information to obtain multiple first reasoning results also includes:
[0014] In response to obtaining third request information requesting to use a preset language model to infer input information, taking the plurality of first inference results and the third request information as a same batch request to obtain second request batch information;
[0015] Based on the second request batch information, hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
[0016] Furthermore, the language model is used to perform full reasoning and mixed reasoning on the multiple request information segments and the second request information to obtain multiple first reasoning results, including:
[0017] Using the language model, perform full reasoning, or full reasoning and mixed reasoning, on a first request segment information among the multiple request information segments to obtain first historical result information output by the full reasoning, or the full reasoning and mixed reasoning;
[0018] In response to obtaining second request information requesting to use a preset language model to infer input information, taking second request fragment information in the multiple request information segments and the second request information as the same batch request to obtain first request batch information;
[0019] Based on the first historical result information and the first request batch information, hybrid reasoning is performed using the language model to obtain at least one of the first reasoning results and second historical result information;
[0020] In response to obtaining the third request information requesting to use a preset language model to infer the input information, taking the first inference result, the third request segment information in the multiple request segments, and the third request information as a same batch request to obtain third request batch information;
[0021] Based on the second historical result information and the third request batch information, hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
[0022] Furthermore, the language model is used to perform full reasoning and mixed reasoning on the multiple request information segments and the second request information to obtain multiple first reasoning results, including:
[0023] Merging the multiple segments of request information and the second request information in batches, and performing zero-redundancy full reasoning and hybrid reasoning on the multiple segments of request information and the second request information using the language model to obtain multiple first reasoning results;
[0024] Furthermore, the multiple request information segments and the second request information are merged in batches, and the language model is used to perform zero-redundancy full reasoning and hybrid reasoning on the multiple request information segments and the second request information to obtain multiple first reasoning results, including:
[0025] Using the language model, perform full reasoning, or full reasoning and mixed reasoning, on a first request segment information among the multiple request information segments to obtain historical result information output by the full reasoning, or the full reasoning and mixed reasoning;
[0026] In response to obtaining the second request information requesting to use a preset language model to infer the input information, merging the second request fragment information in the multiple request information segments with the second request information to obtain first merged information;
[0027] Based on the historical result information and the first merged information, zero-redundancy hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
[0028] Furthermore, the zero-redundancy hybrid reasoning is performed using the language model based on the historical result information and the first merged information to obtain multiple first reasoning results, including:
[0029] Performing label embedding and layer normalization processing on the first merged information and the historical result information respectively to obtain first merged input information;
[0030] Performing a vector linear transformation on the first combined input information using the self-attention layer of the language model to obtain first vector input information;
[0031] Based on the first vector input information, using the self-attention layer to perform zero-redundancy full-quantity operation to obtain a first output result;
[0032] Performing full feed-forward processing on the first output result using the feed-forward layer of the language model to obtain a first intermediate result;
[0033] Based on N first merged input information, first vector input information, first output result and first intermediate result, multiple first inference results are obtained, where N is a natural number greater than 1.
[0034] Furthermore, based on the first vector input information, the self-attention layer is used to perform zero-redundancy full-quantity operation to obtain a first output result, including:
[0035] Distribute the first vector input information into multiple independent operators in the self-attention layer according to a preset partitioning strategy;
[0036] Using the multiple independent operators to respectively perform full operations on the first vector input information that is input in a scattered manner, to obtain multiple first operation results, and merging the multiple first operation results into a preset video memory corresponding to the language model;
[0037] The multiple first operation results merged and written into the preset video memory are split to obtain a first output result.
[0038] Further, the merging of the multiple request information segments and the second request information in batches, performing zero-redundancy full reasoning and hybrid reasoning on the multiple request information segments and the second request information using the language model to obtain multiple first reasoning results, further includes:
[0039] In response to obtaining third request information requesting to use a preset language model to infer input information, merging the first inference result and the third request information to obtain second merged information;
[0040] Based on the second merged information, zero-redundancy hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
[0041] Furthermore, the multiple request information segments and the second request information are merged in batches, and the language model is used to perform zero-redundancy full reasoning and hybrid reasoning on the multiple request information segments and the second request information to obtain multiple first reasoning results, including:
[0042] Using the language model, perform full reasoning, or full reasoning and mixed reasoning, on a first request segment information among the multiple request information segments to obtain first historical result information output by the full reasoning, or the full reasoning and mixed reasoning;
[0043] In response to obtaining the second request information requesting to use a preset language model to infer the input information, merging the second request fragment information in the multiple request information segments with the second request information to obtain first merged information;
[0044] Based on the first historical result information and the first merged information, using the language model to perform zero-redundancy hybrid reasoning to obtain at least one of the first reasoning results and the second historical result information;
[0045] In response to obtaining third request information requesting to use a preset language model to infer input information, merging the first inference result, the third request segment information in the multiple segments of request information, and the third request information to obtain third merged information;
[0046] Based on the second historical result information and the third merged information, zero-redundancy hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
[0047] The embodiment of the present application also provides a language model reasoning optimization device, the reasoning optimization device comprising:
[0048] A segmentation processing module, configured to, in response to obtaining at least one request information requesting to use a preset language model to infer input information, divide the first request information into a plurality of segments of request information according to a preset length;
[0049] a hybrid reasoning module, configured to perform full reasoning and hybrid reasoning on the multiple request information segments and the second request information using the language model to obtain multiple first reasoning results, wherein the second request information is at least one request information obtained simultaneously with the first request information or obtained during the full reasoning or hybrid reasoning process on the multiple request information segments;
[0050] An incremental reasoning module is used to perform incremental reasoning on multiple first reasoning results using the language model to obtain a reasoning result output by the language model when reasoning on the input information.
[0051] An embodiment of the present application also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the language model reasoning optimization method as described above are performed.
[0052] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned language model reasoning optimization method are executed.
[0053] The embodiment of the present application provides a method for optimizing the reasoning of a language model, and the method comprises: in response to obtaining at least one request information requesting to use a preset language model to reason about input long text information, dividing the first request information into multiple request information segments according to a preset length; using the language model to perform full reasoning and mixed reasoning on the multiple request information segments and the second request information to obtain multiple first reasoning results, wherein the second request information is at least one request information obtained at the same time as the first request information or obtained during the full reasoning or mixed reasoning process on the multiple request information segments; using the language model to perform incremental reasoning on multiple first reasoning results to obtain the reasoning result output by the language model when reasoning about the input information.
[0054] Compared with the method in the prior art of using language models to process ultra-long input of long text requests, based on the idea of continuous batch processing, at least one request information is subjected to segmented full-scale reasoning and hybrid reasoning, thereby achieving a smaller occupation of video memory when the language model performs reasoning on the input long text information, improving the throughput of the language model reasoning service in practical applications, and optimizing the video memory occupation and model reasoning performance during the reasoning process through zero-redundancy processing in full-scale reasoning and hybrid reasoning, thereby improving the efficiency of the language model in performing computational reasoning.
[0055] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0057] Figure 1 A hierarchical abstract schematic diagram of a reasoning framework of a language model provided in an embodiment of the present application;
[0058] Figure 2 A flowchart of a method for optimizing the reasoning of a language model provided in an embodiment of the present application;
[0059] Figure 3 A segmented schematic diagram of full-scale reasoning of a language model provided in an embodiment of the present application;
[0060] Figure 4 A schematic diagram of reasoning for zero-redundancy processing of a language model provided in an embodiment of the present application;
[0061] Figure 5 A schematic diagram of the structure of a language model reasoning optimization device provided in an embodiment of the present application;
[0062] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0063] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.
[0064] At present, the classic architecture of the Transformer model in the language model includes three main components: the Masked Multi-Head Attention part, which is used to establish the dependency relationship between the positions in the input sequence; the residual connection and normalization operation part (Add&Norm); the feedforward neural network part (Feed Forward) and the multi-layer perceptron (MLP) part. Through the collaboration of these components, the Transformer model can effectively capture the key information in the input sequence and achieve powerful sequence modeling capabilities.
[0065] The study found that the intermediate memory that accounts for a relatively large proportion in the reasoning process based on the Transformer model structure mainly includes the input and output parts of the model, the output part of the self-attention layer, and the temporary memory part required by the feedforward layer in the process of first increasing the dimension and then reducing the dimension; among them, the proportion of memory is linearly related to the length of the text token processed by the language model. The size of the intermediate memory part of the language model in the reasoning process has an important impact on processing long text inputs. In large-scale model reasoning, storage resources have become an important competitive factor for high-performance computing. When processing ultra-long inputs, the occupancy of memory resources is very tight, which limits the progress of the language model in reasoning about long texts.
[0066] Here, when the language model based on the Transformer model structure receives an extremely long input of a long text request, the language model may take several seconds to perform full inference. The extremely long input cannot be incrementally inferred at the same time as the short input, which makes the waiting time for incremental inference too long, reducing the efficiency of the language model when performing computational inference. In addition, in large-scale model inference, storage resources have become an important competitive factor for high-performance computing. When processing extremely long inputs, the occupancy of video memory resources is very tight, which limits the progress of the language model in performing long-text inference.
[0067] In response to at least one of the above problems, an embodiment of the present application provides a method for optimizing reasoning of a language model.
[0068] In the embodiment of the present application, when the preset language model is inferring the input long text information, the overall service framework of the language model includes but is not limited to the application business layer, the inference service processor and the inference engine, etc. Among them, the application business layer includes a variety of application businesses, for example, implementing application businesses such as conversation chat, intelligent assistant and creation; the inference service processor is responsible for processing multiple concurrent requests, for example, the initialization of the request, the call of the inference engine and the sampling of the inference results; the inference engine can be classified according to the characteristics of the inference, for example, full inference, incremental inference and hybrid inference.
[0069] Figure 1A hierarchical abstract schematic diagram of a reasoning framework of a language model provided in an embodiment of the present application.
[0070] like Figure 1 As shown in the figure, this example includes three abstract layers. The first abstract layer (the top layer) refers to the model abstraction, which includes but is not limited to the full reasoning process of the execution context, the incremental reasoning process of the autoregression, and the hybrid reasoning including the full reasoning and incremental reasoning; the second abstract layer (the middle layer) refers to the model layer abstraction used to execute all operations of the same layer, and the model layer abstraction mainly includes but is not limited to Self Attention and FeedForward; the third abstract layer (the bottom layer) refers to the operator abstraction, which includes but is not limited to FMHA (multi-head self-attention), linear transformation (Linear), addition and normalization (Add&Norm) and softmax and other parts.
[0071] First, the full reasoning, hybrid reasoning, and incremental reasoning involved in the embodiments of this application are introduced:
[0072] Full reasoning means that when a language model processes input text data information, it will perform a one-time comprehensive analysis and processing of the entire data set stored in the model to generate a complete output or result. During the processing, the language model will consider all information and context related to the input text data information to ensure the accuracy and comprehensiveness of the output.
[0073] As an example of full reasoning, suppose a user inputs a complex question to a chatbot based on a language model. The chatbot can perform full reasoning in the following way: first, the chatbot analyzes the grammatical structure of the question and understands what the user wants to ask; then, the chatbot searches its own knowledge base to find information and answers related to the question; finally, the chatbot integrates this information to generate a complete, coherent and accurate answer and presents it to the user.
[0074] Hybrid reasoning refers to a strategy that flexibly selects and combines methods between full reasoning and incremental reasoning according to different parts or stages of the problem when solving complex problems.
[0075] Incremental reasoning is suitable for scenarios where the input data does not change much or the reasoning task needs to be repeated. When the input data only changes locally, only the changed part is re-reasoned instead of repeating the reasoning on the entire data set or input. Incremental reasoning can significantly reduce redundant calculations, reduce the amount of calculations and power consumption, thereby improving the execution speed and efficiency of reasoning.
[0076] As an example of incremental reasoning, suppose that in a video target detection task, there is a large amount of similar information between consecutive frames in the video. In this case, incremental reasoning can be performed only on areas with large differences between frames. In this way, repeated processing of the entire frame image can be reduced.
[0077] See also Figure 2 , Figure 2 A flowchart of a method for optimizing the reasoning of a language model provided in an embodiment of the present application.
[0078] like Figure 2 As shown in , the reasoning optimization method provided by the embodiment of the present application specifically includes:
[0079] Step S100: In response to obtaining at least one request information requesting to use a preset language model to infer input information, the first request information is divided into multiple request information segments according to a preset length.
[0080] In an embodiment of the present application, when inputting information to a preset language model and requesting the language model to infer the input information, at least one request information may be obtained in response to the request. Here, the input information includes but is not limited to long text information, short text information, context information, metadata, feature representation, and other information. The at least one request information refers to one or more information determined based on the information input to the language model, and the request information is used to instruct the language model to infer the input information.
[0081] In addition, the first request information is the request information that needs to be split in the at least one request information. The preset length can be specifically calibrated and set according to the preset language model performance and the input long text information.
[0082] In this step, during the specific implementation, first, at least one request information of an external request to infer the input information input into a preset language model is obtained; then, in response to the at least one request information being obtained, a preset length corresponding to the external request is obtained; finally, the first request information in the at least one request information is divided into multiple request information segments according to the preset length.
[0083] For example, assuming that the length of the first request information is 4000 and the preset length is 2000, the first request information with a length of 4000 can be divided into two request information segments according to the preset length of 2000, wherein the length of each request information segment is 2000.
[0084] Step S200: Use the language model to perform full reasoning and mixed reasoning on the multiple segments of request information and the second request information to obtain multiple first reasoning results, wherein the second request information is obtained at the same time as the first request information or is at least one request information obtained during the full reasoning or mixed reasoning process on the multiple segments of request information.
[0085] Since it is necessary to infer multiple segments of request information and multiple batches of request information in the embodiment of the present application, the processing method for the multiple segments of request information and the second request information may include two processing methods: batch parallel processing or combined processing.
[0086] In one approach, in order to improve the efficiency of a language model in processing input information, the information may be processed by processing multiple segments of request information and second request information in batches and in parallel.
[0087] Another way is to reduce resource usage while improving the processing efficiency of the language model on the input information, by merging multiple request information and the second request information.
[0088] Furthermore, considering that there are multiple division situations of the multiple request information segments divided by the first request information, one situation is that the multiple request information segments are divided into N segments, and another situation is that the multiple request information segments are divided into N+1 segments. N is a natural number greater than 1.
[0089] Based on this, for each of the two processing methods of batch parallel processing or combined processing, there are two cases in which multiple request information segments are divided into N pieces and another case in which multiple request information segments are divided into N+1 pieces.
[0090] The following will describe two situations in two processing modes in step S200 with specific examples.
[0091] (1) The processing method is batch parallel processing:
[0092] In the first case of batch parallel processing, for example, assuming that the multiple request information is divided into two, the language model is used to perform full reasoning and mixed reasoning on the multiple request information and the second request information to obtain multiple first reasoning results, which may specifically include:
[0093] Step S211: Use the language model to perform full reasoning, or full reasoning and mixed reasoning, on the first request segment information in the multiple request information segments to obtain historical result information output by the full reasoning, or the full reasoning and mixed reasoning.
[0094] In an embodiment of the present application, the specific reasoning method to be performed by the language model when reasoning about the request information is determined based on whether the acquired request information contains historical information and the input length of the request information.
[0095] Specifically, when the request information does not contain historical information, full reasoning is performed on the request information; when the request information contains historical information and the input length of the request information is 1, incremental reasoning is performed on the request information; when the request information contains historical information and the input length of the request information is not 1, mixed reasoning is performed on the request information.
[0096] In this step, based on whether the first request fragment information in the divided multi-segment request information contains historical result information and the input length of the first request fragment information, it is determined whether to perform full reasoning, or full reasoning and mixed reasoning on the first request fragment; then, the historical result information output by the full reasoning, or the full reasoning and the mixed reasoning, is obtained.
[0097] For example, assuming that request information A does not contain historical information and the input length is 2000, full inference is performed on request information A.
[0098] For another example, assuming that request information B and request information C are obtained, request information B contains historical information and the input length is 1, and incremental reasoning is performed on request information B; request information C contains historical information and the input length is not 1, and hybrid reasoning is performed on request information C.
[0099] Here, the first request segment information is not the first segment in a physical sense, but information that needs to be inferred after the first request information is segmented and before the second request information is received.
[0100] Specifically, in the process of executing full reasoning and full reasoning with historical information, the fragment information of executing full reasoning and the fragment information of full reasoning with historical information (which can be called hybrid reasoning) are both determined as the first requested fragment information.
[0101] As an example of performing full reasoning on the first request fragment information, the reasoning of the first request fragment information in the multiple request information is performed through the statement "model.run_gpt(input_tokens=split_1_2000)", wherein the length (tokens) of the first request information is 2000. Further, the full reasoning of the first request fragment information is performed through the statement "input_lengths=
[2000] ; resource_indices=[0]; infer_categories=[0]; history_lengths=[0]; is_first_finished_list=[0]", wherein the first request fragment information uses the resource with subscript 0, and the length of the historical result information is 0.
[0102] Figure 3 A segmented schematic diagram of full-scale reasoning of a language model provided in an embodiment of the present application.
[0103] like Figure 3 As shown in , when the language model performs full reasoning on the request information, it is assumed that the first request information with an input length of 8000 is divided into four request information with a length of 2000, and by adding a full reasoning with historical information, executing a full reasoning with an input length of 8000 is converted into executing four full reasonings with historical information; wherein, the length of the historical information of the first full reasoning is 0. In this way, since the size of the intermediate video memory is linearly related to the length of the processed text token, the segmented processing of the full reasoning can optimize the peak video memory occupancy.
[0104] In such Figure 3 In the example shown in , full inference through information segmentation reduces the system's intermediate video memory usage by three quarters. Compared with the common method of optimizing video memory by offloading the host device, full inference through segmentation avoids the overhead of copying data between devices and the problem of complex heterogeneous scheduling implementation.
[0105] Step S212: In response to obtaining the second request information requesting to use a preset language model to infer the input information, the second request segment information in the multiple request information segments and the second request information are taken as the same batch request to obtain first request batch information.
[0106] In an embodiment of the present application, during the process of performing full reasoning, or full reasoning and mixed reasoning on the first request fragment information in the first request information, the second request information requesting to use a preset language model to reason about the input information is obtained. At this time, the second request fragment information in the first request information can be merged with the second request information into the same batch for parallel processing.
[0107] For example, suppose that during the full inference, or the full inference and mixed inference process of the D1 request fragment information in the D request information, the E request information is obtained. At this time, the D2 request fragment information in the D request information needs to be merged with the E request information into the same batch request so as to process the same batch request in parallel.
[0108] Here, it should be understood that when at least one request information is obtained, the first request information needs to be immediately divided into multiple request information segments, and the second request fragment information in the multiple request information segments and the second request information need to be further merged into the same request batch information to process the request batch information in parallel.
[0109] The at least one request information is not necessarily acquired at one time, and may be second request information acquired during the processing of the first request information.
[0110] Step S213: Based on the historical result information and the first request batch information, hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
[0111] In this step, during the specific implementation, the historical result information and the first request batch information are used as inputs of the language model. At this time, since the input contains historical result information and the input length is not 1, the language model can be used to perform mixed reasoning on the input to obtain multiple first reasoning results.
[0112] Optionally, in addition to using the language model to perform full reasoning and mixed reasoning on the multiple request information and the second request information to obtain multiple first reasoning results as described in the above steps, steps S214 to S215 are also included. Specifically, steps S214 to S215 are used to illustrate a reasoning method for obtaining the third request information after reasoning on the obtained second request information, and specifically include:
[0113] Step S214: In response to obtaining the third request information requesting to use a preset language model to infer the input information, the plurality of first inference results and the third request information are taken as the same batch request to obtain second request batch information.
[0114] In an embodiment of the present application, when the second request information is obtained and the third request information is obtained in the process of reasoning the first request fragment information in the first request information using a language model, the first reasoning result and the third request information can be taken as the same batch request to obtain the second request batch information.
[0115] For example, suppose that the D2 request fragment information in the D request information and the E request information are merged into the first request batch information, and the historical result information obtained by reasoning the D1 request fragment information in the D request information and the first request batch information are input into the language model for mixed reasoning to obtain multiple first reasoning results S1. At this time, the F request information is obtained, and the first reasoning result S1 and the F request information are taken as the same batch request to obtain the second request batch information.
[0116] Step S215: Based on the second request batch information, hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
[0117] In this step, during the specific implementation, the second request batch information is used as the input of the language model. At this time, since the input contains historical result information and the input length is not 1, the language model can be used to perform mixed reasoning on the input to obtain multiple first reasoning results corresponding to the first request information, the second request information and the third request information.
[0118] As an example of the first case in the batch parallel processing method, first, the D request information obtained first is segmented to obtain D1 request fragment information and D2 request fragment information; then, the D1 request fragment information is inferred to obtain historical result information Z, and the E request information is obtained in the process; thereafter, the D2 request fragment information and the E request information need to be merged into the first request batch information, and the historical result information Z and the first request batch information are input into the language model for mixed reasoning to obtain multiple first reasoning results S1; at this time, the F request information is obtained again, and the first reasoning result S1 and the F request information are taken as the same batch request to obtain the second request batch information; finally, the second request batch information is input into the language model for mixed reasoning to obtain multiple first reasoning results S2.
[0119] In the second case of batch parallel processing, for example, assuming that the multiple request information is divided into three segments, the language model is used to perform full reasoning and mixed reasoning on the multiple request information and the second request information to obtain multiple first reasoning results, which may specifically include:
[0120] Step S221: Use the language model to perform full reasoning, or full reasoning and mixed reasoning, on the first request segment information in the multiple request information segments to obtain first historical result information output by the full reasoning, or the full reasoning and mixed reasoning.
[0121] Step S222: in response to obtaining the second request information requesting to use a preset language model to infer the input information, taking the second request segment information in the multiple request information segments and the second request information as the same batch request to obtain first request batch information;
[0122] Step S223: Based on the first historical result information and the first request batch information, hybrid reasoning is performed using the language model to obtain at least one of the first reasoning results and second historical result information;
[0123] Among them, the description of steps S221 to S223 can refer to the description of steps S211 to S213, and can achieve the same technical effect, which will not be repeated here.
[0124] Step S224, in response to obtaining the third request information requesting to use a preset language model to infer the input information, the first inference result, the third request fragment information in the multiple request information segments, and the third request information are taken as the same batch request to obtain third request batch information.
[0125] In an embodiment of the present application, since the third request fragment information is included in multiple request information segments, the third request fragment information is divided and the third request fragment information is taken as the same batch request together with the first inference result and the third request information to obtain the third request batch information, so as to perform mixed inference on the third request batch information.
[0126] Step S225: Based on the second historical result information and the third request batch information, hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
[0127] In this step, the second historical result information and the third request batch information are used as inputs of the language model, and the language model is used to perform mixed reasoning on the inputs to obtain multiple first reasoning results.
[0128] As an example of the second case in the batch parallel processing method, first, the D request information obtained first is segmented to obtain D1 request fragment information, D2 request fragment information and D3 request fragment information; then, the D1 request fragment information is inferred to obtain the first historical result information Z1, and in the process, the E request information is obtained, and the D2 request fragment information and the E request information need to be merged into the first request batch information; then, the first historical result information Z1 and the first request batch information are input into the language model for mixed reasoning to obtain at least one first reasoning result S1 and the second historical result information Z2; then, the F request information is obtained, and the first reasoning result S1, the D3 request fragment information, and the F request information are taken as the same batch request to obtain the third request batch information; finally, the third request batch information and the second historical result information Z2 are input into the language model for mixed reasoning to obtain multiple first reasoning results S2.
[0129] (2) The processing method is a combined processing method:
[0130] In the first case of the combined processing method, for example, assuming that the multiple request information is divided into two, the language model is used to perform full reasoning and mixed reasoning on the multiple request information and the second request information to obtain multiple first reasoning results, which may specifically include:
[0131] Step S230: merge the multiple segments of request information and the second request information in batches, and use the language model to perform zero-redundancy full reasoning and hybrid reasoning on the multiple segments of request information and the second request information to obtain multiple first reasoning results.
[0132] In one implementation of the present application, during specific implementation, step S230 may include:
[0133] Step S231: Use the language model to perform full reasoning, or full reasoning and mixed reasoning, on the first request segment information in the multiple request information segments to obtain historical result information output by the full reasoning, or the full reasoning and the mixed reasoning.
[0134] Among them, the description of S231 can refer to the description of S211, and can achieve the same technical effect, which is not repeated here.
[0135] Step S232: In response to obtaining the second request information requesting to use a preset language model to infer the input information, the second request fragment information in the multiple request information segments is merged with the second request information to obtain first merged information.
[0136] In an embodiment of the present application, during the process of reasoning the first request fragment information, the second request information is obtained, and the second request fragment information can be combined with the second request information to reduce the resource usage of the language model reasoning through zero-redundancy reasoning.
[0137] In an example of the present application, the second request information is obtained through the statement "request_2=GetLastRequest()", and the second request fragment information and the second request information are merged through the statement "input_tokens=Merge(split_2_2000,request_2)" to obtain the first merged information.
[0138] For example, suppose that H request information is obtained during the process of reasoning about G1 request fragment information in G request information. At this time, the G1 request fragment information and the H request information need to be merged into first merged information to perform zero-redundancy reasoning on the first merged information.
[0139] Step S233: Based on the historical result information and the first merged information, use the language model to perform zero-redundancy hybrid reasoning to obtain multiple first reasoning results.
[0140] In the embodiment of the present application, the language model will perform reasoning in N layers during the reasoning process, including multiple hidden layers in the N layers, and the multiple hidden layers are multi-level abstractions of input features to linearly divide the data of the input layer into different types of data. N is a natural number greater than 1.
[0141] In one embodiment of the present application, in a specific implementation, in step S233, zero-redundancy mixed reasoning is performed on the vectors included in the hidden layer to obtain multiple first reasoning results, specifically including:
[0142] S2331. Perform label embedding and layer normalization processing on the first merged information and the historical result information respectively to obtain first merged input information.
[0143] In this step, during the specific implementation, first, the first merged information and historical result information are embedded with model labels; then, the data distribution characteristics included in the information embedded with the model labels are calculated, and the data distribution characteristics are adjusted to a specified range for normalization processing; finally, a preset scaling factor (scale) and translation factor (shift) are introduced into the normalized data to obtain the first merged input information.
[0144] S2332. Use the self-attention layer of the language model to perform vector linear transformation on the first merged input information to obtain first vector input information.
[0145] In this step, the self-attention layer of the language model is used to transform the first merged input information through three different linear transformation methods. Specifically, the first merged input information is multiplied by three different weight matrices to obtain three vectors Q, K, and V, and these three vectors are determined as the first vector input information.
[0146] Among them, the three different weight matrices can correspond to the trainable parameter matrix W Q , W K , W V .
[0147] S2333. Based on the first vector input information, use the self-attention layer to perform zero-redundancy full-quantity operations to obtain a first output result.
[0148] In one implementation of the present application, during specific implementation, step S2333 may include:
[0149] Step S23331: Distribute the first vector input information into multiple independent operators in the self-attention layer according to a preset division strategy.
[0150] In this step, based on the amount of information in the first vector input information, the first vector input information is sequentially and dispersedly input into multiple independent operators in the self-attention layer.
[0151] For example, assuming that the first vector input information includes information whose lengths are 1000, 1, 1000, and 1 respectively, the first vector input information is input into four independent operators respectively to perform subsequent operator operations.
[0152] Step S23332: Use the multiple independent operators to perform full operations on the first vector input information that is separately input, obtain multiple first operation results, and merge the multiple first operation results and write them into the preset video memory corresponding to the language model.
[0153] In the embodiment of the present application, after processing the entire input sequence, the language model will generate multiple first operation results at one time, and merge the multiple first operation results into a preset video memory corresponding to the language model.
[0154] Step S23333: split the multiple first operation results merged and written into the preset video memory to obtain a first output result.
[0155] In this step, according to the input token information after the request processing, multiple first operation results are split and processed inside the preset video memory to obtain the first output result.
[0156] Step S2334: Use the feedforward layer of the language model to perform full feedforward processing on the first output result to obtain a first intermediate result.
[0157] In an embodiment of the present application, based on the data processing capability of the feedforward layer in the language model, the first output result is fully feedforward processed to obtain a first intermediate result, and the first intermediate result is operated on through other processing included in the language model.
[0158] Step S2335: Based on N of the first merged input information, the first vector input information, the first output result and the first intermediate result, obtain multiple first inference results, where N is a natural number greater than 1.
[0159] In an embodiment of the present application, the hidden layer vector includes but is not limited to first merged input information, first vector input information, a first output result and a first intermediate result.
[0160] Here, it should be understood that since no inference result is obtained after the feedforward layer processes the entire first output result, after the model runs all layers, the language model performs other processing on the hidden layer vector to obtain multiple of the first inference results.
[0161] Figure 4 A reasoning diagram of zero-redundancy processing of a language model provided in an embodiment of the present application.
[0162] As an example of zero-redundancy reasoning, Figure 4 As shown in , it is assumed that the first merged information includes 4 input information, whose lengths are 1000, 1, 1000, and 1 respectively, and the shape size of the first merged information input is 2002; after the first merged information is embedded (Embedding) and layer normalized (Layer Norm) respectively, since the above processing is for the text token (Token) marking of each input information, the obtained intermediate video memory shape size is [2002, H] (H represents the width of the hidden layer).
[0163] Furthermore, in the reasoning process of the self-attention layer (Self-Attention), since the attn operator for calculating the attention score is for each input information in the first merged information, after the first merged input information is vector linearly transformed (QKV linear), the first vector input information is dispersed into 4 independent operators (attn1-4) for execution, so that the obtained multiple first operation results are internally split and written into the same preset video memory; then, the first output result is fully feed-forwarded using the feed-forward layer (feed forward) of the language model to obtain a first intermediate result; finally, based on the first merged input information, the first vector input information, the first output result and the first intermediate result, multiple first inference results are obtained through processing by other layers of the model.
[0164] By performing zero-redundancy processing at the input end of the language model, separation and merging operations are implemented in the operator operation part, thereby reducing the occupation of temporary video memory by the language model during reasoning.
[0165] Optionally, in addition to the full reasoning and mixed reasoning of merging the multiple request information and the second request information using the language model as described in the above steps to obtain multiple first reasoning results, steps S231 to S233 are also included. Specifically, steps S234 to S235 are used to illustrate the reasoning method for obtaining the third request information after reasoning on the obtained second request information, which specifically includes:
[0166] Step S234: In response to obtaining third request information requesting to use a preset language model to infer the input information, the first inference result and the third request information are merged to obtain second merged information.
[0167] In an embodiment of the present application, with respect to the acquired third request information, the first inference result and the third request information may be combined to obtain second combined information, so as to perform zero-redundancy hybrid inference on the second combined information.
[0168] Step S235: Based on the second merged information, use the language model to perform zero-redundancy hybrid reasoning to obtain multiple first reasoning results.
[0169] Among them, the description of step S235 can refer to the description of steps S2331 to S2335, and can achieve the same technical effect, which will not be repeated here.
[0170] As an example of the first case in the merging processing method, first, the G request information obtained first is segmented to obtain G1 request fragment information and G2 request fragment information; then, the G1 request fragment information is inferred to obtain historical result information Y, and H request information is obtained in the process; thereafter, the G2 request fragment information and the H request information need to be merged into the first merged information, and the historical result information Y and the first merged information are input into the language model for zero-redundant mixed reasoning to obtain multiple first reasoning results X1; at this time, the I request information is obtained again, and the first reasoning result X1 and the I request information are merged to obtain the second merged information; finally, the second merged information is input into the language model for zero-redundant mixed reasoning to obtain multiple first reasoning results X2.
[0171] In the second case of the combined processing method, for example, assuming that the multiple request information is divided into three, the language model is used to perform full reasoning and mixed reasoning on the multiple request information and the second request information to obtain multiple first reasoning results, which may specifically include:
[0172] Step S241: Use the language model to perform full reasoning, or full reasoning and mixed reasoning, on the first request segment information in the multiple request information segments to obtain first historical result information output by the full reasoning, or the full reasoning and mixed reasoning.
[0173] Step S242: In response to obtaining the second request information requesting to use a preset language model to infer the input information, the second request fragment information in the multiple request information segments is merged with the second request information to obtain first merged information.
[0174] Step S243: Based on the first historical result information and the first merged information, use the language model to perform zero-redundancy hybrid reasoning to obtain at least one of the first reasoning results and the second historical result information.
[0175] Among them, the description of steps S241 to S243 can refer to the description of steps S231 to S233, and can achieve the same technical effect, which is not repeated here.
[0176] Step S244: In response to obtaining third request information requesting to use a preset language model to infer the input information, the first inference result, the third request segment information in the multiple segments of request information, and the third request information are merged to obtain third merged information.
[0177] In an embodiment of the present application, for the acquired third request information, the first inference result can be merged with the third request fragment information and the third request information to obtain third merged information, so as to perform zero-redundancy mixed inference on the third merged information and the second historical result information.
[0178] Step S245: Based on the second historical result information and the third merged information, use the language model to perform zero-redundancy hybrid reasoning to obtain multiple first reasoning results.
[0179] Among them, the description of step S245 can refer to the description of steps S2331 to S2335, and can achieve the same technical effect, which is not repeated here.
[0180] As an example of the second case in the batch parallel processing method, first, the G request information obtained first is segmented to obtain G1 request fragment information, G2 request fragment information and G3 request fragment information; then, the G1 request fragment information is inferred to obtain the first historical result information Y1, and in the process, the H request information is obtained, and the G2 request fragment information and the H request information need to be merged into the first merged information; then, the first historical result information Y1 and the first merged information are input into the language model for zero-redundant hybrid reasoning to obtain at least one first reasoning result X1 and second historical result information Y2; then, the I request information is obtained, and the first reasoning result X1, D3 request fragment information, and I request information are merged into the third merged information; finally, the second historical result information Y2 and the third merged information are input into the language model for zero-redundant hybrid reasoning to obtain multiple first reasoning results X2.
[0181] Return to reference Figure 2 , step 300, using the language model to perform incremental reasoning on multiple first reasoning results to obtain the reasoning results output by the language model when reasoning on the input information.
[0182] In this step, when processing multiple segments of request information and second request information in batches in parallel, multiple first inference results are taken as results of the same batch to obtain first request batch results, and the language model is used to perform incremental inference on the first request batch results to obtain inference results output by the language model when reasoning about the input information.
[0183] When multiple segments of request information and second request information are processed by merging, this step may include: merging multiple of the first inference results to obtain merged information, and using the language model to perform zero-redundancy incremental reasoning on the merged information to obtain a merged inference result output by the language model when reasoning about the input information.
[0184] As an example of performing incremental reasoning on merged information, incremental reasoning on the merged information is performed through the statement “logits=model.run_gpt(input_tokens=input_tokens, input_lengths=[1,1,1], resource_indices=[0,1,2], infer_categories=[1,1,1], history_lengths=[4001,2001,1000], is_first_finished_list=[1,1,1])”.
[0185] Among them, input_tokens represents the list of input text tokens, whose shape is [total_length], and padding needs to be removed when used; input_lengths represents the list of input lengths, whose shape is [max_batch_size]; resource_indices represents the location list of computing resources, whose shape is [max_batch_size], and the value range is [0, max_batch_size-1]; infer_categories represents the input reasoning category list, whose shape is [max_batch_size], and the parameter 0 indicates full reasoning, and the parameter 1 indicates incremental reasoning; history_lengths represents the input of the number of tokens that have been reasoned, and its shape is [max_batch_size]; is_first_finished_list indicates whether the input has been completed in the current full reasoning, that is, whether it is the last segment of input, and its shape is [max_batch_size], and the parameter -1 indicates an invalid value (for incremental reasoning); the parameter is 0, which means that the reasoning has not yet ended; the parameter is 1, which means that the reasoning has ended.
[0186] Here, incremental reasoning is a full reasoning with historical information, and its input length is 1. That is, full reasoning with historical information naturally supports incremental reasoning. Since the difficulty of reasoning in the batch processing of inputs with a length of 1 lies in the IO bandwidth, the short input with a length of 1 is combined with the long input for reasoning, and the reasoning delay caused by the short input can be ignored.
[0187] In this way, through the hybrid reasoning process of segmented batch processing, continuous requests can be processed efficiently, and while ensuring real-time performance, the advantages of batch processing can be fully utilized to dynamically increase the batch size of incremental reasoning, so that the system can effectively utilize computing resources and improve the throughput and efficiency of reasoning.
[0188] The language model reasoning optimization method provided in the embodiment of the present application is based on the idea of continuous batch processing, and performs segmented full reasoning and hybrid reasoning on at least one request information, thereby achieving a smaller occupation of video memory when the language model performs reasoning on the input long text information, improving the throughput of the language model reasoning service in practical applications, and optimizing the video memory occupancy and model reasoning performance in the reasoning process through zero redundancy processing in full reasoning and hybrid reasoning, thereby improving the efficiency of the language model when performing computational reasoning.
[0189] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a language model reasoning optimization device provided in an embodiment of the present application. Figure 5 As shown in , the reasoning optimization device 500 includes:
[0190] The segmentation processing module 510 is used to, in response to obtaining at least one request information requesting to use a preset language model to infer the input information, divide the first request information into multiple segments of request information according to a preset length;
[0191] A hybrid reasoning module 520, configured to perform full reasoning and hybrid reasoning on the multiple request information segments and the second request information using the language model to obtain multiple first reasoning results, wherein the second request information is at least one request information obtained simultaneously with the first request information or obtained during the full reasoning or hybrid reasoning process on the multiple request information segments;
[0192] The incremental reasoning module 530 is used to perform incremental reasoning on the plurality of the first reasoning results using the language model to obtain a reasoning result output by the language model when reasoning on the input information.
[0193] Further, when the hybrid reasoning module 520 is used to perform full reasoning and hybrid reasoning on the multiple request information segments and the second request information using the language model to obtain multiple first reasoning results, the hybrid reasoning module 520 is used to:
[0194] Using the language model, perform full reasoning, or full reasoning and mixed reasoning, on a first request segment information among the multiple request information segments to obtain historical result information output by the full reasoning, or the full reasoning and mixed reasoning;
[0195] In response to obtaining the second request information requesting to use a preset language model to infer the input information, taking the second request fragment information in the multiple request information segments and the second request information as the same batch request to obtain first request batch information;
[0196] Based on the historical result information and the first request batch information, hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
[0197] Further, when the hybrid reasoning module 520 is used to perform full reasoning and hybrid reasoning on the multiple request information segments and the second request information using the language model to obtain multiple first reasoning results, the hybrid reasoning module 520 is also used to:
[0198] In response to obtaining third request information requesting to use a preset language model to infer input information, taking the plurality of first inference results and the third request information as a same batch request to obtain second request batch information;
[0199] Based on the second request batch information, hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
[0200] Further, when the hybrid reasoning module 520 is used to perform full reasoning and hybrid reasoning on the multiple request information segments and the second request information using the language model to obtain multiple first reasoning results, the hybrid reasoning module 520 is also used to:
[0201] Using the language model, perform full reasoning, or full reasoning and mixed reasoning, on a first request segment information among the multiple request information segments to obtain first historical result information output by the full reasoning, or the full reasoning and mixed reasoning;
[0202] In response to obtaining the second request information requesting to use a preset language model to infer the input information, taking the second request fragment information in the multiple request information segments and the second request information as the same batch request to obtain first request batch information;
[0203] Based on the first historical result information and the first request batch information, hybrid reasoning is performed using the language model to obtain at least one of the first reasoning results and second historical result information;
[0204] In response to obtaining third request information requesting to use a preset language model to infer input information, taking the first inference result, the third request segment information in the multiple request segments, and the third request information as a same batch request to obtain third request batch information;
[0205] Based on the second historical result information and the third request batch information, hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
[0206] Further, when the hybrid reasoning module 520 is used to perform full reasoning and hybrid reasoning on the multiple request information segments and the second request information using the language model to obtain multiple first reasoning results, the hybrid reasoning module 520 is also used to:
[0207] Merging the multiple segments of request information and the second request information in batches, and performing zero-redundancy full reasoning and hybrid reasoning on the multiple segments of request information and the second request information using the language model to obtain multiple first reasoning results;
[0208] Further, when the hybrid reasoning module 520 is used to merge the multiple segments of request information and the second request information in batches, and use the language model to perform zero-redundancy full reasoning and hybrid reasoning on the multiple segments of request information and the second request information to obtain multiple first reasoning results, the hybrid reasoning module 520 is also used to:
[0209] Using the language model, perform full reasoning, or full reasoning and mixed reasoning, on a first request segment information among the multiple request information segments to obtain historical result information output by the full reasoning, or the full reasoning and mixed reasoning;
[0210] In response to obtaining the second request information requesting to use a preset language model to infer the input information, merging the second request fragment information in the multiple request information segments with the second request information to obtain first merged information;
[0211] Based on the historical result information and the first merged information, zero-redundancy hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
[0212] Further, when the hybrid reasoning module 520 is used to perform zero-redundancy hybrid reasoning using the language model based on the historical result information and the first merged information to obtain multiple first reasoning results, the hybrid reasoning module 520 is also used to:
[0213] Performing label embedding and layer normalization processing on the first merged information and the historical result information respectively to obtain first merged input information;
[0214] Performing a vector linear transformation on the first combined input information using the self-attention layer of the language model to obtain first vector input information;
[0215] Based on the first vector input information, using the self-attention layer to perform zero-redundancy full-quantity operation to obtain a first output result;
[0216] Performing full feed-forward processing on the first output result using the feed-forward layer of the language model to obtain a first intermediate result;
[0217] Based on N first merged input information, first vector input information, first output result and first intermediate result, multiple first inference results are obtained, where N is a natural number greater than 1.
[0218] Further, when the hybrid reasoning module 520 is used to perform zero-redundancy full-quantity operation based on the first vector input information using the self-attention layer to obtain a first output result, the hybrid reasoning module 520 is also used to:
[0219] Distribute the first vector input information into multiple independent operators in the self-attention layer according to a preset partitioning strategy;
[0220] Using the multiple independent operators to respectively perform full operations on the first vector input information that is input in a scattered manner, to obtain multiple first operation results, and merging the multiple first operation results into a preset video memory corresponding to the language model;
[0221] The multiple first operation results merged and written into the preset video memory are split to obtain a first output result.
[0222] Further, when the hybrid reasoning module 520 is used to merge the multiple segments of request information and the second request information in batches, and use the language model to perform zero-redundancy full reasoning and hybrid reasoning on the multiple segments of request information and the second request information to obtain multiple first reasoning results, the hybrid reasoning module 520 is also used to:
[0223] In response to obtaining third request information requesting to use a preset language model to infer input information, merging the first inference result and the third request information to obtain second merged information;
[0224] Based on the second merged information, zero-redundancy hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
[0225] Further, when the hybrid reasoning module 520 is used to merge the multiple segments of request information and the second request information in batches, and use the language model to perform zero-redundancy full reasoning and hybrid reasoning on the multiple segments of request information and the second request information to obtain multiple first reasoning results, the hybrid reasoning module 520 is also used to:
[0226] Using the language model, perform full reasoning, or full reasoning and mixed reasoning, on a first request segment information among the multiple request information segments to obtain first historical result information output by the full reasoning, or the full reasoning and mixed reasoning;
[0227] In response to obtaining the second request information requesting to use a preset language model to infer the input information, merging the second request fragment information in the multiple request information segments with the second request information to obtain first merged information;
[0228] Based on the first historical result information and the first merged information, using the language model to perform zero-redundancy hybrid reasoning to obtain at least one of the first reasoning results and the second historical result information;
[0229] In response to obtaining third request information requesting to use a preset language model to infer input information, merging the first inference result, the third request segment information in the multiple segments of request information, and the third request information to obtain third merged information;
[0230] Based on the second historical result information and the third merged information, zero-redundancy hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
[0231] The language model reasoning optimization device provided in the embodiment of the present application performs segmented full reasoning and hybrid reasoning on at least one request information based on the idea of continuous batch processing, thereby achieving a smaller occupation of video memory when the language model performs reasoning on input long text information, improving the throughput of the language model reasoning service in practical applications, and optimizing the video memory occupancy and model reasoning performance during the reasoning process through zero-redundancy processing in full reasoning and hybrid reasoning, thereby improving the efficiency of the language model when performing computational reasoning.
[0232] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown in , the electronic device 600 includes a processor 610 , a memory 620 and a bus 630 .
[0233] The memory 620 stores machine-readable instructions executable by the processor 610. When the electronic device 600 is running, the processor 610 communicates with the memory 620 via the bus 630. When the machine-readable instructions are executed by the processor 610, the above-mentioned Figure 2 The steps of the method for optimizing the reasoning of the language model in the method embodiment shown, and the specific implementation method thereof can be found in the method embodiment, which will not be described in detail here.
[0234] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 2The steps of the method for optimizing the reasoning of the language model in the method embodiment shown, and the specific implementation method thereof can be found in the method embodiment, which will not be described in detail here.
[0235] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0236] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0237] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0238] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0239] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0240] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A method for optimizing reasoning of a language model, characterized in that: The reasoning optimization method comprises: In response to obtaining at least one request information requesting to use a preset language model to infer input information, dividing the first request information into a plurality of request information segments according to a preset length; Using the language model to perform full reasoning and mixed reasoning on the multiple segments of request information and the second request information, to obtain multiple first reasoning results, wherein the second request information is at least one request information obtained at the same time as the first request information or obtained in the process of performing full reasoning or mixed reasoning on the multiple segments of request information; Incremental reasoning is performed on the plurality of first reasoning results using the language model to obtain a reasoning result output by the language model when reasoning on the input information.
2. The method according to claim 1, characterized in that The method of using the language model to perform full reasoning and mixed reasoning on the multiple request information segments and the second request information to obtain multiple first reasoning results includes: Using the language model, perform full reasoning, or full reasoning and mixed reasoning, on a first request segment information among the multiple request information segments to obtain historical result information output by the full reasoning, or the full reasoning and mixed reasoning; In response to obtaining the second request information requesting to use a preset language model to infer the input information, taking the second request fragment information in the multiple request information segments and the second request information as the same batch request to obtain first request batch information; Based on the historical result information and the first request batch information, hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
3. The method according to claim 1, characterized in that The method of using the language model to perform full reasoning and mixed reasoning on the multiple request information segments and the second request information to obtain multiple first reasoning results also includes: In response to obtaining third request information requesting to use a preset language model to infer input information, taking the plurality of first inference results and the third request information as a same batch request to obtain second request batch information; Based on the second request batch information, hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
4. The method according to claim 1, characterized in that The method of using the language model to perform full reasoning and mixed reasoning on the multiple request information segments and the second request information to obtain multiple first reasoning results includes: Using the language model, perform full reasoning, or full reasoning and mixed reasoning, on a first request segment information among the multiple request information segments to obtain first historical result information output by the full reasoning, or the full reasoning and mixed reasoning; In response to obtaining the second request information requesting to use a preset language model to infer the input information, taking the second request fragment information in the multiple request information segments and the second request information as the same batch request to obtain first request batch information; Based on the first historical result information and the first request batch information, hybrid reasoning is performed using the language model to obtain at least one of the first reasoning results and second historical result information; In response to obtaining third request information requesting to use a preset language model to infer input information, taking the first inference result, the third request segment information in the multiple request segments, and the third request information as a same batch request to obtain third request batch information; Based on the second historical result information and the third request batch information, hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
5. The method according to claim 1, characterized in that The method of using the language model to perform full reasoning and mixed reasoning on the multiple request information segments and the second request information to obtain multiple first reasoning results includes: The multiple segments of request information and the second request information are merged in batches, and zero-redundancy full reasoning and hybrid reasoning are performed on the multiple segments of request information and the second request information using the language model to obtain multiple first reasoning results.
6. The method according to claim 5, characterized in that The step of merging the multiple request information segments and the second request information in batches, and performing zero-redundancy full-volume reasoning and hybrid reasoning on the multiple request information segments and the second request information using the language model to obtain multiple first reasoning results, including: Using the language model, perform full reasoning, or full reasoning and mixed reasoning, on a first request segment information among the multiple request information segments to obtain historical result information output by the full reasoning, or the full reasoning and mixed reasoning; In response to obtaining the second request information requesting to use a preset language model to infer the input information, merging the second request fragment information in the multiple request information segments with the second request information to obtain first merged information; Based on the historical result information and the first merged information, zero-redundancy hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
7. The method according to claim 6, characterized in that The method of performing zero-redundancy hybrid reasoning based on the historical result information and the first merged information using the language model to obtain a plurality of the first reasoning results includes: Performing label embedding and layer normalization processing on the first merged information and the historical result information respectively to obtain first merged input information; Performing a vector linear transformation on the first combined input information using the self-attention layer of the language model to obtain first vector input information; Based on the first vector input information, using the self-attention layer to perform zero-redundancy full-quantity operation to obtain a first output result; Performing full feed-forward processing on the first output result using the feed-forward layer of the language model to obtain a first intermediate result; Based on N first merged input information, first vector input information, first output result and first intermediate result, multiple first inference results are obtained, where N is a natural number greater than 1.
8. The method according to claim 7, characterized in that The step of performing a zero-redundancy full-quantity operation based on the first vector input information using the self-attention layer to obtain a first output result includes: Distribute the first vector input information into multiple independent operators in the self-attention layer according to a preset partitioning strategy; Using the multiple independent operators to respectively perform full operations on the first vector input information that is input in a scattered manner, to obtain multiple first operation results, and merging the multiple first operation results into a preset video memory corresponding to the language model; The multiple first operation results merged and written into the preset video memory are split to obtain a first output result.
9. The method according to claim 5, characterized in that The step of merging the multiple request information segments and the second request information in batches, and performing zero-redundancy full-volume reasoning and hybrid reasoning on the multiple request information segments and the second request information using the language model to obtain multiple first reasoning results further includes: In response to obtaining third request information requesting to use a preset language model to infer input information, merging the first inference result and the third request information to obtain second merged information; Based on the second merged information, zero-redundancy hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
10. The method according to claim 5, characterized in that The step of merging the multiple request information segments and the second request information in batches, and performing zero-redundancy full-volume reasoning and hybrid reasoning on the multiple request information segments and the second request information using the language model to obtain multiple first reasoning results, including: Using the language model, perform full reasoning, or full reasoning and mixed reasoning, on a first request segment information among the multiple request information segments to obtain first historical result information output by the full reasoning, or the full reasoning and mixed reasoning; In response to obtaining the second request information requesting to use a preset language model to infer the input information, merging the second request fragment information in the multiple request information segments with the second request information to obtain first merged information; Based on the first historical result information and the first merged information, using the language model to perform zero-redundancy hybrid reasoning to obtain at least one of the first reasoning results and the second historical result information; In response to obtaining third request information requesting to use a preset language model to infer input information, merging the first inference result, the third request segment information in the multiple segments of request information, and the third request information to obtain third merged information; Based on the second historical result information and the third merged information, zero-redundancy hybrid reasoning is performed using the language model to obtain multiple first reasoning results.
Citation Information
Patent Citations
Data processing method and related equipment
CN116541492A
Large language model reasoning optimization method and device, computer equipment and storage medium
CN117194056A
Model reasoning method and device based on calculation unit deployment, equipment and medium
CN117494816A
Inference method and device and electronic equipment
CN118036741A
Large language reasoning system and method
CN118052282A
Cited By
Inference optimization method and apparatus for language model, electronic device, and storage medium
WO2026031675A1
