Large language model reasoning acceleration method and device, equipment and medium

Abstract

The invention relates to the field of artificial intelligence, and discloses a reasoning acceleration method and device for a large language model, equipment and a medium, and the method comprises the steps: obtaining a to-be-processed text in a reasoning request in response to the reasoning request input by a user; performing video memory demand prediction according to the to-be-processed text to obtain a video memory prediction result; distributing a corresponding initial video memory space for the large language model according to the video memory prediction result so as to execute a corresponding reasoning task on the to-be-processed text; when the large language model executes the reasoning task, monitoring an occupation index of the initial video memory space; and when the occupation index meets a preset condition, triggering a dynamic allocation mechanism, and allocating a new video memory space for the large language model according to the dynamic allocation mechanism. The corresponding initial video memory space is allocated by predicting the video memory demand in advance, the use condition of the initial video memory space is monitored in real time, and a dynamic allocation mechanism is triggered when resources are insufficient, so that flexible and controllable dynamic video memory allocation is realized, the waste of video memory resources is reduced, and the overall reasoning speed of the model is improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a method, device, equipment and medium for accelerating reasoning of a large language model. Background Art

[0002] With the emergence of large-scale pre-trained language models (LLMs), artificial intelligence applications based on large language models are becoming more and more common in various industries. For example, in the financial field, large language models can well complete text-related reasoning tasks such as intelligent question answering and translation.

[0003] The main challenge facing current large language models, such as those based on the Transformer architecture, when performing reasoning tasks is memory management. The traditional approach is to convert input and output data into key-value form and store them in the GPU memory, but this method wastes memory resources, especially when the output content size is not fixed, thereby reducing the overall model reasoning speed. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a method, device, equipment and medium for accelerating the reasoning of a large language model that can be applied to financial technology or other related fields. Its main purpose is to flexibly and controllably allocate dynamic video memory for the large language model, reduce the waste of video memory resources, and improve the overall reasoning speed of the model.

[0005] The technical solution of the present invention is as follows:

[0006] A first aspect of the present invention provides a method for accelerating reasoning of a large language model, comprising:

[0007] In response to an inference request input by a user, obtaining a text to be processed in the inference request;

[0008] Performing video memory demand prediction based on the text to be processed to obtain a video memory prediction result;

[0009] Allocating corresponding initial video memory space for the large language model according to the video memory prediction result, so as to perform corresponding reasoning tasks on the text to be processed;

[0010] When the large language model executes the reasoning task, monitoring the occupancy index of the initial video memory space;

[0011] When the occupancy index meets a preset condition, a dynamic allocation mechanism is triggered, and new video memory space is allocated to the large language model according to the dynamic allocation mechanism.

[0012] A second aspect of the present invention provides a large language model reasoning acceleration device, comprising:

[0013] A text acquisition module, used to respond to an inference request input by a user and acquire the text to be processed in the inference request;

[0014] A demand prediction module, used for predicting video memory demand according to the text to be processed, and obtaining a video memory prediction result;

[0015] An initial allocation module, used to allocate corresponding initial video memory space to the large language model according to the video memory prediction result, so as to perform corresponding reasoning tasks on the text to be processed;

[0016] A monitoring module, used for monitoring the occupancy index of the initial video memory space when the large language model performs the reasoning task;

[0017] The dynamic allocation module is used to trigger a dynamic allocation mechanism when the occupancy index meets a preset condition, and allocate new video memory space to the large language model according to the dynamic allocation mechanism.

[0018] A third aspect of the present invention provides a computer device, comprising at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned method for accelerating reasoning of a large language model.

[0021] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned large language model reasoning acceleration method.

[0022] Beneficial effects: The present invention discloses a method, device, equipment and medium for accelerating the reasoning of a large language model. Compared with the prior art, the embodiment of the present invention responds to the reasoning request input by the user, obtains the text to be processed in the reasoning request; performs a video memory demand prediction based on the text to be processed to obtain a video memory prediction result; allocates the corresponding initial video memory space to the large language model according to the video memory prediction result to perform the corresponding reasoning task on the text to be processed; when the large language model performs the reasoning task, monitors the occupation index of the initial video memory space; triggers a dynamic allocation mechanism when the occupation index meets a preset condition, and allocates new video memory space to the large language model according to the dynamic allocation mechanism. By predicting the video memory demand in advance and then allocating the corresponding initial video memory space, and monitoring the usage of the initial video memory space in real time, and triggering the dynamic allocation mechanism when the video memory resources are tight, it can not only ensure that the video memory demand of the large language model is met, but also avoid multiple video memory allocations to consume additional computing resources, realize flexible and controllable dynamic video memory allocation for the large language model, reduce the waste of video memory resources, and improve the overall reasoning speed of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the scheme in the present invention, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 A schematic diagram of an application environment of the method for accelerating reasoning of a large language model provided by an embodiment of the present invention;

[0025] Figure 2 A flow chart of a method for accelerating reasoning of a large language model provided by an embodiment of the present invention;

[0026] Figure 3 A flow chart of step S202 in the method for accelerating reasoning of a large language model provided by an embodiment of the present invention;

[0027] Figure 4 A flow chart of step S205 in the method for accelerating reasoning of a large language model provided by an embodiment of the present invention;

[0028] Figure 5 A schematic diagram of functional modules of a large language model reasoning acceleration device provided by an embodiment of the present invention;

[0029] Figure 6 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0031] The large language model reasoning acceleration method provided by the embodiment of the present invention can be applied to Figure 1 In the application environment of FIG. 1 , it includes a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0032] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0033] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0034] The server 105 may be a server that provides various services, such as a background server that provides support for the content browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (for example only). The background server may analyze and process the received data such as the user request, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device. The server 105 may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server 105 may also be a server for a distributed system, or a server combined with a blockchain.

[0035] It should be noted that the reasoning acceleration method of the large language model provided in the embodiment of the present application can generally be executed by the first terminal device 101, the second terminal device 102 or the third terminal device 103. Accordingly, the reasoning acceleration device of the large language model provided in the embodiment of the present invention can also be set in the first terminal device 101, the second terminal device 102 or the third terminal device 103. Alternatively, the reasoning acceleration method of the large language model provided in the embodiment of the present invention can also be generally executed by the server 105. Accordingly, the reasoning acceleration device of the large language model provided in the embodiment of the present invention can generally be set in the server 105.

[0036] It should be understood that the numbers of the above terminal devices, networks and servers are only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0037] like Figure 2 As shown, the method for accelerating the reasoning of a large language model provided by an embodiment of the present invention specifically includes the following steps:

[0038] S201. In response to an inference request input by a user, obtain a text to be processed in the inference request.

[0039] In this embodiment, after deploying a large language model based on application requirements in different fields, users can input corresponding reasoning requests through the front-end interactive interface. For example, developers can build an API interface for calling a large language model to receive reasoning requests from the client. The client can be a web page, mobile application, or desktop application. The user enters the text to be processed in the input box of the client and clicks the send or submit button to trigger the reasoning request. After the API interface receives the request, the back-end service will parse the request content to identify the data format sent by the client (such as JSON, XML, etc.) and extract the text to be processed. At the same time, the back-end service also extracts the metadata of the reasoning task related to the reasoning request, such as task type, priority, request ID, etc., so as to clarify the reasoning task of the current reasoning request for the text to be processed and ensure the accuracy of the subsequent reasoning process.

[0040] For example, in a specific embodiment of a financial scenario, the back-end service of a financial institution receives an inference request, which contains a report text on market trend analysis. The model is required to analyze the text content and predict the trend. At this time, the report text is obtained as the text to be processed, and the inference task is confirmed to be to analyze the text content and predict the trend. The text to be processed is obtained by receiving and parsing, and after being passed to the large language model to perform the corresponding inference task, it can generate a prediction result or analysis report to provide decision support for the financial institution.

[0041] S202: Predict video memory demand based on the text to be processed to obtain a video memory prediction result.

[0042] In this embodiment, after obtaining the text to be processed, when allocating video memory resources to the large language model to perform the reasoning task, since the lengths of different texts to be processed vary greatly and the size of the output content is not fixed, in order to ensure the utilization of video memory resources, before allocating video memory space, the video memory demand of the text to be processed is predicted. The specific prediction method can be based on a deep learning model, that is, training a deep learning model to predict the video memory demand. The model can predict the video memory demand in the reasoning process based on the size and complexity of the input data and other related factors; it can also be based on statistical prediction of historical data, that is, by collecting historical video memory usage data when the model is running, using statistical methods to predict video memory requirements under inputs of different sizes, etc. In addition, predictions can be made based on the text to be processed and the amount of model parameters. That is, the video memory required for model parameters, gradients, and optimizer parameters can be calculated and predicted based on the data type (such as fp32, fp16, int8). For example, a parameter with fp32 precision requires 4 bytes, fp16 requires 2 bytes, and int8 only requires 1 byte, etc., thereby achieving more accurate video memory demand prediction. The video memory prediction results can help the system to reasonably allocate video memory resources. For example, when the request involves a large amount of data analysis and complex model operations, the prediction results may indicate that a higher video memory is required, which avoids resource waste while also ensuring efficient execution of reasoning tasks.

[0043] S203: Allocate corresponding initial video memory space for the large language model according to the video memory prediction result, so as to perform corresponding reasoning tasks on the text to be processed.

[0044] In this embodiment, according to the video memory prediction result, the system allocates an initial video memory space of appropriate size to the large language model, so that the large language model can use the initial video memory space to perform corresponding reasoning tasks on the text to be processed. The specific video memory prediction result can be a specific video memory size, or it can also be a video memory range. Based on the video memory prediction result, the system can choose whether to further process the video memory prediction result based on the pre-set settings. For example, the back-end service of a financial institution receives an inference request, which contains a report text on market trend analysis, and the model is required to analyze the text content and predict the trend. The video memory required for this request is 2GB. The system can directly allocate 2GB of initial video memory space to the large language model according to the video memory prediction result to save video memory resources as much as possible; or the video memory prediction result can be adjusted and then the initial video memory space can be allocated to the large language model. For example, when the video memory resources are sufficient, a certain amount of space can be increased proportionally on the basis of 2GB to ensure that the large language model can perform reasoning tasks sequentially, avoid calculation interruptions caused by prediction errors, and improve the reasoning efficiency of the large language model.

[0045] S204: When the large language model executes the reasoning task, monitor the occupancy index of the initial video memory space.

[0046] In this embodiment, when the Transformer-based large language model performs reasoning tasks on the text to be processed, the text sequence of the text to be processed will be predicted and output based on the self-attention mechanism. In the self-attention mechanism, the input text sequence is mapped to query, key, and value vectors respectively. By calculating the similarity between the query and the key, the attention weight of each query and other elements is obtained. Finally, the attention weight is multiplied by the corresponding value vector, and the weighted sum is obtained to obtain the final output.

[0047] When performing reasoning tasks, the large language model uses the pre-allocated initial video memory space to create query, key, and value matrices, and stores the matrices in the pre-allocated initial video memory space. Since the key and value matrices will not change during the decoding process, they will be shared by all sequences, saving video memory. If the key or value matrix needs to be modified (for example, to implement a specific attention mechanism), the system will first copy a copy of the matrix and then modify it on the copied matrix to ensure that the matrices of other sequences are not affected. If there are multiple reasoning tasks executed in parallel, each text sequence independently decodes its context vector to generate the next token, that is, the decoding process is performed in parallel, and the decoding of each sequence will not affect other sequences. During the decoding process, if a sequence generates a new token, the embedding vector of this token will be added to the input sequence of the sequence and become part of the subsequent attention calculation. The decoding process is executed in a loop until it is completed.

[0048] During the reasoning process of the large language model, the system will monitor the occupation index of the initial video memory space to obtain the usage of the video memory in real time, where the occupation index can be the total amount of video memory occupied, video memory usage rate, video memory peak value, etc. Specifically, you can use monitoring tools for monitoring, such as real-time and regular checks on GPU usage, including video memory occupation rate, GPU utilization rate, etc., or you can add monitoring logic to the code and obtain the current video memory usage through the corresponding API, etc., so as to effectively monitor the video memory usage of the large language model when performing reasoning tasks, so as to adjust the resource allocation strategy in time according to the actual usage to ensure the smooth progress of the reasoning task.

[0049] S205: triggering a dynamic allocation mechanism when the occupancy index meets a preset condition, and allocating new video memory space to the large language model according to the dynamic allocation mechanism.

[0050] In this embodiment, when the occupancy index meets the preset conditions, it indicates that the pre-allocated initial video memory space is insufficient. The specific preset conditions may be, for example, that the video memory usage rate exceeds 80% or the video memory is in a high load state for 30 consecutive seconds, etc., indicating that the video memory occupancy is close to the upper limit when monitored at this time. The system will trigger the dynamic allocation mechanism, and allocate additional video memory space for the large language model to perform the reasoning task according to the dynamic allocation mechanism, so as to ensure the stable operation of the model and the smooth completion of the reasoning task.

[0051] The dynamic allocation mechanism refers to the system dynamically allocating additional video memory resources from the available video memory pool according to the current video memory usage and task requirements. This can be done by adjusting the video memory allocation ratio between different tasks, or migrating tasks in a multi-GPU environment to balance the load, etc. The size of the newly allocated video memory space during dynamic allocation can be determined in different ways, such as allocating a fixed size of video memory each time, or using the video memory usage trend based on a machine learning prediction model to allocate resources in advance, or setting a threshold to automatically expand the video memory allocation according to a preset ratio when the video memory usage approaches the upper limit, etc. Different dynamic allocation mechanisms can be flexibly set according to specific needs. By monitoring and dynamically adjusting video memory allocation, system crashes or task failures caused by insufficient video memory can be prevented, ensuring the continuity and stability of reasoning tasks. By dynamically allocating video memory, it can also ensure that video memory resources are not wasted, while meeting the video memory requirements of the model at different stages.

[0052] In the above embodiment, the present invention discloses a method for accelerating the reasoning of a large language model, which obtains the text to be processed in the reasoning request in response to the reasoning request input by the user; predicts the video memory demand according to the text to be processed to obtain the video memory prediction result; allocates the corresponding initial video memory space to the large language model according to the video memory prediction result to perform the corresponding reasoning task on the text to be processed; monitors the occupation index of the initial video memory space when the large language model performs the reasoning task; triggers the dynamic allocation mechanism when the occupation index meets the preset condition, and allocates new video memory space to the large language model according to the dynamic allocation mechanism. By predicting the video memory demand in advance and then allocating the corresponding initial video memory space, and monitoring the usage of the initial video memory space in real time, and triggering the dynamic allocation mechanism when the video memory resources are tight, it can not only ensure that the video memory demand of the large language model is met, but also avoids consuming additional computing resources by multiple video memory allocations, realizes flexible and controllable dynamic video memory allocation for the large language model, reduces the waste of video memory resources, and improves the overall reasoning speed of the model.

[0053] In one embodiment, Figure 3 As shown, step S202 includes:

[0054] S301, preprocessing and format conversion of the text to be processed to generate a text sequence in a specified input format;

[0055] S302: Input the text sequence into a trained video memory prediction model, predict the size of video memory space required by the large language model to process the text sequence, and obtain a video memory prediction result.

[0056] In this embodiment, a trained deep learning model is used to predict video memory requirements. When making video memory predictions for the current inference request, the received text to be processed is first preprocessed and format converted to generate a text sequence in a specified input format, and the text is segmented into sentences or paragraphs to meet the input requirements of the model, or the text is converted into the encoding format required by the model, such as converting characters into a token sequence that the model can understand, etc. Through preprocessing and format conversion, the quality of the input text and the efficiency of the model processing are ensured, providing an accurate data foundation for subsequent reasoning tasks.

[0057] The processed text sequence is input into the pre-trained video memory prediction model. The video memory prediction model can predict the video memory size required to process the text sequence based on the length and complexity of the text sequence. For example, the model analyzes the length of the text sequence, the complexity of the vocabulary, and the number of financial terms contained in it. Based on historical data and pre-trained algorithms, it predicts the video memory space required to process the text sequence and outputs the video memory prediction result. The video memory prediction model can predict in advance the video memory size required to process the current text sequence, thereby achieving more reasonable video memory resource management and scheduling, and avoiding service interruptions or performance degradation caused by insufficient video memory.

[0058] In one embodiment, the preprocessing and format conversion of the text to be processed to generate a text sequence in a specified input format includes:

[0059] Performing text cleaning and text segmentation on the text to be processed to generate a number of word units;

[0060] Encode each word according to the preset index and convert each word into a unique digital identifier;

[0061] The digital identifiers of all word units are serialized to generate corresponding text sequences.

[0062] In this embodiment, when performing preprocessing and format conversion, the text to be processed is first cleaned, for example, irrelevant content such as headers, footers, special characters, punctuation marks, etc. are removed, and then the text is segmented into individual tokens, i.e., tokens, which can be words, punctuation marks, or basic units in other languages, etc. Text cleaning ensures the quality of input data, removes noise that may interfere with model understanding, and text segmentation enables the model to process and understand text more effectively, laying the foundation for subsequent tokenization and encoding processing.

[0063] For all tokens after word segmentation, they are encoded according to the preset index, for example, the code corresponding to each token is obtained according to the index of the BERT vocabulary, so that each token is converted into a unique digital identifier, so that professional terms such as "operating income" and "net profit" are converted into digital identifiers that the model can understand, ensuring that the model can accurately identify and process each token in the text. And the encoded digital identifiers are serialized, that is, the digital identifiers of all tokens are arranged in the order in which they appear in the text to be processed to form a continuous digital sequence. This sequence can be regarded as a digital representation of the text, which is used as the input of the model. The serialization process allows the text to be processed by the model in a unified way, improving the consistency of data processing and the efficiency of model training and reasoning.

[0064] In one embodiment, the video memory prediction model includes a feature extraction model and a logistic regression model, and the text sequence is input into the trained video memory prediction model, and the size of the video memory space required by the large language model to process the text sequence is predicted to obtain a video memory prediction result, including:

[0065] Adding a CLS token at the beginning of the text sequence, and performing word embedding processing on the CLS token and the text sequence to obtain a corresponding embedding vector;

[0066] Input the embedding vector into the feature extraction model to perform forward propagation and output a corresponding output vector;

[0067] Extracting a feature vector of the CLS token from the output vector, inputting the feature vector of the CLS token into the logistic regression model, and predicting and outputting a video memory requirement value of the large language model for processing the text sequence;

[0068] The video memory requirement value is converted into a corresponding video memory prediction interval according to a preset rule as the video memory prediction result.

[0069] In this embodiment, the video memory prediction model includes a feature extraction model and a logistic regression model. The specific feature extraction model can be a DistilBERT model. When performing video memory prediction, a special classification label symbol, namely a CLS token, is first added to the beginning of the text sequence. After adding the CLS tag at the beginning of the text, the CLS token and the entire text sequence are converted into a vector representation in a high-dimensional space through a pre-trained word embedding model (such as Word2Vec or GloVe) to obtain a corresponding embedding vector. By adding the CLS tag and performing word embedding processing, the model can obtain a rich semantic representation of the text, providing a data basis for subsequent feature extraction and logistic regression prediction.

[0070] The embedding vector obtained in the previous step is input into a pre-trained feature extraction model, such as the DistilBERT model. The model is forward propagated through its multi-layer Transformer architecture. Each layer contains a self-attention mechanism and a feedforward neural network, which enables the model to extract the deep semantic features of the text and generate an output vector that encodes the key information of the text. Since the feature vector of the CLS token can capture the contextual information of the entire input sequence, the feature vector corresponding to the CLS token is extracted from the output of the feature extraction model for subsequent video memory prediction, that is, the feature vector is input into a logistic regression model. The logistic regression model is trained to predict the video memory size required for different feature vectors. Therefore, the logistic regression model processes the CLS token feature vector output by DistilBERT and predicts the video memory size that may be occupied by the inference request, thereby providing accurate predictions for dynamic video memory allocation.

[0071] Since the video memory demand value output by the logistic regression model is a continuous value, in order to convert this value into a more practical prediction interval, the system will convert this value according to preset rules (for example, rounding the demand value to the nearest integer, or classifying it into small, medium, large and other video memory demand levels according to the range of the demand value). The final prediction result can be a specific numerical range, such as "8-10GB". The continuous video memory demand value output by the logistic regression model is converted into the corresponding video memory prediction interval [n,m], where n is the minimum demand and m is the maximum demand, which is used to guide the subsequent video memory allocation strategy to make more reasonable video memory allocation decisions.

[0072] In one embodiment, converting the video memory requirement value into a corresponding video memory prediction interval according to a preset rule as the video memory prediction result includes:

[0073] Obtaining a preset confidence level and a standard error of a prediction output of the video memory prediction model, wherein the standard error is obtained based on statistics of historical prediction results of the video memory prediction model;

[0074] An error range is calculated according to the standard error and the confidence level, and the video memory requirement value is converted into a corresponding video memory prediction interval according to the error range as the video memory prediction result.

[0075] In this embodiment, when converting the video memory demand value into the corresponding video memory prediction interval, the conversion is performed based on the preset confidence level and the error of the model prediction to improve the rationality of the prediction. First, the preset confidence level and the standard error of the prediction output of the video memory prediction model are obtained. For example, a 95% confidence level is pre-set, that is, it is hoped that the prediction result is accurate in 95% of the cases, and the standard error is calculated based on the prediction results of the video memory prediction model for different financial text tasks in the past and the actual video memory usage. The video memory prediction interval is calculated using the standard error and confidence level. For example, if the confidence level is 95%, the corresponding z-score is 1.96 (indicating the critical value of the normal distribution), the error range = standard error * z-score, if the predicted value is 512MB, the standard error is 64MB, then the error range is 125.44MB, so the video memory prediction interval is 512MB with a fluctuation of 125.44MB, that is, [386.56MB, 637.44MB]. By converting a single predicted value into an interval, more information about the possible range of variation of the predicted value can be provided, which increases the flexibility and adaptability of the prediction, so that appropriate video memory resources can be allocated according to this interval to ensure the stable operation of the model.

[0076] In one embodiment, allocating corresponding initial video memory space for the large language model according to the video memory prediction result specifically includes:

[0077] The upper limit value of the video memory prediction interval is used as the initial video memory size, and an initial video memory space of a corresponding size is allocated to the large language model.

[0078] In this embodiment, when pre-allocating video memory, based on the obtained video memory prediction interval [n, m], the system will use the upper limit of the interval as the initial video memory size, that is, allocate video memory of size m as the initial allocation for the reasoning task, which can be specifically implemented through the GPU driver API to ensure that the allocated video memory block is located in the GPU video memory for fast access. By allocating sufficient video memory space, it is ensured that the model has sufficient video memory for calculation, avoiding calculation interruptions caused by insufficient video memory during the model reasoning process, providing flexibility for the model to process financial texts with high uncertainty, and ensuring the continuity and stability of the task.

[0079] In one embodiment, Figure 4 As shown, step S205 includes:

[0080] S501, dividing the currently idle video memory space into blocks to obtain a plurality of video memory blocks;

[0081] S502. When the occupancy index of the initial video memory space is greater than a preset upper limit value, a new logic block is added to a pre-created video memory block linked list, and a corresponding video memory block is allocated to the new logic block; the video memory block linked list is used to map continuous logic blocks to non-continuous video memory blocks to form a continuous virtual video memory space.

[0082] In this embodiment, when new video memory space is dynamically added, the currently free video memory space is divided into blocks to obtain a number of video memory blocks. Specifically, the additional video memory can be divided into blocks of fixed size, and each block is sufficient to store the key and value tensors of a certain number of tokens. At the same time, the system also pre-creates and maintains a video memory block linked list, which is used to map continuous logical blocks to non-continuous video memory blocks to form a continuous virtual video memory space. That is, the mapping relationship between all logical blocks and physical video memory blocks is recorded in the video memory block linked list. Even if the physical video memory blocks are not continuous in the GPU memory, the system can map them into a continuous virtual video memory space through the linked list for use by the model.

[0083] If the usage of video memory is monitored and it is found that the pre-allocated initial video memory space is insufficient, that is, the occupancy rate of the initial video memory space exceeds the preset upper limit, the dynamic allocation mechanism is activated, a new logical block is added to the video memory block list, and the corresponding physical video memory block is allocated from the free video memory block to the new logical block to meet the current demand. This process is dynamic and added according to demand, thereby ensuring the efficient use of video memory, and by maintaining the video memory block list, a continuous virtual video memory space can be formed. Even if the physical video memory space is not continuous, it can ensure that the operators on the GPU can correctly read and write data to meet the needs of the inference process, avoiding the problem of video memory fragmentation and improving the overall utilization of video memory.

[0084] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0085] Further references Figure 5 , as a response to the above Figure 2 The present invention provides an embodiment of a large language model reasoning acceleration device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0086] like Figure 5 As shown, the inference acceleration device 50 of the large language model described in this embodiment includes:

[0087] The text acquisition module 501 is used to respond to the inference request input by the user and acquire the text to be processed in the inference request;

[0088] The demand prediction module 502 is used to predict the demand for video memory according to the text to be processed and obtain a video memory prediction result;

[0089] An initial allocation module 503, configured to allocate corresponding initial video memory space to the large language model according to the video memory prediction result, so as to perform corresponding reasoning tasks on the text to be processed;

[0090] A monitoring module 504, configured to monitor an occupancy index of the initial video memory space when the large language model performs the reasoning task;

[0091] The dynamic allocation module 505 is used to trigger a dynamic allocation mechanism when the occupancy index meets a preset condition, and allocate new video memory space for the large language model according to the dynamic allocation mechanism.

[0092] The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which are more suitable for describing the reasoning acceleration execution process of large language models than programs. For the specific implementation methods of each module, please refer to the corresponding method embodiments above, which will not be repeated here.

[0093] In one embodiment, the demand forecasting module 502 includes:

[0094] A preprocessing unit, used for preprocessing and format conversion of the text to be processed to generate a text sequence in a specified input format;

[0095] The demand prediction unit is used to input the text sequence into a trained video memory prediction model, predict the size of the video memory space required by the large language model to process the text sequence, and obtain a video memory prediction result.

[0096] In one embodiment, the pre-processing unit comprises:

[0097] A cleaning and segmentation unit, used for performing text cleaning and text segmentation on the text to be processed to generate a plurality of word units;

[0098] The encoding unit is used to encode each word element according to a preset index and convert each word element into a unique digital identifier;

[0099] The serialization unit is used to serialize the digital identifiers of all word units to generate corresponding text sequences.

[0100] In one embodiment, the demand forecasting unit includes:

[0101] A word embedding unit, used for adding a CLS token at the beginning of the text sequence, and performing word embedding processing on the CLS token and the text sequence to obtain a corresponding embedding vector;

[0102] A feature extraction unit, used for inputting the embedding vector into the feature extraction model to perform forward propagation and outputting a corresponding output vector;

[0103] A prediction output unit, used for extracting a feature vector of the CLS token from the output vector, inputting the feature vector of the CLS token into the logistic regression model, and predicting and outputting a video memory requirement value of the large language model for processing the text sequence;

[0104] The interval conversion unit is used to convert the video memory demand value into a corresponding video memory prediction interval according to a preset rule as the video memory prediction result.

[0105] In one embodiment, the interval conversion unit includes:

[0106] A parameter acquisition unit, used to acquire a preset confidence level and a standard error of the prediction output of the video memory prediction model, wherein the standard error is obtained based on statistics of historical prediction results of the video memory prediction model;

[0107] An interval calculation unit is used to calculate an error range according to the standard error and the confidence level, and convert the video memory requirement value into a corresponding video memory prediction interval according to the error range as the video memory prediction result.

[0108] In one embodiment, the initial allocation module 503 is specifically configured to:

[0109] The upper limit value of the video memory prediction interval is used as the initial video memory size, and an initial video memory space of a corresponding size is allocated to the large language model.

[0110] In one embodiment, the dynamic allocation module 505 includes:

[0111] A block division unit is used to divide the currently idle video memory space into blocks to obtain a plurality of video memory blocks;

[0112] A dynamic appending unit is used to append a new logic block to a pre-created video memory block linked list when the occupancy index of the initial video memory space is greater than a preset upper limit value, and allocate a corresponding video memory block to the new logic block; the video memory block linked list is used to map continuous logic blocks to non-continuous video memory blocks to form a continuous virtual video memory space.

[0113] In the above embodiment, the present invention discloses an inference acceleration device for a large language model, which obtains the text to be processed in the inference request in response to the inference request input by the user; performs a memory demand prediction based on the text to be processed to obtain a memory prediction result; allocates a corresponding initial memory space to the large language model according to the memory prediction result to perform a corresponding inference task on the text to be processed; monitors the occupancy index of the initial memory space when the large language model performs the inference task; triggers a dynamic allocation mechanism when the occupancy index meets a preset condition, and allocates new memory space to the large language model according to the dynamic allocation mechanism. By predicting the memory demand in advance and then allocating the corresponding initial memory space, and monitoring the usage of the initial memory space in real time, and triggering the dynamic allocation mechanism when the memory resources are tight, it can not only ensure that the memory demand of the large language model is met, but also avoids consuming additional computing resources by multiple memory allocations, thereby realizing flexible and controllable dynamic memory allocation for the large language model, reducing the waste of memory resources, and improving the overall inference speed of the model.

[0114] Another embodiment of the present invention provides a computer device, such as Figure 6 As shown, the computer device 60 includes:

[0115] One or more processors 601 and memory 602, Figure 6 A processor 601 is used as an example for the description. The processor 601 and the memory 602 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0116] Processor 601 is used to complete various control logics of computer device 60, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component or any combination of these components. In addition, processor 601 can also be any traditional processor, microprocessor or state machine. Processor 601 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0117] The memory 602, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the method for accelerating the reasoning of a large language model in the embodiment of the present invention. The processor 601 executes various functional applications and data processing of the computer device 60 by running the non-volatile software programs, instructions and units stored in the memory 602, that is, the method for accelerating the reasoning of a large language model in the above method embodiment is implemented.

[0118] The memory 602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the computer device 60, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 602 may optionally include a memory remotely arranged relative to the processor 601, and these remote memories may be connected to the computer device 60 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. One or more units are stored in the memory 602, and when executed by one or more processors 601, the steps of the reasoning acceleration method of the large language model in any of the above-mentioned method embodiments are executed.

[0119] In the above embodiment, the present invention discloses a computer device, which obtains the text to be processed in the reasoning request by responding to the reasoning request input by the user; performs a video memory demand prediction based on the text to be processed to obtain a video memory prediction result; allocates corresponding initial video memory space to the large language model according to the video memory prediction result to perform corresponding reasoning tasks on the text to be processed; monitors the occupation index of the initial video memory space when the large language model performs the reasoning task; triggers a dynamic allocation mechanism when the occupation index meets a preset condition, and allocates new video memory space to the large language model according to the dynamic allocation mechanism. By predicting the video memory demand in advance and then allocating the corresponding initial video memory space, and monitoring the usage of the initial video memory space in real time, and triggering the dynamic allocation mechanism when the video memory resources are tight, it can not only ensure that the video memory demand of the large language model is met, but also avoids multiple video memory allocations to consume additional computing resources, and realizes flexible and controllable dynamic video memory allocation for the large language model, reduces the waste of video memory resources, and improves the overall reasoning speed of the model.

[0120] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the steps of the method for accelerating reasoning of a large language model in any of the above method embodiments are executed.

[0121] In the above embodiment, the present invention discloses a non-volatile computer-readable storage medium, which obtains the text to be processed in the inference request in response to the inference request input by the user; performs a video memory demand prediction based on the text to be processed to obtain a video memory prediction result; allocates a corresponding initial video memory space to a large language model based on the video memory prediction result to perform a corresponding inference task on the text to be processed; monitors the occupancy index of the initial video memory space when the large language model performs the inference task; triggers a dynamic allocation mechanism when the occupancy index meets a preset condition, and allocates new video memory space to the large language model according to the dynamic allocation mechanism. By predicting the video memory demand in advance and then allocating the corresponding initial video memory space, and monitoring the usage of the initial video memory space in real time, and triggering the dynamic allocation mechanism when the video memory resources are tight, it can not only ensure that the video memory demand of the large language model is met, but also avoids multiple video memory allocations to consume additional computing resources, thereby realizing flexible and controllable dynamic video memory allocation for the large language model, reducing the waste of video memory resources, and improving the overall inference speed of the model.

[0122] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0123] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0124] In summary, the invention discloses a method, device, equipment and medium for accelerating the reasoning of a large language model, wherein the method comprises: in response to a reasoning request input by a user, obtaining a text to be processed in the reasoning request; predicting the video memory demand according to the text to be processed to obtain a video memory prediction result; allocating a corresponding initial video memory space to the large language model according to the video memory prediction result to perform a corresponding reasoning task on the text to be processed; monitoring the occupation index of the initial video memory space when the large language model performs the reasoning task; triggering a dynamic allocation mechanism when the occupation index meets a preset condition, and allocating new video memory space to the large language model according to the dynamic allocation mechanism. By predicting the video memory demand in advance and then allocating the corresponding initial video memory space, and monitoring the usage of the initial video memory space in real time, and triggering the dynamic allocation mechanism when the video memory resources are tight, it can not only ensure that the video memory demand of the large language model is met, but also avoid multiple video memory allocations to consume additional computing resources, thereby realizing flexible and controllable dynamic video memory allocation for the large language model, reducing the waste of video memory resources, and improving the overall reasoning speed of the model.

[0125] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0126] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of the present application, they are merely used for illustration and do not represent actual use. It should be understood that the application of the present invention is not limited to the above examples, and that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes shall fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for accelerating reasoning of a large language model, characterized in that: include: In response to an inference request input by a user, obtaining a text to be processed in the inference request; Performing video memory demand prediction based on the text to be processed to obtain a video memory prediction result; Allocating corresponding initial video memory space for the large language model according to the video memory prediction result, so as to perform corresponding reasoning tasks on the text to be processed; When the large language model executes the reasoning task, monitoring the occupancy index of the initial video memory space; When the occupancy index meets a preset condition, a dynamic allocation mechanism is triggered, and new video memory space is allocated to the large language model according to the dynamic allocation mechanism.

2. The method for accelerating the reasoning of a large language model according to claim 1, characterized in that: The step of performing video memory demand prediction according to the text to be processed to obtain a video memory prediction result includes: Preprocessing and format conversion are performed on the text to be processed to generate a text sequence in a specified input format; The text sequence is input into a trained video memory prediction model, and the size of the video memory space required by the large language model to process the text sequence is predicted to obtain a video memory prediction result.

3. The method for accelerating the reasoning of a large language model according to claim 2, characterized in that: The preprocessing and format conversion of the to-be-processed text to generate a text sequence in a specified input format includes: Performing text cleaning and text segmentation on the text to be processed to generate a number of word units; Encode each word according to the preset index and convert each word into a unique digital identifier; The digital identifiers of all word units are serialized to generate corresponding text sequences.

4. The method for accelerating the reasoning of a large language model according to claim 2, characterized in that: The video memory prediction model includes a feature extraction model and a logistic regression model. The text sequence is input into the trained video memory prediction model, and the size of the video memory space required by the large language model to process the text sequence is predicted to obtain a video memory prediction result, including: Adding a CLS token at the beginning of the text sequence, and performing word embedding processing on the CLS token and the text sequence to obtain a corresponding embedding vector; Input the embedding vector into the feature extraction model to perform forward propagation and output a corresponding output vector; Extracting a feature vector of the CLS token from the output vector, inputting the feature vector of the CLS token into the logistic regression model, and predicting and outputting a video memory requirement value of the large language model for processing the text sequence; The video memory requirement value is converted into a corresponding video memory prediction interval according to a preset rule as the video memory prediction result.

5. The method for accelerating the reasoning of a large language model according to claim 4, characterized in that: The converting the video memory requirement value into a corresponding video memory prediction interval according to a preset rule as the video memory prediction result includes: Obtaining a preset confidence level and a standard error of a prediction output of the video memory prediction model, wherein the standard error is obtained based on statistics of historical prediction results of the video memory prediction model; An error range is calculated according to the standard error and the confidence level, and the video memory requirement value is converted into a corresponding video memory prediction interval according to the error range as the video memory prediction result.

6. The method for accelerating the reasoning of a large language model according to claim 4, characterized in that: The allocating corresponding initial video memory space for the large language model according to the video memory prediction result specifically includes: The upper limit value of the video memory prediction interval is used as the initial video memory size, and an initial video memory space of a corresponding size is allocated to the large language model.

7. The method for accelerating the reasoning of a large language model according to claim 1, characterized in that: The triggering of a dynamic allocation mechanism when the occupancy index meets a preset condition, and allocating new video memory space for the large language model according to the dynamic allocation mechanism, includes: Divide the currently free video memory space into blocks to obtain a number of video memory blocks; When the occupancy index of the initial video memory space is greater than a preset upper limit value, a new logic block is added to a pre-created video memory block linked list, and a corresponding video memory block is allocated to the new logic block; the video memory block linked list is used to map continuous logic blocks to non-continuous video memory blocks to form a continuous virtual video memory space.

8. A large language model reasoning acceleration device, characterized in that: include: A text acquisition module, used to respond to an inference request input by a user and acquire the text to be processed in the inference request; A demand prediction module, used for predicting video memory demand according to the text to be processed, and obtaining a video memory prediction result; An initial allocation module, used to allocate corresponding initial video memory space to the large language model according to the video memory prediction result, so as to perform corresponding reasoning tasks on the text to be processed; A monitoring module, used for monitoring the occupancy index of the initial video memory space when the large language model performs the reasoning task; The dynamic allocation module is used to trigger a dynamic allocation mechanism when the occupancy index meets a preset condition, and allocate new video memory space to the large language model according to the dynamic allocation mechanism.

9. A computer device, characterized in that: comprising at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the reasoning acceleration method for a large language model as described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the reasoning acceleration method for a large language model as described in any one of claims 1-7.

Images