An accelerated reasoning method and system for cognitive model question-answering prediction

Abstract

The present invention discloses an accelerated reasoning method for cognitive model question-answer prediction, including: obtaining question text, obtaining question tokens based on the question text; distributing question tokens to cognitive model reasoning engines; determining running memory based on question tokens through time-delayed neural networks; the cognitive model reasoning engine performs accelerated reasoning prediction on question tokens based on running memory, model parameter quantization and model hybrid parallelism to obtain predicted answers. It supports efficient reasoning of various cognitive models and supports efficient overloading of cognitive model sessions.

Description

Technical Field

[0001] The present invention relates to the field of cognitive model technology, and more specifically to an accelerated reasoning method and system for cognitive model question and answer prediction. Background Art

[0002] At present, large cognitive models based on transformer structures perform well in NLP tasks, and the more model parameters there are, the better the effect. Different reasoning frameworks are used to accelerate the reasoning of large cognitive models, such as vLLM, llamacpp, chatglmcpp, TensorRT-LLM, etc., but each has different tendencies:

[0003] vLLM: is an open source LLM reasoning and service engine developed by the UC Berkeley team. It implements RollingBatch batch processing and PagedAttention attention algorithms. Compared with static batch, vLLM provides up to dozens of times the throughput without any model architecture changes; llamacpp: is a cross-platform tensor operation library based on ggml, supporting 8-bit and 4-bit quantization and cross-platform operation of llama series models; chatglmcpp: is similar to llamacpp, but supports chatglm series models; TensorRT-LLM: is an LLM reasoning framework developed by NVIDIA based on TensorRT, supporting PagedAttention, tensor parallelism, 8-bit and 4-bit quantization.

[0004] However, existing reasoning frameworks all have problems: vLLM currently does not support model quantization, and the minimum precision only supports fp16. In addition, insufficient video memory during use will directly lead to service unavailability; llamacpp does not support multi-card operation, tensor parallelism, or acceleration algorithms such as PageAttention; TensorRT-LLM only supports the NVIDIA platform, and TensorRT is not open source, which is not friendly for adapting to new cognitive models. Based on the above problems, existing reasoning frameworks cannot achieve efficient reasoning of cognitive models and do not support cognitive model session reloading.

[0005] Therefore, how to support efficient reasoning of various cognitive models and support efficient overloading of cognitive model sessions is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0006] In view of this, the present invention provides an accelerated reasoning method and system for cognitive model question and answer prediction, which supports efficient reasoning of various cognitive models and supports efficient reloading of cognitive model conversations.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] An accelerated reasoning method for cognitive model question answering prediction, comprising:

[0009] Obtain a question text, and obtain a question Token based on the question text;

[0010] Distribute the question Token to the cognitive model reasoning engine;

[0011] Determining the running memory based on the problem token by delaying the neural network by time;

[0012] The cognitive model reasoning engine performs accelerated reasoning and prediction on the question Token based on the running memory, model parameter quantization and model hybrid parallelism to obtain a predicted answer.

[0013] Preferably, it also includes:

[0014] Get the historical question and answer text;

[0015] Obtaining a historical token based on the historical question and answer text;

[0016] Distribute the historical Token to the cognitive model reasoning engine.

[0017] Preferably, obtaining a question token based on the question text specifically includes:

[0018] Convert the question text into question Token based on the vocabulary;

[0019] The method for constructing the vocabulary at least includes: Byte-PairEncoding method, WordPiece method or SentencePiece method;

[0020] The Byte-PairEncoding method is specifically as follows: split the words in the database into character granularity to obtain multiple adjacent bytes, count the frequency of each adjacent byte pair, merge the adjacent byte pairs with the highest frequency into new subwords, repeat the merging until the preset word list size is reached, and obtain the final word list;

[0021] The WordPiece method is specifically as follows: splitting the words in the database into character granularity to obtain multiple adjacent bytes, counting the frequency of each adjacent byte pair, merging the adjacent byte pairs with the highest frequency into new subwords, merging the adjacent subwords that can improve the probability of the language model the most and adding them to the vocabulary, and repeating the merging and adding to the vocabulary until the preset vocabulary size is reached to obtain the final vocabulary;

[0022] The SentencePiece method is specifically as follows: words and spaces in the database are split into character granularity to obtain multiple adjacent bytes, the frequency of occurrence of each adjacent byte pair is counted, and the adjacent byte pairs with the highest frequency are merged into new subwords, and the merging is repeated until the preset word list size is reached to obtain the final word list.

[0023] Preferably, the specific process of determining the running memory is:

[0024] The problem token and the historical token are concatenated to obtain a concatenated token;

[0025] Obtaining time series data based on the spliced ​​Tokens;

[0026] The time series data is input into the time delay neural network consisting of an input layer, a hidden layer and an output layer;

[0027] The input layer receives and sends the time series data to the hidden layer;

[0028] The multiple neurons in the hidden layer receive the time series data respectively, and perform convolution operation through a weight matrix of a fixed window to obtain multiple first processed data;

[0029] The plurality of first processed data are respectively processed by a nonlinear activation function to obtain a plurality of second processed data;

[0030] The output layer receives the plurality of second processed data for processing to obtain a predicted running memory size.

[0031] Preferably, it also includes: judging whether the running memory obtained during the reasoning process of the cognitive model reasoning engine is sufficient for calculating the output result;

[0032] If it is not enough, new memory is acquired in proportion to the size of the used memory;

[0033] And merge the new memory with the used memory.

[0034] Preferably, the model parameter quantization specifically includes:

[0035] Converting the parameters calculated in the cognitive model inference engine from high-precision data to low-precision data;

[0036] The model parameter quantization method at least includes: symmetric quantization, asymmetric quantization and hierarchical quantization.

[0037] Preferably, the hybrid parallel operation specifically includes:

[0038] During the reasoning process, the cognitive model reasoning engine adopts a hybrid parallel mode of model parallelism and tensor parallelism;

[0039] The model parallelism comprises: dividing all the operators in the cognitive model reasoning engine into a plurality of first sub-parts, each of the first sub-parts respectively running on a corresponding GPU;

[0040] The tensor parallelism is as follows: based on the model parallelism, the operator is split into multiple second sub-parts, and each of the second sub-parts runs on a corresponding GPU.

[0041] Preferably, it also includes: when the video memory of the cognitive model reasoning engine is insufficient, data that is temporarily not needed for reasoning calculation is migrated from the video memory to the internal memory, and when calculation is needed, the data in the internal memory is moved to the video memory for reasoning calculation.

[0042] Preferably, it also includes: for questions and answers that have no conversations within a preset time, the kv cache corresponding to the historical records in the questions and answers is saved to a cache database, and when the questions and answers are asked again, the corresponding kv cache is read from the cache database for inference calculation.

[0043] An accelerated reasoning system for cognitive model question-answer prediction, comprising: a question encoding acquisition module, a distribution module, a memory determination module and a prediction module;

[0044] The question code acquisition module is used to acquire the question text and obtain the question Token based on the question text;

[0045] The distribution module is used to distribute the question Token to the cognitive model reasoning engine;

[0046] The memory determination module is used to determine the running memory based on the problem Token by time delaying the neural network;

[0047] The prediction module is used for the cognitive model reasoning engine to perform accelerated reasoning and prediction on the question Token based on the running memory, model parameter quantization and model hybrid parallelism to obtain a predicted answer.

[0048] Preferably, it also includes: a historical code push module;

[0049] The historical coding push module is used to obtain historical question and answer texts; obtain historical tokens based on the historical question and answer texts; and distribute the historical tokens to the cognitive model reasoning engine.

[0050] Preferably, the memory determination module is further used for:

[0051] Concatenate the problem Token and the history Token to obtain a concatenated Token;

[0052] Obtaining time series data based on the spliced ​​Tokens;

[0053] The time series data is input into the time delay neural network consisting of an input layer, a hidden layer and an output layer;

[0054] The input layer receives and sends the time series data to the hidden layer;

[0055] The multiple neurons in the hidden layer receive the time series data respectively, and perform convolution operation through a weight matrix of a fixed window to obtain multiple first processed data;

[0056] The plurality of first processed data are respectively processed by a nonlinear activation function to obtain a plurality of second processed data;

[0057] The output layer receives the plurality of second processed data for processing to obtain a predicted running memory size.

[0058] Preferably, it also includes: a memory judgment acquisition module;

[0059] The memory judgment acquisition module is used to determine whether the running memory obtained during the reasoning process of the cognitive model reasoning engine is sufficient to calculate the output result; if not, new memory is obtained in proportion to the size of the used memory; and the new memory is merged with the used memory.

[0060] 14. The accelerated reasoning system for cognitive model question-answering prediction according to claim 10, characterized in that it also includes: a data migration module;

[0061] The data migration module is used to migrate data that is temporarily not needed for reasoning calculations from the video memory to the internal memory when the video memory of the cognitive model reasoning engine is insufficient, and to move the data in the internal memory to the video memory for reasoning calculations when calculations are needed.

[0062] Preferably, it also includes: a data cache reading module;

[0063] The data cache reading module is used to save the kv cache corresponding to the historical records in the questions and answers that have no conversations within a preset time to the cache database. When the questions and answers are asked again, the corresponding kv cache is read from the cache database for inference calculation.

[0064] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses an accelerated reasoning method and system for cognitive model question and answer prediction, which supports efficient reasoning of various cognitive models and efficient reloading of cognitive model conversations, and has the following beneficial effects:

[0065] 1. Supports mixed operation of CPU and GPU, and can run on any GPU card without GPU card.

[0066] 2. It can support and apply multiple cognitive models, making it easy to promote and use.

[0067] 3. Supports conversation memory. If the conversation is interrupted, you can continue the conversation at any time by continuing with the previous conversation record, which is more practical.

[0068] 4. Efficient question and answer generation achieved through in-memory prediction, model parameter quantization, model hybrid parallelism and kv database. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0070] Figure 1 A flow chart of an accelerated reasoning method for cognitive model question and answer prediction provided by the present invention.

[0071] Figure 2 A flow chart of the method for determining running memory provided by the present invention.

[0072] Figure 3 A schematic diagram of the structure of an accelerated reasoning system for cognitive model question and answer prediction provided by the present invention.

[0073] Figure 4 Another preferred schematic diagram of the structure of an accelerated reasoning system for cognitive models provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] Example 1

[0076] like Figure 1 As shown, an embodiment of the present invention discloses an accelerated reasoning method for cognitive model question-answering prediction, including:

[0077] Get the question text and get the question Token based on the question text;

[0078] Distribute the question token to the cognitive model reasoning engine;

[0079] Determine the running memory based on the question token by delaying the neural network in time;

[0080] The cognitive model reasoning engine accelerates the reasoning and prediction of the question token based on running memory, model parameter quantization and model hybrid parallelism to obtain the predicted answer.

[0081] Example 2

[0082] The embodiment of the present invention discloses an accelerated reasoning method for cognitive model question-answer prediction, including:

[0083] Get the question text and get the question Token based on the question text.

[0084] Preferably, a user request is obtained, and the question text is obtained based on the user request.

[0085] Preferably, obtaining a question token based on the question text specifically includes:

[0086] Convert question text into question tokens based on the vocabulary;

[0087] The vocabulary construction method includes at least: Byte-PairEncoding method, WordPiece method or SentencePiece method;

[0088] The Byte-PairEncoding method is as follows: split the words in the database into character granularity to obtain multiple adjacent bytes, count the frequency of each adjacent byte pair, merge the adjacent byte pairs with the highest frequency into new subwords, and repeat the merging until the preset word list size is reached to obtain the final word list;

[0089] The WordPiece method is as follows: split the words in the database into character granularity to obtain multiple adjacent bytes, count the frequency of each adjacent byte pair, merge the adjacent byte pairs with the highest frequency into new subwords, merge the adjacent subwords that can improve the probability of the language model the most and add them to the vocabulary, repeat the merging and adding to the vocabulary until the preset vocabulary size is reached to obtain the final vocabulary;

[0090] The SentencePiece method is as follows: split the words and spaces in the database into character granularity to obtain multiple adjacent bytes, count the frequency of each adjacent byte pair, merge the adjacent byte pairs with the highest frequency into new subwords, and repeat the merging until the preset vocabulary size is reached to obtain the final vocabulary.

[0091] Preferably, the question text is called prompt in the cognitive model, and the question token is called question token in the cognitive model.

[0092] Preferably, the method further includes: obtaining historical question and answer texts; obtaining historical tokens based on the historical question and answer texts; and distributing the historical tokens to the cognitive model reasoning engine.

[0093] Preferably, the history token and the question token are sent to the knowledge model reasoning engine at the same time.

[0094] Distribute the question token to the cognitive model reasoning engine.

[0095] Preferably, since cognitive models occupy more computer resources, different cognitive models are deployed on different machines, user requests are distributed to cognitive model reasoning engines for reasoning, and the reasoning results are returned to the user.

[0096] Preferably, a unified request interface is provided to users through a proxy service, and the proxy service distributes user requests to different cognitive model reasoning engines for reasoning.

[0097] Preferably, distribution means handing over different user requests to different cognitive model inference engines for processing, and distribution will be performed according to the requested model and the load of the machine.

[0098] Delay the neural network through time and determine the running memory based on the question token.

[0099] Preferably, Figure 2 As shown in the figure, the specific process of determining the running memory is:

[0100] The problem token and the historical token are concatenated to obtain the concatenated token;

[0101] Get time series data based on splicing tokens;

[0102] The time series data is input into a time-delay neural network consisting of an input layer, a hidden layer, and an output layer;

[0103] The input layer receives and sends time series data to the hidden layer;

[0104] Multiple neurons in the hidden layer receive the time series data respectively, and perform convolution operation through a weight matrix of a fixed window to obtain multiple first processed data;

[0105] The plurality of first processed data are processed by a nonlinear activation function to obtain a plurality of second processed data;

[0106] The output layer receives a plurality of second processed data for processing to obtain a predicted running memory size.

[0107] Preferably, neurons in the hidden layer have different time delay windows and are able to process data at different time steps.

[0108] Preferably, when the cognitive model predicts the generated result, the next answer text to be answered is predicted based on the question text and the existing historical question and answer text. In this process, the kv cache (kv_cache) corresponding to the question text and the historical question and answer text will be saved to speed up the calculation process. During the calculation process, the kv cache (kv_cache) corresponding to the newly generated text and the kv cache (kv_cache) corresponding to the historical question and answer text need to be spliced, which will result in a large amount of memory application, release, and copying. The present invention uses a time-delayed neural network to predict the size of the running memory required during the reasoning process based on the input content, and applies for a whole block of memory at the beginning, which reduces the time for memory application, release, and copying, reduces memory operations, and reduces the time consumed by reasoning.

[0109] Preferably, since the TDNN network has a simple structure, can effectively process context information, has high parallelism and high speed, and will not affect the performance of the entire reasoning process, this embodiment uses the TDNN (Time Delay Neural Network) network to predict the memory size required during the reasoning process.

[0110] Preferably, the TDNN network is a time-delay neural network used to process time series data. It can capture time-related information in sequence data and can learn and predict future data. The TDNN network is mainly composed of an input layer, a hidden layer, and an output layer.

[0111] Preferably, the TDNN network has a wide range of applications in speech recognition, natural language processing and other fields. It can process input sequences of different lengths and has the characteristics of time invariance and position invariance. The TDNN network can also extract more advanced features by adding convolutional layers and pooling layers to further improve the performance of the model.

[0112] Preferably, other networks such as RNN structure networks and Transformer structure networks can also be used to predict the memory size required during the reasoning process.

[0113] Preferably, it also includes: determining whether the running memory obtained during the reasoning process of the cognitive model reasoning engine is sufficient for calculating the output result;

[0114] If it is not enough, new memory is acquired in proportion to the size of the used memory;

[0115] And merge the new memory with the used memory.

[0116] The cognitive model reasoning engine accelerates the reasoning and prediction of the question token based on running memory, model parameter quantization and model hybrid parallelism to obtain the predicted answer.

[0117] Preferably, the model parameter quantization specifically includes:

[0118] Convert the parameters of the computing unit in the cognitive model inference engine from high-precision data to low-precision data for calculation, which speeds up the calculation and reduces the video memory bandwidth;

[0119] The model parameter quantization method includes at least: symmetric quantization, asymmetric quantization and hierarchical quantization.

[0120] Preferably, the quantization precision supported by this embodiment is 4-bit and 8-bit quantization, and the quantization kernel of various operators (computational units) in the cognitive model is implemented to accelerate the reasoning process of the cognitive model.

[0121] Preferably, mixed parallelism specifically includes:

[0122] During the reasoning process, the cognitive model reasoning engine adopts a hybrid parallel approach of model parallelism and tensor parallelism;

[0123] Model parallelism: All operators in the cognitive model reasoning engine are divided into multiple first sub-parts, and each first sub-part runs on a corresponding GPU;

[0124] Tensor parallelism: Based on model parallelism, the operator is split into multiple second sub-parts, and each second sub-part runs on the corresponding GPU.

[0125] Preferably, for example: model parallelism divides the computing unit into two first sub-computing parts, and the two first sub-computing parts run on two GPUs respectively; tensor parallelism is based on model parallelism, and the two first sub-computing parts are divided into two second sub-computing parts respectively, to obtain four second sub-computing parts, which run on four GPUs respectively.

[0126] Preferably, another example: tensor parallelism is to run one self-attention layer on two GPUs, and two self-attention layers are also run on two GPUs. Assuming that there are 4 self-attention layers, it will lead to insufficient video memory, then run two self-attention layers on two GPUs, and the other two self-attention layers on the other two GPUs.

[0127] Preferably, in the deep learning reasoning process, according to the different ways of splitting the model, it can be divided into model parallelism and tensor parallelism. Model parallelism means that different devices are responsible for the calculation of different parts of a single calculation graph; tensor parallelism means that the parameters (tensors) within the layer in the calculation graph are split to different devices (i.e., intra-layer parallelism), and each device only has a part of the model.

[0128] Preferably, for the MLP and self-attention layers in the cognitive model, model parallelism is used to accelerate the calculation. However, in the cognitive model, the self-attention calculation is input with a kv cache (kv_cache). When the historical sequence is very long, the kv cache (kv_cache) will be very large. The video memory of a single device cannot store the kv cache (kv_cache) of all self-attention layers. It is necessary to combine model parallelism and tensor parallelism to run some self-attention layers of the model on different GPU devices.

[0129] Preferably, kv_cache is a large model reasoning performance optimization technology.

[0130] Preferably, a predicted answer Token is obtained by inference, and the predicted answer Token is converted into a predicted answer text based on a vocabulary.

[0131] Preferably, it also includes: when the cognitive model inference engine has insufficient video memory, data temporarily not needed for inference calculation is migrated from the video memory to the internal memory, and when calculation is needed, the data in the internal memory is moved to the video memory for inference calculation. Supporting the mixed operation of CPU and GPU, so that the method of the present invention is applicable to devices with relatively low performance and relatively small video memory.

[0132] Preferably, it also includes: for questions and answers that have no conversations within a preset time, the kv cache corresponding to the historical records in the questions and answers is saved to the cache database. When the question and answer is repeated, the corresponding kv cache is read from the cache database and loaded into the video memory for inference calculation, supporting session memory. For example, after the session is interrupted, the conversation can be continued at any time using the previous conversation record, thereby realizing user session state management.

[0133] Example 3

[0134] like Figure 3 As shown, an accelerated reasoning system for cognitive model question-answer prediction includes: a question encoding acquisition module, a distribution module, a memory determination module and a prediction module;

[0135] The question code acquisition module is used to obtain the question text and obtain the question token based on the question text;

[0136] The distribution module is used to distribute the question token to the cognitive model reasoning engine;

[0137] A memory determination module is used to determine the running memory based on the question token through a time delay neural network;

[0138] The prediction module is used by the cognitive model reasoning engine to accelerate the reasoning and prediction of the question token based on running memory, model parameter quantization and model hybrid parallelism to obtain the predicted answer.

[0139] Example 4

[0140] like Figure 4 As shown, a preferred accelerated reasoning system for cognitive model question-answer prediction includes: a question encoding acquisition module, a distribution module, a memory determination module and a prediction module;

[0141] The question code acquisition module is used to obtain the question text and obtain the question token based on the question text;

[0142] The distribution module is used to distribute the question token to the cognitive model reasoning engine;

[0143] A memory determination module is used to determine the running memory based on the question token by delaying the neural network over time;

[0144] The prediction module is used by the cognitive model reasoning engine to accelerate the reasoning and prediction of the question token based on running memory, model parameter quantization and model hybrid parallelism to obtain the predicted answer.

[0145] Preferably, it also includes: a historical code push module;

[0146] The historical coding push module is used to obtain historical question and answer texts; obtain historical tokens based on historical question and answer texts; and distribute historical tokens to the cognitive model reasoning engine.

[0147] Preferably, the memory determination module is also used for:

[0148] Concatenate the problem token and the history token to get the concatenated token;

[0149] Get time series data based on splicing tokens;

[0150] The time series data is input into a time-delay neural network consisting of an input layer, a hidden layer, and an output layer;

[0151] The input layer receives and sends time series data to the hidden layer;

[0152] Multiple neurons in the hidden layer receive the time series data respectively, and perform convolution operation through a weight matrix of a fixed window to obtain multiple first processed data;

[0153] The plurality of first processed data are processed by a nonlinear activation function to obtain a plurality of second processed data;

[0154] The output layer receives a plurality of second processed data for processing to obtain a predicted running memory size.

[0155] Preferably, it also includes: a memory judgment acquisition module;

[0156] The memory judgment acquisition module is used to determine whether the running memory obtained during the reasoning process of the cognitive model reasoning engine is sufficient to calculate the output results; if not, new memory is obtained in proportion to the size of the used memory; and the new memory is merged with the used memory.

[0157] Preferably, it also includes: a data migration module;

[0158] The data migration module is used to migrate data that is temporarily not needed for reasoning calculations from the video memory to the internal memory when the video memory of the cognitive model reasoning engine is insufficient. When calculation is needed, the data in the internal memory is moved to the video memory for reasoning calculations.

[0159] Preferably, it also includes: a data cache reading module;

[0160] The data cache reading module is used to save the kv cache corresponding to the historical records in the questions and answers that have no conversations within a preset time to the cache database. When the question and answer is repeated, the corresponding kv cache is read from the cache database for inference calculation.

[0161] Example 5

[0162] The present invention proposes a high-performance cognitive model reasoning system, which aims to support efficient reasoning of various cognitive models and efficient reloading of cognitive model sessions.

[0163] This invention proposes a high-performance cognitive model reasoning system, which aims to accelerate the reasoning performance of cognitive models. The system supports:

[0164] 1. Supports CPU and GPU mixed operation, and can run on any GPU card after GPU card is removed

[0165] 2. Support multiple cognitive models

[0166] 3. Support conversation memory, if the conversation is interrupted today, you can continue the conversation tomorrow by following the previous conversation record

[0167] 4. Support efficient generation of answers

[0168] The system consists of the following parts:

[0169] 1. Cognitive Model Proxy Service

[0170] Since cognitive models occupy more computer resources, different cognitive models will be deployed on different machines. Therefore, a unified request interface is provided to users through proxy services. The proxy service distributes user requests to different cognitive model inference engines for inference [distribution means that different requests are sent to different machines for processing, and they will be distributed according to the requested model and the load of the machine], and returns the inference results to the user. At the same time, the status of the user session is also managed. The request here is the question text, and the response is the corresponding answer text

[0171] Tokenizer module

[0172] The tokenizer module is used to convert the user's question text (called prompt in the cognitive model) into the corresponding Token (called Token in the cognitive model) using the vocabulary, and can also convert the Token sequence into the corresponding text. The vocabulary construction method supports:

[0173] Byte-Pair Encoding (BPE) method: Split words into character granularity, count the frequency of each adjacent byte pair, merge the highest frequency adjacent byte pairs into new subwords, and repeat the merging until the set word list size is reached.

[0174] WordPiece: The steps are basically the same as BPE, but the merging algorithm is to merge the adjacent subwords that can improve the probability of the language model the most and add them to the vocabulary

[0175] SentencePiece: The steps are basically the same as BPE, except that when splitting words, spaces are also treated as special characters.

[0176] 3. Cognitive Reasoning Engine

[0177] The cognitive reasoning engine generates corresponding answers based on historical conversation information and current questions.

[0178] In the cognitive reasoning engine, we use the following methods to accelerate the reasoning process: quantization, hybrid parallelism, and reasoning memory optimization.

[0179] 4. Quantification

[0180] The role of model quantization is to convert high-precision data into low-precision data for calculation, which speeds up calculation and reduces the memory bandwidth. Currently, we support quantization precision of 4-bit and 8-bit quantization. We support quantization methods such as symmetric quantization, asymmetric quantization, and hierarchical quantization. At the same time, we have implemented the quantization kernel of various operators in the cognitive model to accelerate the reasoning process of the cognitive model.

[0181] 5. Hybrid Parallelism

[0182] In the process of deep learning reasoning, according to the different ways of splitting the model, it can be divided into model parallelism and tensor parallelism. Model parallelism means that different devices are responsible for the calculation of different parts of a single calculation graph; tensor parallelism means that the parameters (tensors) within the layer in the calculation graph are split to different devices (i.e., intra-layer parallelism), and each device only has a part of the model.

[0183] For the MLP multi-layer perceptron and self-attention layer in the cognitive model, model parallelism is preferred for computational acceleration. However, in cognitive models, computing self-attention is an optimization technique for performance when the input kv_cache is large. When the historical sequence is long, the kv_cache will be very large, and the video memory of a single device cannot store the kv_cache of all self-attention layers. It is necessary to combine model parallelism and tensor parallelism to run some self-attention layers of the model on different GPU devices.

[0184] Here is an explanation. For example, tensor parallelism is to run one self-attention layer on two GPUs, and two self-attention layers on two GPUs. Assuming there are 4 self-attention layers, it will lead to insufficient video memory. Then run two self-attention layers on two graphics cards, and the other two self-attention layers on the other two graphics cards.

[0185] 6. Reasoning Memory Optimization

[0186] a. Inference memory prediction

[0187] When the cognitive model predicts the generated result, it predicts the next text to be answered based on the question text and the existing answer text. In this process, the question text and the kv_cache corresponding to the generated text are saved to speed up the calculation process. During the calculation process, the kv_cache corresponding to the newly generated text and the kv_cache corresponding to the historical text need to be spliced, which will result in a large amount of memory application, release, and copying. In our system, the TDNN network is used to predict the memory size required during the reasoning process based on the input content, and the entire block of memory is applied at the beginning to reduce the cost of memory application, release, and copying.

[0188] TDNN (Time Delay Neural Network) is a time delay neural network used to process time series data. It can capture time-related information in sequence data and can learn and predict future data. The TDNN network is mainly composed of an input layer, a hidden layer, and an output layer. The input layer receives time series data, and the neurons in the hidden layer have different time delay windows and process data at different time steps. Each neuron receives input from the previous layer and performs convolution operations using a weight matrix of a fixed window, and then processes it through a nonlinear activation function. Depending on the task, the output layer can be a regression problem or a classification problem. The TDNN network has a wide range of applications in speech recognition, natural language processing and other fields. It can process input sequences of different lengths and has the characteristics of time invariance and position invariance. In addition, the TDNN network can also extract more advanced features by adding convolutional layers and pooling layers to further improve the performance of the model. Since the TDNN network has a simple structure, can effectively process contextual information, has high parallelism and fast speed, and will not affect the performance of the entire reasoning process, the TDNN network structure is preferred. Others such as RNN structure networks and Transformer structure networks are also applicable.

[0189] b. Memory expansion

[0190] In the actual calculation process, if the pre-applied memory is still not enough to calculate the output results, we apply for new memory size in proportion to the used memory size and merge the newly applied memory with the used memory.

[0191] 7. GPU offload

[0192] When we want to run our cognitive model system on a device with lower performance and smaller video memory, we support migrating data that does not need to be calculated temporarily from the video memory to the main memory when the video memory is insufficient, and then moving the data in the main memory to the video memory for calculation when actual calculation is needed.

[0193] 8. kv_cache database

[0194] For inactive sessions (which can be understood as no Q&A in the session for an hour), we persist (save) the kv_cache corresponding to the historical records in the session to the database. When the session is reactivated, the corresponding kv_cache is read from the database and loaded into the video memory for calculation.

[0195] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses an accelerated reasoning method and system for cognitive model question and answer prediction, which supports efficient reasoning of various cognitive models and efficient reloading of cognitive model conversations, and has the following beneficial effects:

[0196] 1. Supports mixed operation of CPU and GPU, and can run on any GPU card without GPU card.

[0197] 2. It can support and apply multiple cognitive models, making it easy to promote and use.

[0198] 3. Supports conversation memory. If the conversation is interrupted, you can continue the conversation at any time by continuing with the previous conversation record, which is more practical.

[0199] 4. Efficient question and answer generation achieved through in-memory prediction, model parameter quantization, model hybrid parallelism and kv database.

[0200] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0201] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An accelerated reasoning method for cognitive model question-answering prediction, characterized in that: include: Obtain a question text, and obtain a question Token based on the question text; Distribute the question Token to the cognitive model reasoning engine; Determining the running memory based on the problem token by delaying the neural network by time; The cognitive model reasoning engine performs accelerated reasoning and prediction on the question token based on the running memory, model parameter quantization and model hybrid parallelism to obtain a predicted answer; Also includes: Get the historical question and answer text; Obtaining a historical token based on the historical question and answer text; Distribute the historical Token to the cognitive model reasoning engine; The specific process of determining the running memory is: The problem token and the historical token are concatenated to obtain a concatenated token; Obtaining time series data based on the spliced ​​Tokens; The time series data is input into the time delay neural network consisting of an input layer, a hidden layer and an output layer; The input layer receives and sends the time series data to the hidden layer; The multiple neurons in the hidden layer receive the time series data respectively, and perform convolution operation through a weight matrix of a fixed window to obtain multiple first processed data; The plurality of first processed data are respectively processed by a nonlinear activation function to obtain a plurality of second processed data; The output layer receives the plurality of second processed data for processing to obtain a predicted running memory size.

2. The accelerated reasoning method for cognitive model question-answering prediction according to claim 1, characterized in that: The question Token is obtained based on the question text, specifically including: Convert the question text into question Token based on the vocabulary; The method for constructing the vocabulary at least includes: Byte-PairEncoding method, WordPiece method or SentencePiece method; The Byte-PairEncoding method is specifically as follows: split the words in the database into character granularity to obtain multiple adjacent bytes, count the frequency of each adjacent byte pair, merge the adjacent byte pairs with the highest frequency into new subwords, repeat the merging until the preset word list size is reached, and obtain the final word list; The WordPiece method is specifically as follows: splitting the words in the database into character granularity to obtain multiple adjacent bytes, counting the frequency of each adjacent byte pair, merging the adjacent byte pairs with the highest frequency into new subwords, merging the adjacent subwords that can improve the probability of the language model the most and adding them to the vocabulary, and repeating the merging and adding to the vocabulary until the preset vocabulary size is reached to obtain the final vocabulary; The SentencePiece method is specifically as follows: words and spaces in the database are split into character granularity to obtain multiple adjacent bytes, the frequency of occurrence of each adjacent byte pair is counted, and the adjacent byte pairs with the highest frequency are merged into new subwords, and the merging is repeated until the preset word list size is reached to obtain the final word list.

3. The accelerated reasoning method for cognitive model question-answering prediction according to claim 1, characterized in that: Also includes: Determine whether the operating memory obtained during the reasoning process of the cognitive model reasoning engine is sufficient to calculate the output result; If it is not enough, new memory is acquired in proportion to the size of the used memory; And merge the new memory with the used memory.

4. The accelerated reasoning method for cognitive model question-answering prediction according to claim 1, characterized in that: The model parameter quantification specifically includes: Converting the parameters of operators in the cognitive model inference engine from high-precision data to low-precision data; The model parameter quantization method at least includes: symmetric quantization, asymmetric quantization and hierarchical quantization.

5. The accelerated reasoning method for cognitive model question-answering prediction according to claim 4, characterized in that: The hybrid parallelism specifically includes: During the reasoning process, the cognitive model reasoning engine adopts a hybrid parallel mode of model parallelism and tensor parallelism; The model parallelism comprises: dividing all the operators in the cognitive model reasoning engine into a plurality of first sub-parts, each of the first sub-parts respectively running on a corresponding GPU; The tensor parallelism is as follows: based on the model parallelism, the operator is split into multiple second sub-parts, and each of the second sub-parts runs on a corresponding GPU.

6. The accelerated reasoning method for cognitive model question-answering prediction according to claim 1, characterized in that: Also includes: When the video memory of the cognitive model inference engine is insufficient, data temporarily not needed for inference calculation is migrated from the video memory to the internal memory, and when calculation is needed, the data in the internal memory is moved to the video memory for inference calculation.

7. The accelerated reasoning method for cognitive model question-answering prediction according to claim 1, characterized in that: Also includes: For questions and answers that have no conversations within a preset time, the kv cache corresponding to the historical records in the question and answer is saved to the cache database. When the question and answer is repeated, the corresponding kv cache is read from the cache database for inference calculation.

8. An accelerated reasoning system for cognitive model question-answer prediction, characterized in that: include: Question encoding acquisition module, distribution module, memory determination module and prediction module; The question code acquisition module is used to acquire the question text and obtain the question Token based on the question text; The distribution module is used to distribute the question Token to the cognitive model reasoning engine; The memory determination module is used to determine the running memory based on the problem Token by time delaying the neural network; The prediction module is used for the cognitive model reasoning engine to perform accelerated reasoning and prediction on the question token based on the running memory, model parameter quantization and model hybrid parallelism to obtain a predicted answer; Also includes: historical coding push module; The historical code push module is used to obtain historical question and answer texts; obtain historical tokens based on the historical question and answer texts; and distribute the historical tokens to the cognitive model reasoning engine; The memory determination module is also used for: Concatenate the problem Token and the history Token to obtain a concatenated Token; Obtaining time series data based on the spliced ​​Tokens; The time series data is input into the time delay neural network consisting of an input layer, a hidden layer and an output layer; The input layer receives and sends the time series data to the hidden layer; The multiple neurons in the hidden layer receive the time series data respectively, and perform convolution operation through a weight matrix of a fixed window to obtain multiple first processed data; The plurality of first processed data are respectively processed by a nonlinear activation function to obtain a plurality of second processed data; The output layer receives the plurality of second processed data for processing to obtain a predicted running memory size.

9. The accelerated reasoning system for cognitive model question-answering prediction according to claim 8, characterized in that: Also includes: Memory judgment acquisition module; The memory judgment acquisition module is used to judge whether the operating memory acquired during the reasoning process of the cognitive model reasoning engine is sufficient for calculating the output result; If it is not enough, new memory is acquired in proportion to the size of the used memory; And merge the new memory with the used memory.

10. The accelerated reasoning system for cognitive model question-answering prediction according to claim 8, characterized in that: Also includes: Data migration module; The data migration module is used to migrate data that is temporarily not needed for reasoning calculations from the video memory to the internal memory when the video memory of the cognitive model reasoning engine is insufficient, and to move the data in the internal memory to the video memory for reasoning calculations when calculations are needed.

11. The accelerated reasoning system for cognitive model question-answering prediction according to claim 8, characterized in that: Also includes: Data cache reading module; The data cache reading module is used to save the kv cache corresponding to the historical records in the questions and answers that have no conversations within a preset time to the cache database. When the questions and answers are asked again, the corresponding kv cache is read from the cache database for inference calculation.

Images