Inference calculation method and device, electronic equipment and storage medium

Abstract

The invention provides a reasoning calculation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a target text to be reasoned through a large language model; the target text is preprocessed, and all first tokens in the target text are obtained; through a target data loading strategy obtained by limiting parameters of each second token in N second tokens which need to be loaded and are processed recently and uniformly storing row and column data of a feedforward neural network layer of the large language model, target parameters used for reasoning are loaded from a persistent storage medium storing the large language model; wherein the second token comprises the first token and / or a token obtained by reasoning based on the first token; and performing reasoning calculation on the integer ID through the target parameter to complete reasoning of the new second token. According to the invention, the problems of high data transmission cost and low reasoning speed in the prior art are solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an inference calculation method and device, an electronic device and a storage medium. Background Art

[0002] With the widespread application of large language models (LLM) in fields such as chatbots, natural language generation, text understanding, and machine translation, private cloud service providers also need to support the needs of customers who provide large language model services.

[0003] In the related art, when a large language model deployed on a private cloud service provider is called, data is usually loaded from a non-volatile storage device (e.g., flash memory) to memory and then processed by the CPU or GPU. This method does not specifically optimize the data loading and storage process, resulting in high data transmission costs, which in turn leads to slow inference speeds. In addition, since the cloud service provider's memory resources are limited, when a customer's large language model runs on a memory-constrained device, it will be further limited by memory, resulting in high data transmission costs and slow inference speeds.

[0004] Therefore, there are technical problems in the related technology such as high data transmission cost and slow inference speed. Summary of the invention

[0005] The present application provides an inference calculation method and device, an electronic device and a storage medium to at least solve the technical problems of high data transmission cost and slow inference speed in related technologies.

[0006] According to one aspect of an embodiment of the present application, there is provided an inference calculation method, including:

[0007] Obtain the target text to be inferred by the large language model;

[0008] Preprocessing the target text to obtain all first tokens in the target text;

[0009] By limiting the parameters of each second token of the N second tokens that are most recently processed and the target data loading strategy obtained by uniformly storing the row and column data of the feedforward neural network layer of the large language model, the target parameters for inference are loaded from the persistent storage medium storing the large language model, wherein the second token includes the first token and / or a token inferred based on the first token;

[0010] The integer ID is inferred and calculated using the target parameter to complete the inference of the new second token, wherein the new second token is the second token located one position after the N second tokens.

[0011] Optionally, as in the aforementioned method, the parameters of each second token in the latest N second tokens to be loaded are limited, including:

[0012] Initialize the sliding buffer;

[0013] Load the parameters of each of the N second tokens processed most recently through the sliding buffer to reuse the neurons activated in the most recent N time steps

[0014] Optionally, as in the aforementioned method, the unified storage of row and column data of the feedforward neural network layer of the large language model includes:

[0015] The first weight matrix projected upward and the second weight matrix projected downward in the feedforward neural network layer are stored together to obtain a continuous data block.

[0016] Optionally, as in the aforementioned method, the loading of the target parameters for reasoning from the persistent storage medium storing the large language model comprises:

[0017] Predicting the sparsity of the feedforward neural network to obtain a prediction result, wherein the prediction result is used to indicate zero elements in the continuous data block;

[0018] Ignore zero elements in the continuous data blocks based on the prediction results to obtain a sparse matrix corresponding to the feedforward neural network;

[0019] The target parameter is obtained by loading the parameter of each second token in the N second tokens and the sparse matrix.

[0020] Optionally, as in the aforementioned method, the method further comprises:

[0021] According to the memory space required by the predetermined parameters for inference, memory pre-allocation is performed to obtain the target memory;

[0022] The step of loading the target parameters for reasoning from the persistent storage medium storing the large language model includes:

[0023] The target parameters are loaded into the target memory.

[0024] Optionally, as in the aforementioned method, loading the target parameters into the target memory includes:

[0025] Create multiple threads;

[0026] The target parameters are loaded into the target memory by the multiple threads.

[0027] Optionally, as in the aforementioned method, the method further comprises:

[0028] Determine the performance and hardware cost corresponding to each of the multiple hardware configurations; determine the usage cost under different candidate data loading strategies, wherein the candidate data loading strategies include corresponding data loading methods and hardware configurations;

[0029] Input the performance and hardware cost corresponding to each hardware configuration and the usage cost under different candidate data loading strategies into a preset hardware cost model to obtain a weighted cost of each candidate data loading strategy;

[0030] Among all the candidate data loading strategies, the candidate data loading strategy corresponding to the lowest weighted cost is determined as the target data loading strategy.

[0031] According to another aspect of the embodiment of the present application, there is also provided an inference computing device, including:

[0032] An acquisition module, used to acquire target text to be inferred by a large language model;

[0033] A preprocessing module, used for preprocessing the target text to obtain all first tokens in the target text;

[0034] A loading module, configured to load target parameters for inference from a persistent storage medium storing the large language model by limiting the parameters of each of the N second tokens processed most recently to be loaded and a target data loading strategy obtained by uniformly storing row and column data of a feedforward neural network layer of the large language model, wherein the second token includes the first token and / or a token inferred based on the first token;

[0035] An inference module is used to perform inference calculation on the integer ID through the target parameter to complete the inference of the new second token, wherein the new second token is the second token located one position after the N second tokens.

[0036] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein the memory is used to store a computer program; and the processor is used to execute the method steps in any of the above embodiments by running the computer program stored in the memory.

[0037] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the method steps in any of the above embodiments when executed.

[0038] In an embodiment of the present application, a target data loading strategy suitable for a large language model is determined by limiting the parameters of each second token in the N second tokens that are most recently processed and the target data loading strategy obtained after uniformly storing the row and column data of the feedforward neural network layer of the large language model, and loading the target parameters for reasoning from the persistent storage medium storing the large language model. Since the parameters of each second token in the N second tokens and the activated neurons need to be used later, the speed of read and write operations is improved by limiting the parameters of each second token in the N second tokens that are most recently processed, and the actual number of accesses to non-volatile storage devices such as flash memory storing the large language model is reduced. Moreover, by uniformly storing the row and column data of the feedforward neural network layer of the large language model, since larger data blocks can be read and written faster, the throughput of data reading and writing can be improved. Thus, the purpose of reducing data transmission time can be achieved, and the technical effect of reducing data transmission costs is achieved, thereby solving the problems of high data transmission costs and slow reasoning speed existing in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 is a schematic diagram of a hardware environment of an optional inference calculation method according to an embodiment of the present application;

[0042] Figure 2 is a flowchart of an optional reasoning calculation method according to an embodiment of the present application;

[0043] Figure 3 is a schematic diagram of another optional reasoning calculation method according to an embodiment of the present application;

[0044] Figure 4 is a flowchart of another optional reasoning calculation method according to an embodiment of the present application;

[0045] Figure 5 is a structural block diagram of an optional inference computing device according to an embodiment of the present application;

[0046] Figure 6 It is a structural block diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0048] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0049] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretation:

[0050] 1. Flash Memory: A non-volatile storage device used for long-term storage of data.

[0051] 2. DRAM: Dynamic random access memory, a volatile memory commonly used for temporary data storage in computers.

[0052] 3. Computing Core: refers to the hardware unit that performs computing tasks, such as CPU or GPU.

[0053] 4. Feed-Forward Neural Network Layer (FFN Layer): A basic layer in the deep learning model, responsible for passing input data to the next layer.

[0054] According to one aspect of the embodiments of the present application, a method for inference calculation is provided. Optionally, in this embodiment, the above-mentioned method for inference calculation can be applied to Figure 1 In the hardware environment composed of terminal 1402 and server 1404 shown in FIG. Figure 1 As shown, server 1404 is connected to terminal 1402 via a network, and can be used to provide services (such as game services, application services, etc.) for the terminal or a client installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 1404.

[0055] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal may not be limited to a PC, a mobile phone, a tablet computer, etc.

[0056] The inference calculation method of the embodiment of the present application can be executed by a server, a terminal, or a server and a terminal together. The terminal can also execute the inference calculation method of the embodiment of the present application by a client installed thereon.

[0057] Taking the inference calculation method in this embodiment executed by the server as an example, Figure 2 An inference calculation method provided in an embodiment of the present application includes the following steps:

[0058] Step S202: Obtain target text to be inferred by the large language model.

[0059] The reasoning calculation method in this embodiment can be applied to scenarios where natural language processing is required through a large language model deployed on a private cloud server, such as: text generation scenarios, natural language understanding scenarios, dialogue scenarios, text translation scenarios, question-answering scenarios, text summary generation scenarios, analysis of sentiment in text, code generation scenarios, etc., and can also be other natural language processing scenarios. In the embodiment of this application, the above-mentioned reasoning calculation method is explained by taking the dialogue scenario as an example. For other types of natural language processing scenarios, the above-mentioned reasoning calculation method is also applicable if there is no contradiction.

[0060] Taking the dialogue scenario as an example, by obtaining the target text to be inferred by the large language model, responding to the target text and realizing the dialogue.

[0061] The large language model is a model deployed on a private cloud server. The target text can be text information sent by the user to the server through a client or a web page, or it can be text information recognized from voice or video information.

[0062] Step S204, preprocessing the target text to obtain all first tokens in the target text.

[0063] Specifically, after obtaining the target text, preprocessing can be performed through the following steps to finally obtain all first tokens. A token refers to the smallest unit in the target text, which can be a word, a phrase, or a symbol, etc.:

[0064] 1. Text cleaning: remove noise data in the target text, such as extra blank characters, punctuation marks, special characters, etc.

[0065] 2. Normalization: Convert the target text into a unified form, such as converting all letters to lowercase to avoid understanding problems caused by differences in uppercase and lowercase letters. Furthermore, the normalization can also include fixing spelling errors, abbreviation expansions, etc. in the target text.

[0066] 3. Word segmentation: The text processed in the previous steps is divided into meaningful basic units, which are usually words or phrases (i.e., tokens). Optionally, the word segmentation method may vary depending on the language. For example, English is usually automatically segmented according to spaces, while Chinese requires a special word segmentation algorithm.

[0067] 4. Word embedding: Convert each basic unit obtained in step 3 into a numerical vector so that the large language model can process it. The basic units can be mapped to a multidimensional space so that basic units with similar meanings are close in the vector space.

[0068] 5. Sequence truncation and padding: If the target text is too long, you may need to truncate the part that exceeds the processing length of the large language model. If the input text is short, you may need to add special padding markers at the end of the sequence (such as <pad>) to keep the length of each sequence input to the large language model consistent.

[0069] 6. Special tags: Add special tags at the beginning of the sequence (such as <s> or [CLS]), add a closing tag at the end of the sequence (such as< / s> or [SEP]). The above markers help the large language model to identify the beginning and end of the sequence.

[0070] 7. Convert to model input format: Convert the processed text to the specific format required by the model, which usually involves converting the vocabulary to its corresponding integer ID. Integer ID is the way the model represents vocabulary internally, and they usually come from the model's vocabulary.

[0071] Step S206, by limiting the parameters of each second token of the N most recently processed second tokens to be loaded and the target data loading strategy obtained by uniformly storing the row and column data of the feedforward neural network layer of the large language model, the target parameters for inference are loaded from the persistent storage medium storing the large language model, wherein the second token includes the first token and / or a token inferred based on the first token.

[0072] In this embodiment, by limiting the parameters of each second token of the N second tokens that are most recently processed and need to be loaded, that is, when processing the target text, only the information of the N second tokens that are most recently processed is retained, so as to reduce memory consumption, and for some application scenarios, it is sufficient to only focus on the most recent information. For example, in a conversation scenario, only the historical information of the last few rounds of conversations needs to be considered. In addition, after obtaining the second token of the new data, the N second tokens can be updated, the highest data among the N second tokens can be deleted, and the second token of the new data can be added.

[0073] Since the output of each time step of the large language model depends not only on the current input, but also on the hidden state of the previous time step (i.e., the state of the previously activated neuron, i.e., the parameters of the corresponding second token), the parameters of each second token in the latest N second tokens to be loaded can be limited to obtain the hidden state of the most recent N time steps, thereby achieving the purpose of reusing the activated neurons.

[0074] As an optional implementation, the parameters of each second token in the N second tokens that are processed most recently and need to be loaded in the aforementioned step S206 can be implemented by the following steps: initialize a sliding buffer; load the parameters of each second token in the N second tokens that are processed most recently through the sliding buffer to reuse the neurons that have been activated in M ​​time steps. Specifically, a sliding buffer can be defined and its size can be set to accommodate the parameters of N second tokens. The parameters of the second token are the hidden states of the corresponding time steps, that is, the initialized buffer is empty or filled with the initial state. When a new second token is processed, the second token and the hidden state in the sliding buffer are used as input to calculate the new hidden state, and the new hidden state is added to the end of the sliding buffer. If the sliding buffer is full (i.e., reaches a predetermined size N), the earliest hidden state (i.e., the hidden state at the beginning of the buffer) is removed to keep the hidden state in the buffer as the hidden state of the most recent N time steps. When processing the next second token, the hidden states corresponding to the N second tokens in the sliding buffer are directly used without recalculation. This can reduce the read operation of the non-volatile storage system and improve the processing efficiency. Whenever a new second token is processed, the above steps are repeated, and the buffer is updated according to the hidden state of the new second token, and the current hidden state in the buffer is used for the next calculation.

[0075] As an optional implementation, the row and column data of the feedforward neural network layer of the large language model can be uniformly stored by the following steps: the first weight matrix projected upward and the second weight matrix projected downward in the feedforward neural network layer are stored together to obtain a continuous data block. Specifically, the first weight matrix (W_{up}) projected upward in the feedforward neural network layer is used to achieve mapping from a lower dimension to a higher dimension. The weight matrix (W_{down}) projected downward in the feedforward neural network layer is used to achieve mapping from a higher dimension back to a lower dimension. After determining the first weight matrix and the second weight matrix, (W_{up}) and (W_{down}) can be stacked in a specific order to form a large continuous matrix (W_{combined}), and then the continuous matrix (W_{combined}) is stored in a continuous manner in the memory to obtain a continuous data block in the memory. Through the method of this embodiment, storing different weight matrices together can effectively improve memory access efficiency and computing performance.

[0076] like Figure 3 As shown, as an optional implementation, as in the aforementioned method, the target parameters for inference in the aforementioned step S206 can be loaded from the persistent storage medium storing the large language model through the following steps S302 to S306:

[0077] Step S302, predict the sparsity of the feedforward neural network and obtain a prediction result, wherein the prediction result is used to indicate the zero elements in the continuous data block. Specifically, in this step, it is necessary to predict the sparsity of the weight matrix in the feedforward neural network, which means that it is necessary to predict which weight values ​​are close to zero so that the calculation of these weight values ​​can be ignored. Optionally, a threshold can be set to determine whether the weight value is close enough to zero, and the weight matrix can be traversed. For each weight value, check whether each weight value is less than the threshold. If it is less than, it is considered that the weight value can be regarded as zero. Finally, the weight values ​​that should be retained are recorded, and the weight values ​​of 0 are ignored.

[0078] Step S304, based on the prediction result, the zero elements in the continuous data block are ignored to obtain a sparse matrix corresponding to the feedforward neural network. Specifically, after the prediction result is determined, since the continuous data block is the weight matrix, a sparse matrix can be created according to the prediction result and the continuous data block, and the non-zero elements in the continuous data block are retained in the sparse matrix, and the zero elements are ignored.

[0079] Step S306, by loading the parameters and sparse matrix of each second token in the N second tokens, the target parameters are obtained. That is, the target parameters are constructed using the parameters and sparse matrix of the N second tokens, and the constructed target parameters are used for subsequent calculation or reasoning process. Optionally, the target parameters at least include the parameters and sparse matrix of the N second tokens.

[0080] As an optional implementation, as in the aforementioned method, before loading data into the memory, memory pre-allocation can be performed in advance according to the memory space required for the predetermined parameters used for reasoning to obtain the target memory. In other words, the memory space required for the large language model to be occupied during the reasoning process is predetermined, and the memory space can be the upper limit of the memory that the large language model will occupy during the reasoning process. Then, memory pre-allocation can be performed according to the memory space to obtain the target memory. After determining the target parameters, the target parameters can be loaded into the allocated target memory.

[0081] As an optional implementation, as in the aforementioned method, the target parameters can be loaded into the target memory by the following steps, including: creating multiple threads; loading the target parameters into the target memory by multiple threads. Optionally, multiple threads for data loading can be created, and then each thread can be started to start executing the data loading task to jointly load the target parameters into the target memory. After ensuring that all threads complete the task, it can be determined that the target parameters are successfully loaded into the target memory.

[0082] Step S208, performing inference calculation on the integer ID through the target parameter to complete the inference of the new second token, wherein the new second token is the second token located one position after the N second tokens.

[0083] Based on the above embodiment, the target parameters of the sparse matrix of the feedforward neural network and the parameters of the N second tokens processed most recently have been obtained. Then, on the CPU or GPU, an input vector can be constructed by using the parameters of the N second tokens, and then the input vector is calculated using the sparse matrix to obtain an output vector, and finally the output vector is converted into an integer ID as the prediction result of the new second token. The new second token is specifically: a second token located after the most recently processed second token among the N second tokens, and the reasoning of all second tokens can be completed by repeatedly executing steps S206 to S208.

[0084] The present disclosure adopts a method of determining a target data loading strategy suitable for a large language model, and the target data loading strategy obtained by limiting the parameters of each second token in the N second tokens that are processed most recently and uniformly storing the row and column data of the feedforward neural network layer of the large language model is loaded from the persistent storage medium storing the large language model to the target parameters for reasoning. Since the parameters of each second token in the N second tokens and the activated neurons need to be used later, the speed of the read and write operations is improved by limiting the parameters of each second token in the N second tokens that are processed most recently, and the actual number of accesses to non-volatile storage devices such as flash memory storing the large language model is reduced. Moreover, by uniformly storing the row and column data of the feedforward neural network layer of the large language model, since larger data blocks can be read and written faster, the throughput of data reading and writing can be improved. Thus, the purpose of reducing data transmission time can be achieved, and the technical effect of reducing data transmission cost is achieved, thereby solving the problems of high data transmission cost and slow reasoning speed existing in the related art.

[0085] like Figure 4 As shown, as an optional implementation, as the above method, the method further includes the following steps:

[0086] Step S402: determining the performance and hardware cost corresponding to each of the multiple hardware configurations.

[0087] Specifically, according to different usage requirements (such as the amount of data to be processed, computing requirements, storage requirements, etc.), appropriate CPU, GPU, memory and other hardware can be selected according to the requirements, and different hardware configurations can be built. For each hardware configuration, the price of each hardware included can be obtained, and the hardware cost of each hardware configuration can be estimated.

[0088] Moreover, for each hardware configuration, you can benchmark the selected hardware configuration to understand its performance indicators, such as computing speed, memory bandwidth, etc. In addition, you can simulate the load of actual applications to evaluate the performance of the hardware configuration under different loads. Finally, you can analyze the performance bottleneck of the hardware configuration and determine which components may become limiting factors. Based on this, you can get the corresponding performance of each hardware configuration.

[0089] Step S404: determining the usage costs under different candidate data loading strategies, wherein the candidate data loading strategies include corresponding data loading methods and hardware configurations.

[0090] Specifically, different data loading methods can be defined, including but not limited to on-demand loading, preloading, sliding buffer, etc. The impact of each data loading method on the performance of each hardware configuration is analyzed, including memory usage, CPU utilization, I / O operations, etc. Then, a candidate data loading strategy containing a one-to-one correspondence between data loading methods and hardware configurations can be obtained.

[0091] Step S406: input the performance and hardware cost corresponding to each hardware configuration and the usage costs under different candidate data loading strategies into a preset hardware cost model to obtain a weighted cost of each candidate data loading strategy.

[0092] Specifically, after obtaining the performance and hardware cost corresponding to each hardware configuration, the usage cost under different candidate data loading strategies is input into the preset hardware cost model. The hardware cost model can be a model for weighted calculation of performance, hardware cost and usage cost. For example, for any hardware configuration, its performance score, hardware cost score and usage cost score can be determined according to its performance, hardware cost and usage cost, and then weighted calculation is performed by using the first weight, second weight and third weight corresponding to the performance score, hardware cost score and usage cost score respectively, so as to obtain the weighted cost.

[0093] Step S408: Determine the candidate data loading strategy with the lowest corresponding weighted cost among all the candidate data loading strategies as the target data loading strategy.

[0094] Specifically, after determining the weighted cost of each candidate data loading strategy, the candidate data loading strategy with the lowest weighted cost among all candidate data loading strategies can be determined, and then the candidate data loading strategy with the lowest weighted cost can be determined as the target data loading strategy.

[0095] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0096] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0097] According to another aspect of an embodiment of the present application, an inference computing device for implementing the above-mentioned inference computing method is also provided. Figure 5 is a structural block diagram of an optional inference computing device according to an embodiment of the present application, such as Figure 5 As shown, the device may include:

[0098] An acquisition module 51 is used to acquire a target text to be inferred by a large language model;

[0099] A preprocessing module 52, used to preprocess the target text to obtain all first tokens in the target text;

[0100] A loading module 53 is configured to load a target parameter for inference from a persistent storage medium storing a large language model by limiting the parameters of each of the N second tokens processed most recently to be loaded, reusing activated neurons, and uniformly storing row and column data of a feedforward neural network layer of the large language model, wherein the second token includes the first token and / or a token inferred based on the first token;

[0101] The inference module 54 is used to perform inference calculation on the integer ID through the target parameter to complete the inference of the new second token, wherein the new second token is the second token located after the N second tokens.

[0102] It should be noted that the acquisition module 51 in this embodiment can be used to execute the above step S202, the preprocessing module 52 in this embodiment can be used to execute the above step S204, the loading module 53 in this embodiment can be used to execute the above step S206, and the inference module 54 in this embodiment can be used to execute the above step S208.

[0103] Through the above modules, the purpose of reducing data transmission time can be achieved, and the technical effect of reducing data transmission costs can be achieved, thereby solving the problems of high data transmission costs and slow inference speed existing in related technologies.

[0104] As an optional implementation, as in the aforementioned device, the loading module 53 includes:

[0105] An initialization unit, used for initializing a sliding buffer;

[0106] A sliding buffer loading unit is used to load the parameters of each of the most recently processed N second tokens through a sliding buffer to reuse neurons that have been activated in M ​​time steps.

[0107] As an optional implementation, as in the aforementioned device, the loading module 53 includes:

[0108] The continuous storage unit is used to store the first weight matrix projected upward and the second weight matrix projected downward in the feedforward neural network layer together to obtain a continuous data block.

[0109] As an optional implementation, as in the aforementioned device, the loading module 53 includes:

[0110] A prediction unit, used for predicting the sparsity of the feedforward neural network to obtain a prediction result, wherein the prediction result is used to indicate a zero element in a continuous data block;

[0111] A sparse matrix determination unit, used for ignoring zero elements in continuous data blocks based on the prediction results to obtain a sparse matrix corresponding to the feedforward neural network;

[0112] The target parameter determination unit is used to obtain the target parameter by loading the parameters of each second token in the N second tokens, the hidden state corresponding to the activated neuron, and the sparse matrix.

[0113] As an optional implementation, the aforementioned device further includes:

[0114] A pre-allocation module is used to pre-allocate memory according to the memory required by the predetermined parameters for inference to obtain the target memory;

[0115] The loading module 53 is also used to load the target parameters into the target memory.

[0116] As an optional implementation, as in the aforementioned device, the loading module 53 is configured as follows:

[0117] Create multiple threads;

[0118] Load target parameters into target memory via multiple threads.

[0119] As an optional implementation, the aforementioned device further includes a policy determination module, wherein the policy determination module is configured to:

[0120] Determine the performance and hardware cost corresponding to each of the multiple hardware configurations; determine the usage cost under different candidate data loading strategies, wherein the candidate data loading strategies include corresponding data loading methods and hardware configurations;

[0121] Input the performance and hardware cost corresponding to each hardware configuration and the usage cost under different candidate data loading strategies into the preset hardware cost model to obtain the weighted cost of each candidate data loading strategy;

[0122] Among all the candidate data loading strategies, the candidate data loading strategy with the lowest corresponding weighted cost is determined as the target data loading strategy.

[0123] The device in this embodiment, in addition to the above modules, may also include a module for executing any method in the embodiments of any of the above-mentioned reasoning and calculation methods.

[0124] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware, wherein the hardware environment includes a network environment.

[0125] According to another aspect of an embodiment of the present application, an electronic device for implementing the above-mentioned inference calculation method is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0126] According to another embodiment of the present application, there is also provided an electronic device, including: Figure 6 As shown, the electronic device may include: a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .

[0127] Memory 1503, used for storing computer programs;

[0128] The processor 1501 is used to implement the following steps when executing the program stored in the memory 1503:

[0129] Step S202: Obtain target text to be inferred by the large language model.

[0130] Step S204, preprocessing the target text to obtain all first tokens in the target text.

[0131] Step S206, by limiting the parameters of each second token of the N most recently processed second tokens to be loaded and the target data loading strategy obtained by uniformly storing the row and column data of the feedforward neural network layer of the large language model, the target parameters for inference are loaded from the persistent storage medium storing the large language model, wherein the second token includes the first token and / or a token inferred based on the first token.

[0132] Step S208, performing inference calculation on the integer ID through the target parameter to complete the inference of the new second token.

[0133] Optionally, in this embodiment, the above-mentioned communication bus can be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the above-mentioned electronic device and other devices.

[0134] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0135] As an example, the memory 1503 may include but is not limited to the acquisition module 51, preprocessing module 52, loading module 53 and reasoning module 54 in the reasoning computing device. In addition, it may also include but is not limited to other module units in the reasoning computing device, which will not be repeated in this example.

[0136] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0137] An embodiment of the present application further provides a computer-readable storage medium, the storage medium including a stored program, wherein the method steps of the above method embodiment are executed when the program is run.

[0138] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0139] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0140] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0141] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0142] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0143] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.

[0144] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0145] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.< / pad>

Claims

1. A method of inference calculation, characterized in that: include: Obtain the target text to be inferred by the large language model; Preprocessing the target text to obtain all first tokens in the target text; By limiting the parameters of each second token of the N second tokens that are most recently processed and the target data loading strategy obtained by uniformly storing the row and column data of the feedforward neural network layer of the large language model, the target parameters for inference are loaded from the persistent storage medium storing the large language model, wherein the second token includes the first token and / or a token inferred based on the first token; The integer ID is inferred and calculated using the target parameter to complete the inference of the new second token, wherein the new second token is the second token located one position after the N second tokens.

2. The method according to claim 1, characterized in that The parameters of each second token in the latest N second tokens to be loaded are limited, including: Initialize the sliding buffer; The parameters of each of the N second tokens processed most recently are loaded through the sliding buffer to reuse neurons activated in the most recent N time steps.

3. The method according to claim 1, characterized in that The unified storage of row and column data of the feedforward neural network layer of the large language model includes: The first weight matrix projected upward and the second weight matrix projected downward in the feedforward neural network layer are stored together to obtain a continuous data block.

4. The method according to claim 3, characterized in that: The step of loading the target parameters for reasoning from the persistent storage medium storing the large language model includes: Predicting the sparsity of the feedforward neural network to obtain a prediction result, wherein the prediction result is used to indicate zero elements in the continuous data block; Ignore zero elements in the continuous data blocks based on the prediction results to obtain a sparse matrix corresponding to the feedforward neural network; The target parameter is obtained by loading the parameter of each second token in the N second tokens and the sparse matrix.

5. The method according to claim 1, characterized in that The method further comprises: According to the memory space required by the predetermined parameters for inference, memory pre-allocation is performed to obtain the target memory; The step of loading the target parameters for reasoning from the persistent storage medium storing the large language model includes: The target parameters are loaded into the target memory.

6. The method according to claim 5, characterized in that The step of loading the target parameters into the target memory comprises: Create multiple threads; The target parameters are loaded into the target memory by the multiple threads.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Determine the performance and hardware cost corresponding to each of the multiple hardware configurations; determine the usage cost under different candidate data loading strategies, wherein the candidate data loading strategies include corresponding data loading methods and hardware configurations; Input the performance and hardware cost corresponding to each hardware configuration and the usage cost under different candidate data loading strategies into a preset hardware cost model to obtain a weighted cost of each candidate data loading strategy; Among all the candidate data loading strategies, the candidate data loading strategy corresponding to the lowest weighted cost is determined as the target data loading strategy.

8. An inference computing device, characterized in that: include: An acquisition module, used to acquire target text to be inferred by a large language model; A preprocessing module, used for preprocessing the target text to obtain all first tokens in the target text; A loading module, configured to load target parameters for inference from a persistent storage medium storing the large language model by limiting the parameters of each of the N second tokens processed most recently to be loaded and a target data loading strategy obtained by uniformly storing row and column data of a feedforward neural network layer of the large language model, wherein the second token includes the first token and / or a token inferred based on the first token; An inference module is used to perform inference calculation on the integer ID through the target parameter to complete the inference of the new second token, wherein the new second token is the second token located one position after the N second tokens.

9. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface and the memory communicate with each other via the communication bus, wherein: The memory is used to store computer programs; The processor is configured to execute the method according to any one of claims 1 to 7 by running the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 7 when executed.

Images