Inference device and method, medium and chip

Abstract

The invention discloses a reasoning device and method, a medium and a chip, the device is used for reasoning operation of a large language model, and the device comprises a memory module, a calculation module and a control module; the control module receives a processing instruction, obtains a target number of sequences, stores the sequences in the memory module, releases the processed sequences and the storage space and the computing resources occupied by the corresponding reasoning result after any sequence is processed, selects another sequence from the remaining sequences, and stores the other sequence in the storage space released by the reasoning sequence. When the to-be-processed text is reasoned, after reasoning of any sequence is completed, the storage space and the computing resources corresponding to the sequence can be released, and the new sequence is stored for reasoning, so that the utilization rate of a memory module and a computing module is greatly improved, and the reasoning efficiency of a large language model is improved.

Description

Technical Field

[0001] The present application relates to the field of reasoning of large language models, and in particular to a reasoning device, method, medium and chip. Background Art

[0002] At present, in the reasoning process of large language models (LLMs), external high-performance chip simulators are often used for reasoning based on batch processing strategies. Among them, the batch processing strategy commonly used is the static batch processing strategy. When the static batch processing strategy is adopted, each batch includes multiple sequences, and multiple sequences in the same batch are inferred and iterated in parallel. However, since an external high-performance chip is called, it is impossible to determine the usage of chip memory and computing resources. Therefore, when performing parallel reasoning iterations on each sequence in the same batch, the reasoning iteration of the batch can only be ended after the last sequence reasoning iteration is completed, and the storage space and computing resources occupied by each sequence in the batch can be released. However, different sequences in the same batch require different numbers of iterations during reasoning, that is, the reasoning iterations of each sequence in the same batch are not completed at the same time. Therefore, among the sequences in the same batch, the memory and computing resources occupied by the sequences that complete the reasoning iteration earlier also need to be released together after the last sequence reasoning iteration, resulting in low utilization of memory and computing resources and low reasoning efficiency. Summary of the invention

[0003] The main purpose of this application is to propose an inference device, method, medium and chip, which aims to solve the problem that when performing inference iterations in the prior art, the memory and computing resources occupied by the sequences that completed the inference iteration earlier in the same batch of sequences need to be released together after the last sequence inference iteration, resulting in low memory and computing resource utilization and low inference efficiency.

[0004] To achieve the above object, the present application proposes an inference device for inference operation of a large language model, wherein the large language model is deployed on the inference device, and the inference device includes: a memory module, a computing module, and a control module; The control module is configured to: in response to receiving a processing instruction, obtain a target number of sequences from the text to be processed to form batch data; Storing the batch data in the memory module for the calculation module to call; After any sequence in the batch data is processed, the storage space and computing resources occupied by the processed sequence and the corresponding inference result are released; Another sequence is selected from the remaining sequences of the text to be processed and stored in the storage space released by the sequence that has been inferred.

[0005] In an embodiment of the present application, the control module is further configured to determine the target quantity based on the following method: Determining a first maximum target number based on the current available memory of the memory module and the maximum length of each sequence in the text to be processed; Determine a second maximum target quantity based on the current available computing resource size of the computing module and the unit sequence computing resource consumption of each sequence in the to-be-processed text; The smaller value between the first maximum target quantity and the second maximum target quantity is used as the target quantity.

[0006] In an embodiment of the present application, the control module is further configured to determine the first maximum target quantity based on the following method:

[0007] in, represents the first maximum target number, represents the total storage space of the memory module, represents the used storage space of the memory module, S1 represents the first safety factor, represents the maximum length of each sequence in the text to be processed, Represents the memory footprint of a sequence of unit length.

[0008] In the embodiment of the present application, the control module is further configured to determine the second maximum target number based on the following method:

[0009] in, represents the second largest number of targets, represents the total computing resource size of the computing module, represents the size of computing resources used by the computing module, S2 represents the second safety factor, Represents the computational resource consumption of a unit sequence.

[0010] In the embodiment of the present application, the control module is further configured to: when the length of any of the sequences exceeds a preset length, divide the sequence into a plurality of blocks, and store each divided block into the memory module; Each block contains a plurality of text information of the text to be processed, the length of each block does not exceed the preset length, and the end of the previous block and the header of the next block of two adjacent blocks have overlapping text information.

[0011] In the embodiment of the present application, the control module is further configured as follows: Assign a prefix to any sequence when calling it; Determine whether the preset database has an inference result corresponding to the prefix; If yes, the inference result corresponding to the prefix is ​​called from the preset database, and the inference process of the sequence is skipped; If not, the sequence is inferred and the inference result is stored in the preset database.

[0012] The present application also proposes a reasoning method, which is applied to the reasoning device described in any of the above embodiments, and the reasoning method includes: In response to receiving the processing instruction, obtaining a target number of sequences from the text to be processed to form batch data; storing the batch data into the memory module; After any sequence in the batch data is processed, the storage space and computing resources occupied by the processed sequence and the corresponding inference result are released; Another sequence is selected from the remaining sequences of the text to be processed and stored in the storage space released by the sequence that has been inferred.

[0013] In the embodiment of the present application, the reasoning method further includes: Determining a first maximum target number based on the current available memory of the memory module and the maximum length of each sequence in the text to be processed; Determine a second maximum target quantity based on the current available computing resource size of the computing module and the unit sequence computing resource consumption of each sequence in the to-be-processed text; The smaller value between the first maximum target quantity and the second maximum target quantity is used as the target quantity.

[0014] In the embodiment of the present application, the first maximum target quantity is determined by the following method:

[0015] in, represents the first maximum target number, represents the total storage space of the memory module, represents the used storage space of the memory module, S1 represents the first safety factor, represents the maximum length of each sequence in the text to be processed, Represents the memory usage per unit length.

[0016] In the embodiment of the present application, the second maximum target number is determined based on the following method:

[0017] in, represents the second largest number of targets, represents the total computing resource size of the computing module, represents the size of computing resources used by the computing module, S2 represents the second safety factor, Represents the computational resource consumption of a unit sequence.

[0018] In the embodiment of the present application, the reasoning method further includes: When the length of any of the sequences exceeds a preset length, the sequence is divided into a plurality of blocks, and each of the divided blocks is stored in the memory module; Each block contains a plurality of text information of the text to be processed, the length of each block does not exceed the preset length, and the end of the previous block and the header of the next block of two adjacent blocks have overlapping text information.

[0019] In the embodiment of the present application, the reasoning method further includes: Assign a prefix to any sequence when calling it; Determine whether the preset database has a processing result corresponding to the prefix; If yes, the processing result corresponding to the prefix is ​​called from the preset database, and the reasoning process of the sequence is skipped; If not, the sequence is inferred and the processing result is stored in the preset database.

[0020] The present application also proposes a readable storage medium, which includes instructions. When the instructions are executed on a computing device, the computing device executes any of the above methods.

[0021] The present application also proposes a chip, comprising the inference device described in any of the above embodiments.

[0022] In an embodiment of the present application, when reasoning is performed on the text to be processed, after any sequence completes reasoning, the storage space and computing resources corresponding to the sequence can be released, and a new sequence can be stored for reasoning, thereby greatly improving the utilization rate of the memory module and the computing module, and improving the reasoning efficiency of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0024] Figure 1 A schematic diagram of iterative reasoning of a single sequence by a large language model in the prior art; Figure 2a , 2b A schematic diagram of batch processing for iterative reasoning of a large language model in the prior art; Figure 3 This is a module diagram of an inference device in one embodiment of the present application; Figure 4a , 4b 4c is a schematic diagram of the reasoning device in the embodiment of the present application when performing reasoning; Figure 5 is a step diagram of the reasoning method in one embodiment of the present application; Figure 6 A module diagram of a readable storage medium in an embodiment of the present application.

[0025] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0026] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present application, and are not intended to limit the scope of the present application in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0027] Those skilled in the art know that the embodiments of the present application can be implemented as a system, device, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0028] According to the implementation manner of the present application, an inference device, method, medium and chip are proposed.

[0029] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0030] The principles and spirit of the present application are explained in detail below with reference to several representative implementations of the present application.

[0031] Exemplary Devices Reasoning in large language models (LLMs) is an iterative process, obtaining an additional completion token after each new feed-forward loop. For example, given the input prompt word (sequence) "What is the capital of California", the large language model needs to go through ten feed-forward loops, outputting the letter sequence ["S","a","c","r","a","m","e","n","t","o"] one by one, and finally these scattered letters are spliced ​​into the complete reasoning result "Sacramento".

[0032] like Figure 1 As shown, Figure 1 This is the iterative process of reasoning a single sequence by a large language model in the prior art. The sequence [X1, X2, X3] is input to the large language model. After three reasoning iterations, the reasoning result [T1, T2, T3] is obtained. The large language model obtains the reasoning result T1 in the first reasoning, the reasoning result T2 in the second reasoning, and the reasoning result T3 in the third reasoning, and then generates the identifier End, indicating that the reasoning iteration of the sequence has ended.

[0033] Figure 1 The inference iteration process for a single sequence is shown. However, in order to improve the inference efficiency of a large language model, inference iteration is often performed on batch data. A batch of data may contain multiple sequences, that is, multiple sequences are inferred and iterated in parallel.

[0034] like Figure 2a As shown, 2a shows a batch of data input to a large language model, which contains sequences 1-4, sequence 1 includes X4-X6, sequence 2 includes X7-X8, sequence 3 includes X9-X11, and sequence 4 includes X12-X15.

[0035] Figure 2b The inference results of this batch of data are shown, where the inference result of sequence 1 is [T4, T5], which means that two inference iteration processes are required; The inference result of sequence 2 is [T6, T7, T8, T9], which means that 4 inference iterations are required; The inference result of sequence 3 is [T10], which means that an iterative inference process is required; The inference result of sequence 4 is [T11, T12], which means two inference iterations are required.

[0036] However, in the prior art, the reasoning process of large language models (LLMs) is often performed based on a batch processing strategy using an external high-performance chip simulator. Since an external high-performance chip is called, the usage of chip memory and computing resources cannot be determined. Therefore, when performing parallel reasoning iterations based on each sequence in the same batch, the reasoning iteration of the batch can only be ended after the last sequence reasoning iteration is completed, and the storage space and computing resources occupied by the sequences in the batch can be released. That is, for sequences 1-4 in the same batch of data, although sequence 3 only needs one iteration process to obtain the reasoning result, and sequences 1 and 4 only need two iteration processes to obtain the reasoning result, it is still necessary to wait for sequence 2 to obtain the reasoning result after four iterations before completing the reasoning process of this batch of data, and then the memory and computing resources occupied by sequences 1-4 and the corresponding reasoning results can be released for storing other batches of data. It can be seen that the memory and computing resources occupied by sequences 1, 3, and 4 are obviously wasted. In addition, for large language models that perform inference based on external high-performance chips, storing 1M batch data in the storage space of the external high-performance chip often takes longer than the time consumed by inference and iteration of 1M batch data. Therefore, occupying memory space for a long time without storing the next sequence to be inferred and iterated into the memory space occupied by the sequence that has been inferred and iterated will undoubtedly lead to low inference efficiency of large language models.

[0037] Based on this, Figure 3 As shown, this exemplary embodiment proposes an inference device for inference operation of a large language model, the large language model is deployed on the inference device, and the inference device includes: a memory module, a calculation module and a control module; The control module is configured to: in response to receiving a processing instruction, obtain a target number of sequences from the text to be processed to form batch data; Storing the batch data in the memory module for the calculation module to call; After any sequence in the batch data is processed, the storage space and computing resources occupied by the processed sequence and the corresponding inference result are released; Another sequence is selected from the remaining sequences of the text to be processed and stored in the storage space released by the sequence that has been inferred.

[0038] Among them, in an embodiment of the present application, the deployment of a large language model on an inference device can be implemented based on assembly language. For example, the large language model is deployed in a control module of the inference device using assembly language, thereby allowing the large language model to directly control various modules of the inference device, such as monitoring the usage of a memory module and a computing module, clearing data in the memory module, releasing computing resources of the computing module, and so on.

[0039] like Figure 3 As shown, in an embodiment of the present application, the memory module may be a memory with read and write functions, the computing module may be a processor with computing capabilities such as a CPU and a GPU, and the control module may be a large language model deployed based on assembly language.

[0040] In an embodiment of the present application, the text to be processed may be a prompt word input to the large language model in the control module, the prompt word includes multiple sequences, and the processing instruction may be an instruction for reasoning based on the input prompt word. When the large language model receives the processing instruction, it obtains a target number of sequences from the multiple sequences in the text to be processed to form a batch of data, and stores the batch of data containing the target number of sequences in the memory module, so that it can be called by the large language model when reasoning based on the calculation module. That is, the target number refers to the number of sequences contained in the batch data.

[0041] In the embodiment of the present application, the control module is further configured to determine the target quantity based on the following method: Determining a first maximum target number based on the current available memory of the memory module and the maximum length of each sequence in the text to be processed; Determine a second maximum target quantity based on the current available computing resource size of the computing module and the unit sequence computing resource consumption of each sequence in the to-be-processed text; The smaller value between the first maximum target quantity and the second maximum target quantity is used as the target quantity.

[0042] Among them, the total storage space of the memory module is known, assuming that , the control module can monitor the used space of the memory module in real time, assuming that , ( ) is the current available memory of the memory module. In the embodiment of the present application, a first safety factor S1 is also introduced, which is generally less than 1, ( ) is the part of the available memory of the current memory module that can be used to store batch data under the premise that the first safety factor is S1. The first safety factor is obtained based on experience and is introduced to ensure that the memory module can store the target number of sequences and also has space for storing the inference results corresponding to the target number of sequences; in addition, the introduction of the first safety factor can also avoid failures caused by the memory module being full.

[0043] In addition, when the text to be processed is input into the large language model, the large language model will divide the text to be processed into multiple sequences. During the division process, the length of each sequence contained in it can be determined. Therefore, the maximum length of each sequence in the text to be processed can be determined. Assume that the maximum length of each sequence is , the memory space occupied by the unit length sequence is known, assuming that , then we can determine that the storage space required for the longest sequence is .

[0044] After determining the maximum storage space that the memory module can use to store batch data while ensuring safety, and determining the storage space required for the longest sequence, the size of the first maximum target quantity that the memory module can store can be determined, as shown in the following formula (1): (1) in, represents the first maximum target number, represents the total storage space of the memory module, represents the used storage space of the memory module, S1 represents the first safety factor, represents the maximum length of each sequence in the training data, Represents the memory footprint of a sequence of unit length.

[0045] In the embodiment of the present application, the total computing resource size of the computing module is known, assuming that , the control module can monitor the size of the computing resources used by the computing module in real time. Assuming that the size of the computing resources used is , then based on The size of the currently available computing resources can be calculated.

[0046] In the embodiment of the present application, a second safety factor S2 is also introduced. S2 is determined based on experience and is generally less than 1. (Then ) is the maximum value of the computing resources currently available in the computing module that can be used for inference iteration. The second safety factor is set to avoid using all the computing resources in the computing module and to ensure that there is some computing power left to avoid failures.

[0047] In addition, the size of the computational resources required for the unit sequence (a sequence) during the inference iteration is the same, assuming that .

[0048] After determining the maximum value of the currently available computing resources of the computing module that can be used for inference iterations and the size of the computing resources consumed by the unit sequence, the second maximum target number can be determined as follows: (2) in, represents the second largest number of targets, represents the total computing resource size of the computing module, represents the size of computing resources used by the computing module, S2 represents the second safety factor, Represents the computational resource consumption of a unit sequence.

[0049] Among them, the first maximum target number is determined based on the available storage space in the memory module, and the second maximum target number is determined based on the size of available computing resources in the computing module. The embodiment of the present application selects the smaller value of the first maximum target number and the second maximum target number as the target number of batch data, which can not only ensure that the memory module has sufficient storage space for storing each sequence in the batch data, but also ensure that the computing module also has sufficient computing resources for calling when performing inference iterations on each sequence in the batch data.

[0050] After the control module determines the target number of sequences contained in the batch data, it can obtain the target number of sequences from the text to be processed to form the batch data and store it in the memory module, such as Figure 4a As shown, assuming that the number of targets is 4, that is, there are 4 sequences in the batch data, and the 4 sequences are divided into: [S1, S1', S1'']; [S2, S2']; [S3, S3', S3'']; [S4, S4', S4'', S4''' ].

[0051] The control module stores the four sequences into the first storage space of the memory module, wherein the first storage space is a storage resource in the storage module for storing sequences, and the second storage space is a location in the storage module for storing inference results. In order to facilitate understanding of this solution, it is divided into the first storage space and the second storage space. In fact, the first storage space and the second storage space are both available storage spaces in the memory module.

[0052] Continue to refer to Figure 4a , after the first inference iteration, the result corresponding to the sequence [S1, S1', S1''] is P1; The inference result corresponding to the sequence [S2, S2'] is Q1; The inference result corresponding to the sequence [S3, S3', S3''] is W1, and the inference iteration ends, that is, the final inference result of the sequence [S3, S3', S3''] is [W1]; The inference result corresponding to the sequence [S4, S4', S4'', S4'''] is E1.

[0053] Then the second inference iteration is performed, such as Figure 4bAs shown, it should be noted that, while performing the second reasoning iteration, the control module clears the sequence [S3, S3', S3''] and the corresponding reasoning result [W1] that have been reasoned in the first reasoning iteration, thereby releasing the storage space occupied by them in the memory module and releasing the computing resources when reasoning the sequence [S3, S3', S3'']. In addition, the control module also selects another sequence from the remaining sequences of the text to be processed and stores it in the storage space released after clearing the sequence [S3, S3', S3''], as shown in FIG. Figure 4b As shown, in the embodiment of the present application, the control module selects the sequence [S5, S5', S5''] from the remaining sequence of the text to be processed and stores it in the storage space released after the sequence [S3, S3', S3''] is cleared.

[0054] In addition, if Figure 4b As shown, the control module is performing the second reasoning iteration while clearing the sequence [S3, S3', S3''] and the reasoning result [W1] and storing them in the sequence [S5, S5', S5'']. It should be noted that the time to clear a sequence and the corresponding reasoning result and store them in a new sequence is not necessarily the same as the time to complete one reasoning iteration. For the convenience of example, in the embodiment of the present application, it is assumed that the time to clear a sequence and the corresponding reasoning result and store them in a new sequence is the same as the time to complete one reasoning iteration. Then, when the new sequence [S5, S5', S5''] is stored in the memory module, the second reasoning iteration is completed. At this time, the corresponding reasoning results of each sequence are as follows: The inference results corresponding to the sequence [S1, S1', S1''] are P1 and P2, and the inference has been completed, that is, the final inference result of the sequence [S1, S1', S1''] is [P1, P2]; The inference results corresponding to the sequence [S2, S2'] are Q1, Q2; The sequence [S5, S5', S5''] is stored in the memory module, and the inference iteration has not yet started; The reasoning results corresponding to the sequence [S4, S4', S4'', S4'''] are E1 and E2, and the reasoning has been completed, that is, the final reasoning result of the sequence [S4, S4', S4'', S4'''] is [E1, E2].

[0055] Then the third inference iteration begins, as Figure 4cAs shown, at the beginning of the third reasoning iteration, the control module also clears the sequence [S1, S1', S1''] and the reasoning result [P1, P2], as well as the sequence [S4, S4', S4'', S4'''] and the reasoning result [E1, E2], and releases the computing resources of the reasoning sequences [S1, S1', S1''] and [S4, S4', S4'', S4''']; in addition, after the clearing, the control module also selects the sequence [S6, S6'] from the remaining sequence of the text to be processed and stores it in the storage space released after clearing the sequence [S1, S1', S1''], and selects the sequence [S7, S7', S7''] from the remaining sequence of the text to be processed and stores it in the storage space released after clearing the sequence [S4, S4', S4'', S4'''].

[0056] Continue to refer to Figure 4c , when the third inference iteration is completed, the inference results corresponding to each sequence are as follows: The sequence [S6, S6'] is stored in the memory module, and the inference iteration has not yet started; The results corresponding to the sequence [S2, S2'] are Q1, Q2, Q3, and the reasoning has been completed, that is, the final reasoning result of the sequence [S2, S2'] is [Q1, Q2, Q3]; The result corresponding to the sequence [S5, S5', S5''] is Y1; The sequence [S7, S7', S7''] is stored in the memory module, and the inference iteration has not yet started.

[0057] At this point, the four sequences [S1, S1', S1''], [S2, S2'], [S3, S3', S3''], S4, S4', S4'', S4'''] in the batch data have all completed reasoning iterations, and new sequences [S5, S5', S5''], [S6, S6'], [S7, S7', S7''] have been stored, and the sequence [S5, S5', S5''] has also completed one reasoning iteration. When starting the next reasoning iteration, the control module will also clear the sequence [S2, S2'] and the reasoning result [Q1, Q2, Q3], and release the reasoning iteration sequence [S2, S2'] and the computing resources, and select a new sequence from the remaining sequences of the text to be processed and store it in the position released after clearing [S2, S2'], until all sequences in the text to be processed are processed.

[0058] In addition, in the embodiment of the present application, when a sequence that has completed inference iterations is cleared and a new sequence is stored, the length of the new sequence needs to be no greater than the length of the cleared sequence.

[0059] In an embodiment of the present application, a large language model is deployed in a control module based on assembly language. When reasoning is performed, the usage of the memory module and the usage of the computing resources of the computing module are monitored in real time. When any sequence completes reasoning, the storage space and computing resources corresponding to the sequence can be released, and a new sequence can be stored for reasoning, thereby greatly improving the utilization rate of the memory module and the computing module, and improving the reasoning efficiency of the large language model.

[0060] In the embodiment of the present application, the control module is further configured to: when the length of any of the sequences exceeds a preset length, divide the sequence into a plurality of blocks, and store each divided block into the memory module; Each block contains a plurality of text information of the text to be processed, the length of each block does not exceed the preset length, and the end of the previous block and the header of the next block of two adjacent blocks have overlapping text information.

[0061] Among them, the sequence length that the large language model can process is limited, and the preset length is determined based on the processing capacity of the large language model. When the sequence length in the text to be processed exceeds the preset length, the control module divides the sequence into multiple blocks when storing each sequence. For example, Figure 4a As shown, the sequence [S1, S1', S1''] includes three blocks S1, S1', S1'', and the lengths of the three blocks do not exceed the preset length, thereby facilitating the processing of large language models.

[0062] In addition, when dividing, the sequence can be divided according to a fixed length (less than a preset length). If the last block of the sequence is less than a fixed length, it can be filled with preset characters or the part less than a fixed length can be directly used as a block.

[0063] In addition, in the embodiment of the present application, when dividing, the end of the previous block and the head of the next block in two adjacent blocks have overlapping text information. For example, in the sequence [S1, S1', S1''], the tail of block S1 and the head of S1', as well as the tail of S1' and the head of S1'' have overlapping parts, thereby ensuring the continuity of information between each block. In addition, in the embodiment of the present application, in order to maintain the consistency of the context, during the reasoning iteration process, the hidden state of the previous block can also be used to initialize the current block to improve the consistency of the reasoning result.

[0064] In an embodiment of the present application, when the large language model performs reasoning iterations based on each block, multiple reasoning results will be generated. After all the multiple blocks are reasoned, they can be integrated based on the multiple reasoning results to obtain the final reasoning result corresponding to the sequence.

[0065] Among them, based on the nature of the task, different types of reasoning results will be obtained when reasoning based on each block. For example, in a dialogue task, the reasoning result obtained based on each block is a text; in a classification task, the reasoning result obtained based on each block is a classification result. Then, when integrating the reasoning results corresponding to each block, different integration strategies can be adopted based on the task type.

[0066] For example, in classification tasks, voting or weighted averaging can be used to integrate the classification results of each block; in dialogue tasks, the end of the reasoning result of the previous block can be used as the starting point of the reasoning result of the next block, thereby ensuring the fluency of the integrated text.

[0067] In the embodiment of the present application, the control module is further configured as follows: Assign a prefix to any sequence when calling it; Determine whether the preset database has an inference result corresponding to the prefix; If yes, the inference result corresponding to the prefix is ​​called from the preset database, and the inference process of the sequence is skipped; If not, the sequence is inferred and the inference result is stored in the preset database.

[0068] Among them, when any sequence is input into the large language model, the large language model will assign a prefix to each sequence, and the prefixes assigned to the same or similar sequences are also the same or similar. In order to improve the reasoning efficiency of the large language model, in an embodiment of the present application, when reasoning for each sequence, the reasoning result will be stored in a preset database, and a mapping relationship will be established between the reasoning result and the corresponding prefix. Thus, in the subsequent reasoning process, after assigning a prefix to each input sequence, first search from the preset database to see if there is a mapping relationship corresponding to the prefix. If so, the corresponding reasoning result in the mapping relationship is directly used as the reasoning result of the sequence, and the reasoning iteration of the sequence is skipped, thereby improving the reasoning efficiency; if the preset data does not have a mapping relationship corresponding to the prefix, the sequence is normally iterated, and after the reasoning iteration is completed, the reasoning result corresponding to the sequence is mapped with the prefix of the sequence and stored in the preset database for subsequent use.

[0069] In an embodiment of the present application, a large language model is deployed in a control module based on assembly language. When reasoning is performed on the processed text, after any sequence completes reasoning, the storage space and computing resources corresponding to the sequence can be released, and a new sequence can be stored for reasoning, thereby greatly improving the utilization rate of the memory module and the computing module, and improving the reasoning efficiency of the large language model.

[0070] Exemplary Methods like Figure 5As shown, this exemplary embodiment proposes a reasoning method, which is applied to the reasoning device described in any embodiment of the above exemplary device. In the embodiment of the present application, the reasoning method includes the following steps S100-S500: Step S100: in response to receiving a processing instruction, obtaining a target number of sequences from the text to be processed to form batch data; Step S200: storing the batch data into the memory module; Step S300: after any sequence in the batch data is processed, the storage space and computing resources occupied by the processed sequence and the corresponding inference result are released; Step S400: another sequence is selected from the remaining sequences of the text to be processed, and stored in the storage space released by the sequence that has been inferred.

[0071] In the embodiment of the present application, the reasoning method further includes: Determining a first maximum target number based on the current available memory of the memory module and the maximum length of each sequence in the text to be processed; Determine a second maximum target quantity based on the current available computing resource size of the computing module and the unit sequence computing resource consumption of each sequence in the to-be-processed text; The smaller value between the first maximum target quantity and the second maximum target quantity is used as the target quantity.

[0072] In the embodiment of the present application, the first maximum target quantity is determined by the following method:

[0073] in, represents the first maximum target number, represents the total storage space of the memory module, represents the used storage space of the memory module, S1 represents the first safety factor, represents the maximum length of each sequence in the text to be processed, Represents the memory usage per unit length.

[0074] In the embodiment of the present application, the second maximum target number is determined based on the following method:

[0075] in, represents the second largest number of targets, represents the total computing resource size of the computing module, represents the size of computing resources used by the computing module, S2 represents the second safety factor, Represents the computational resource consumption of a unit sequence.

[0076] In the embodiment of the present application, the reasoning method further includes: When the length of any of the sequences exceeds a preset length, the sequence is divided into a plurality of blocks, and each of the divided blocks is stored in the memory module; Each block contains a plurality of text information of the text to be processed, the length of each block does not exceed the preset length, and the end of the previous block and the header of the next block of two adjacent blocks have overlapping text information.

[0077] In the embodiment of the present application, the reasoning method further includes: Assign a prefix to any sequence when calling it; Determine whether the preset database has a processing result corresponding to the prefix; If yes, the processing result corresponding to the prefix is ​​called from the preset database, and the reasoning process of the sequence is skipped; If not, the sequence is inferred and the processing result is stored in the preset database.

[0078] The reasoning method proposed in the embodiment of the present application is applied to the reasoning device described in any of the above embodiments, and therefore, it has at least all the beneficial effects of the above reasoning device, which will not be elaborated here one by one.

[0079] Exemplary Media After introducing the method, medium and system of the exemplary embodiments of the present application, next, reference is made to Figure 6 For a description of the computer-readable storage medium of the exemplary embodiment of the present application, please refer to Figure 6, the computer-readable storage medium shown is a CD 70, on which a computer program (i.e., a program product) is stored. When the computer program is executed by the processor, each step recorded in the above method implementation will be implemented, for example, in response to receiving a processing instruction, a target number of sequences are obtained from the text to be processed to form a batch of data; the batch of data is stored in the memory module; after any sequence in the batch of data is processed, the storage space and computing resources occupied by the processed sequence and the corresponding inference result are released; another sequence is selected from the remaining sequences of the text to be processed and stored in the storage space released by the sequence that has been inferred. The specific implementation of each step will not be repeated here. It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which will not be repeated here.

[0080] Exemplary Chips After introducing the method, apparatus, and medium of the exemplary embodiment of the present application, next, the chip of the exemplary embodiment of the present application is introduced.

[0081] In an embodiment of the present application, a chip is provided, including the reasoning device described in any of the above embodiments. Since the chip includes the reasoning device in any of the above embodiments, all the beneficial effects of the reasoning device described in the above embodiments are not described in detail here.

[0082] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

[0083] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

Claims

1. An inference device for inference operation of a large language model, wherein the large language model is deployed on the inference device, and the inference device comprises: Memory module, computing module and control module; The control module is configured to: in response to receiving a processing instruction, obtain a target number of sequences from the text to be processed to form batch data; Storing the batch data in the memory module for the calculation module to call; After any sequence in the batch data is processed, the storage space and computing resources occupied by the processed sequence and the corresponding inference result are released; Another sequence is selected from the remaining sequences of the text to be processed and stored in the storage space released by the sequence that has been inferred.

2. The inference device of claim 1, wherein the control module is further configured to determine the target quantity based on: Determining a first maximum target number based on the current available memory of the memory module and the maximum length of each sequence in the text to be processed; Determine a second maximum target quantity based on the current available computing resource size of the computing module and the unit sequence computing resource consumption of each sequence in the to-be-processed text; The smaller value between the first maximum target quantity and the second maximum target quantity is used as the target quantity.

3. The inference device of claim 2, wherein the control module is further configured to determine the first maximum target quantity based on: in, represents the first maximum target number, represents the total storage space of the memory module, represents the used storage space of the memory module, S1 represents the first safety factor, represents the maximum length of each sequence in the text to be processed, Represents the memory footprint of a sequence of unit length.

4. The inference device of claim 2, wherein the control module is further configured to determine the second maximum target quantity based on: in, represents the second largest number of targets, represents the total computing resource size of the computing module, represents the size of computing resources used by the computing module, S2 represents the second safety factor, Represents the computational resource consumption of a unit sequence.

5. The inference device according to claim 1, wherein the control module is further configured to: when the length of any of the sequences exceeds a preset length, divide the sequence into a plurality of blocks, and store each of the divided blocks into the memory module; in, Each block contains a plurality of text information of the text to be processed, the length of each block does not exceed the preset length, and the end of the previous block and the header of the next block of two adjacent blocks have overlapping text information.

6. The inference device according to claim 1, wherein the control module is further configured to: Assign a prefix to any sequence when calling it; Determine whether the preset database has an inference result corresponding to the prefix; If yes, the inference result corresponding to the prefix is ​​called from the preset database, and the inference process of the sequence is skipped; If not, the sequence is inferred and the inference result is stored in the preset database.

7. A reasoning method, applied to the reasoning device according to any one of claims 1 to 6, the reasoning method comprising: In response to receiving the processing instruction, obtaining a target number of sequences from the text to be processed to form batch data; storing the batch data into the memory module; After any sequence in the batch data is processed, the storage space and computing resources occupied by the processed sequence and the corresponding inference result are released; Another sequence is selected from the remaining sequences of the text to be processed and stored in the storage space released by the sequence that has been inferred.

8. The reasoning method according to claim 7, further comprising: Determining a first maximum target number based on the current available memory of the memory module and the maximum length of each sequence in the text to be processed; Determine a second maximum target quantity based on the current available computing resource size of the computing module and the unit sequence computing resource consumption of each sequence in the to-be-processed text; The smaller value between the first maximum target quantity and the second maximum target quantity is used as the target quantity.

9. The reasoning method according to claim 8, wherein the first maximum number of targets is determined by: in, represents the first maximum target number, represents the total storage space of the memory module, represents the used storage space of the memory module, S1 represents the first safety factor, represents the maximum length of each sequence in the text to be processed, Represents the memory usage per unit length.

10. The inference method according to claim 8, wherein the second maximum number of targets is determined based on the following method: in, represents the second largest number of targets, represents the total computing resource size of the computing module, represents the size of computing resources used by the computing module, S2 represents the second safety factor, Represents the computational resource consumption of a unit sequence.

11. The reasoning method according to claim 7, further comprising: When the length of any of the sequences exceeds a preset length, the sequence is divided into a plurality of blocks, and each of the divided blocks is stored in the memory module; Each block contains a plurality of text information of the text to be processed, the length of each block does not exceed the preset length, and the end of the previous block and the header of the next block of two adjacent blocks have overlapping text information.

12. The reasoning method according to claim 7, further comprising: Assign a prefix to any sequence when calling it; Determine whether the preset database has a processing result corresponding to the prefix; If yes, the processing result corresponding to the prefix is ​​called from the preset database, and the reasoning process of the sequence is skipped; If not, the sequence is inferred and the processing result is stored in the preset database.

13. A readable storage medium comprising instructions, which, when executed on a computing device, causes the computing device to execute the method according to any one of claims 7 to 12.

14. A chip comprising the inference device according to any one of claims 1 to 6.

Images