A data processing method based on an improved large language model reasoning operation framework

Abstract

The present invention discloses a data processing method based on an improved large language model reasoning operation framework. By acquiring the data to be processed, and loading and processing the target large language model corresponding to the improved large language model reasoning operation framework; the improved large language model reasoning operation framework performs data processing including a pre-filling stage and multiple generation stages; in the pre-filling stage, the data to be processed is pre-filled with data processing operations to obtain pre-filling stage processing completion data; in the generation stage, all activation tensor data and key-value cache data in the pre-filling stage processing completion data are respectively processed in the generation stage to obtain generation stage processing completion data. The problem of the decline in overall reasoning performance and uneven load between memories due to memory affinity and data shape being unfriendly to hardware is solved, and the load balancing of the memory under the large language model is achieved, which improves the reasoning performance of the model.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method based on an improved large language model reasoning operation framework. Background Art

[0002] In current language-based large model reasoning, traditional methods usually use open source computing frameworks to perform reasoning by connecting independent small operators in series to form a complete computing graph.

[0003] In the process of implementing the present invention, the inventors found that the prior art has the following defects: At present, the traditional method puts great pressure on device storage. Each independent operator needs to apply for storage space for its input and output data separately, resulting in additional memory fragmentation. At the same time, when there are multiple memories on the same device, there may be uneven loads between memories. In addition, the efficiency of each independent operator, including its efficiency during data transfer, cannot be guaranteed due to memory affinity and data shape that is not hardware-friendly, which leads to a decrease in overall reasoning performance. Summary of the invention

[0004] The present invention provides a data processing method based on an improved large language model reasoning operation framework to achieve load balancing of a memory under a large language model and improve the reasoning performance of the model.

[0005] According to one aspect of the present invention, a data processing method based on an improved large language model reasoning operation framework is provided, which includes:

[0006] Obtain the data to be processed, and load the target large language model corresponding to the improved large language model reasoning running framework;

[0007] The improved large language model inference operation framework includes a large language model loading stage, a pre-filling stage and a generation stage; data processing based on the improved large language model inference operation framework includes data processing in a pre-filling stage once and data processing in multiple generation stages; the processed data in the generation stage includes activation tensor data and key-value cache data;

[0008] In the pre-filling stage, a pre-filling data processing operation is performed on the data to be processed to obtain pre-filling stage processing completed data;

[0009] In the generation stage, all activation tensor data and key-value cache data in the pre-filling stage processing completion data are respectively processed in the generation stage to obtain the generation stage processing completion data.

[0010] According to another aspect of the present invention, a data processing device based on an improved large language model reasoning operation framework is provided, comprising:

[0011] A data acquisition module to be processed, used to acquire the data to be processed and load the target large language model corresponding to the improved large language model reasoning operation framework;

[0012] The improved large language model inference operation framework includes a large language model loading stage, a pre-filling stage and a generation stage; data processing based on the improved large language model inference operation framework includes data processing in a pre-filling stage once and data processing in multiple generation stages; the processed data in the generation stage includes activation tensor data and key-value cache data;

[0013] A pre-filling stage processing completed data determination module is used to perform a pre-filling data processing operation on the data to be processed in the pre-filling stage to obtain pre-filling stage processing completed data;

[0014] The generation phase processing completion data determination module is used to perform generation phase processing on all activation tensor data and key-value cache data in the pre-filling phase processing completion data in the generation phase to obtain generation phase processing completion data.

[0015] According to another aspect of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a data processing method based on an improved large language model reasoning operation framework as described in any embodiment of the present invention is implemented.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a data processing method based on an improved large language model reasoning operation framework as described in any embodiment of the present invention when executed.

[0017] The technical solution of the embodiment of the present invention obtains the data to be processed and loads the target large language model corresponding to the improved large language model reasoning operation framework; wherein the improved large language model reasoning operation framework includes a large language model loading stage, a pre-filling stage and a generation stage; data processing based on the improved large language model reasoning operation framework includes data processing in a pre-filling stage and data processing in multiple generation stages; the processing data in the generation stage includes activation tensor data and key-value cache data; in the pre-filling stage, the data to be processed is pre-filled with data processing operations to obtain pre-filling stage processing completion data; in the generation stage, all activation tensor data and key-value cache data in the pre-filling stage processing completion data are respectively processed in the generation stage to obtain generation stage processing completion data. The problem of the overall reasoning performance decline and uneven load between memories due to the memory affinity and data shape being unfriendly to the hardware is solved, and the load balancing of the memory under the large language model is realized, the reasoning performance of the model is improved, and the text throughput is higher and faster.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1a is a flowchart of a data processing method based on an improved large language model reasoning operation framework provided according to Embodiment 1 of the present invention;

[0021] Figure 1b This is a structural schematic diagram of performing trimming optimization on the last dimension in the word vector dimension in the pre-filling stage in the method provided in the first embodiment of the present invention;

[0022] Figure 2 is a flowchart of another data processing method based on an improved large language model reasoning operation framework provided according to Embodiment 2 of the present invention;

[0023] Figure 3 is a structural schematic diagram of a data processing device based on an improved large language model reasoning operation framework provided according to Embodiment 3 of the present invention;

[0024] Figure 4 It is a schematic diagram of the structure of an electronic device provided according to Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "target", "current", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Embodiment 1

[0028] Figure 1a A flowchart of a data processing method based on an improved large language model reasoning operation framework is provided for Embodiment 1 of the present invention. This embodiment is applicable to the case where data processing is performed on the improved large language model reasoning operation framework. The method can be executed by a data processing device based on the improved large language model reasoning operation framework. The data processing device based on the improved large language model reasoning operation framework can be implemented in the form of hardware and / or software.

[0029] Correspondingly, such as Figure 1a As shown, the method includes:

[0030] S110: Obtain data to be processed, and load and process the target large language model corresponding to the improved large language model reasoning running framework.

[0031] Among them, the improved large language model inference operation framework includes a large language model loading stage, a pre-filling stage and a generation stage; data processing based on the improved large language model inference operation framework includes data processing in a pre-filling stage once and data processing in multiple generation stages; the processed data in the generation stage includes activation tensor data and key-value cache data.

[0032] In this embodiment, after the data to be processed is obtained, the target large language model needs to be loaded first, and the processing operations of the pre-filling stage and the generating stage are performed through the loaded target large language model.

[0033] The improved large language model inference operation framework performs data processing, including a pre-filling stage of data processing. The pre-filling stage also includes activation tensor data and key-value cache data, but because the pre-filling stage of data processing is only performed once, the number of times the storage space is used is relatively small.

[0034] For data processing in the improved large language model inference operation framework, multiple generation phases of data processing are included, so the model spends most of its time performing generation phase calculations. For data processing in multiple generation phases, multiple rounds of inference calculations need to be performed on activation tensor data and key-value cache data, respectively. Among them, activation tensor data and key-value cache data are both tensor data.

[0035] S120. In the pre-filling stage, a pre-filling data processing operation is performed on the data to be processed to obtain pre-filling stage processing completed data.

[0036] Optionally, in the pre-filling stage, the data to be processed is subjected to a pre-filling data processing operation to obtain data processed in the pre-filling stage, including: trimming and optimizing the data to be processed in the pre-filling stage in the improved large language model inference running framework on the last dimension in the word vector dimension, and performing calculations on the word vector dimension through a multi-layer perceptron and a fully connected layer to obtain data processed in the pre-filling stage.

[0037] At present, the industry standard open source library often samples and outputs the last word vector dimension after the calculation of the fully connected layer is completed. The amount of calculation of the fully connected layer is positively correlated with the number of word vectors calculated. Therefore, there is a certain amount of redundant calculation here. Because the real effective information only exists in the last word vector dimension. However, the previous word vector dimensions also participate in the calculation of the fully connected layer, which increases the amount of calculation.

[0038] like Figure 1bThe figure shows the structure diagram of the pruning optimization on the last dimension of the word vector dimension in the pre-filling stage. In the pre-filling stage corresponding to the large language model, the final sampling stage is mainly a fully connected layer calculation, which calculates the probability of each word unit appearing in the word vector dimension through the fully connected layer. There is a multi-layer perceptron calculation before the fully connected layer, and a sampling calculation after the fully connected layer to select the real word unit.

[0039] In this embodiment, the word vector dimension is cut in advance, that is, the last dimension of the word vector dimension (that is, Figure 1b This not only reduces the amount of computation of the fully connected layer, but also reduces the amount of computation of the multi-layer perceptron in front of it, and makes the computation of the two constants, which no longer increase with the increase of the number of tokens.

[0040] Furthermore, the data after trimming and optimization is calculated through multi-layer perceptron and fully connected layer in the word vector dimension to obtain the data processed in the pre-filling stage.

[0041] S130. In the generation phase, all activation tensor data and key-value cache data in the pre-filling phase processing completion data are respectively processed in the generation phase to obtain generation phase processing completion data.

[0042] Optionally, it also includes: for the activation tensor data in the generation stage, when the generation stage processing calculations are performed, the storage space remains constant, and the storage space required for the next round of generation stage processing calculations will reuse the storage space of the previous round; for the key-value cache data in the generation stage, when the generation stage processing calculations are performed, the maximum storage space matching the key-value cache data is obtained, and the generation stage processing calculations of the key-value cache data are performed according to the maximum storage space.

[0043] In this embodiment, for activation tensor data, activation tensor data refers to input and output data generated by the input text features corresponding to the data to be processed. Since the storage space required for activation tensor data is constant each time it is calculated during the generation phase, the storage space will only be allocated once, and the storage space required for the next round of generation phase processing calculations will reuse the storage space of the previous round of activation tensor data to achieve full reuse throughout the entire reasoning cycle, greatly reducing memory allocation and release.

[0044] In addition, for key-value cache data, key-value cache data is mainly used to store historical information generated during each generation phase calculation. The storage requirement of this data block will continue to grow as the output text becomes longer, and each round of output will require more storage than the previous round.

[0045] However, if you actually allocate storage every time you need more storage, it will cause memory fragmentation. Therefore, you need to pre-calculate the maximum storage space required after N rounds of output, and apply only once according to the maximum storage space. The data storage space required for each round of calculation comes from this maximum storage space. The next round of storage can overwrite the previous round of storage, which can avoid repeated storage and release of storage space and avoid memory fragmentation.

[0046] In this embodiment, generation phase processing is performed on all activation tensor data and key-value cache data in the pre-filling phase processing completion data to obtain generation phase processing completion data, and feedback processing operations are performed on the obtained generation phase processing completion data.

[0047] The technical solution of the embodiment of the present invention obtains the data to be processed and loads the target large language model corresponding to the improved large language model reasoning operation framework; wherein the improved large language model reasoning operation framework includes a large language model loading stage, a pre-filling stage and a generation stage; data processing based on the improved large language model reasoning operation framework includes data processing in a pre-filling stage and data processing in multiple generation stages; the processing data in the generation stage includes activation tensor data and key-value cache data; in the pre-filling stage, the data to be processed is pre-filled with data processing operations to obtain pre-filling stage processing completion data; in the generation stage, all activation tensor data and key-value cache data in the pre-filling stage processing completion data are respectively processed in the generation stage to obtain generation stage processing completion data. The problem of the overall reasoning performance decline and uneven load between memories due to the memory affinity and data shape being unfriendly to the hardware is solved, and the load balancing of the memory under the large language model is realized, the reasoning performance of the model is improved, and the text throughput is higher and faster.

[0048] Embodiment 2

[0049] Figure 2 A flowchart of another data processing method based on an improved large language model reasoning operation framework is provided for Embodiment 2 of the present invention. This embodiment is optimized based on the above embodiments, and before obtaining the data to be processed and loading and processing the target large language model corresponding to the improved large language model reasoning operation framework, it also includes conversion of the transport type of tensor data.

[0050] Correspondingly, such as Figure 2 As shown, the method includes:

[0051] S210. Before tensor data processing and calculation, in the process of transferring the tensor data from the third-level cache to the second-level cache, obtaining matrix weight data.

[0052] Among them, the multi-level storage of the memory includes a first-level cache, a second-level cache and a third-level cache; the data copy speeds corresponding to the second-level cache and the third-level cache are both lower than the first-level cache.

[0053] In this embodiment, since the data copy speed of the second-level cache and the third-level cache is relatively low, it is necessary to rearrange the data and convert the original data transfer type from slice copy transfer to linear copy transfer. In addition, for the acquired matrix weight data, the matrix weight data is generally constant tensor data.

[0054] Exemplarily, since the multi-level storage of the memory includes a first-level cache, a second-level cache, and a third-level cache, the first-level cache can be represented as L1 storage; the second-level cache can be represented as L2 storage; and the third-level cache can be represented as L3 storage. Specifically, the L2 data copy speed is lower than the L1 data copy speed; and the L3 data copy speed is lower than both the L2 data copy speed and the L1 data copy speed.

[0055] In addition, L3 storage capacity is large, but the read and write speed is slow; L2 storage capacity is small, but the read and write speed is fast. Before data calculation, it is often necessary to move data from L3 storage to L2 storage. However, due to the small capacity of L2, the complete data is often not moved to L2 at one time, but in multiple times, each time a part of the data is moved.

[0056] Furthermore, according to the different behaviors of transporting small slices, the transport types are divided into linear copy transport and slice copy transport. When performing data calculations, slice copying is often required due to the needs of computing characteristics. However, in terms of transport efficiency, since slice copy transport requires jumping to fetch data, while linear copy transport is a one-time complete data fetch, the performance and efficiency of slice copy transport are much lower than those of linear copy transport.

[0057] Since the slice copying and moving of matrix weight data will frequently appear in the entire inference calculation process, it is necessary to perform linear copying and moving processing operations on the matrix weight data through a pre-set matrix multiplication weight data rearrangement method.

[0058] S220. Obtain a target operator slicing method for the matrix weight data through a preset matrix multiplication weight data rearrangement method.

[0059] The target operator slicing method includes copying the row size of the slices each time and copying the column size of the slices each time.

[0060] For example, for matrix weight data, it can be expressed as . Get the row size and column size of each copy block in the target operator slicing method. Assume that the row size of each copy block is rows; the size of each copy block is List.

[0061] S230, processing the matrix weight data according to the row size of each copy and the column size of each copy, and calculating the number of row dimension copies and the number of column dimension copies respectively.

[0062] Continuing from the previous example, by the formula , to calculate the number of row dimension copies; through the formula , to calculate the number of column dimension copies.

[0063] S240, generating a four-dimensional data matrix weight according to the number of row dimension copies, the row size of each copy, the number of column dimension copies, and the column size of each copy.

[0064] In this embodiment, the matrix weight data can be expanded by the number of times the row dimension is copied, the row size of each copy, the number of times the column dimension is copied, and the column size of each copy to obtain the four-dimensional data matrix weight.

[0065] Continuing from the previous example, copy the number of times based on the row dimension , Each copy block row size , Column dimension copy times , and the size of each copy block column , to generate a four-dimensional data matrix with weights .

[0066] S250, performing shape transposition processing on the four-dimensional data matrix weights to obtain the four-dimensional data matrix weights after the shape transposition processing, and converting the transport type of the processed four-dimensional data matrix weights within the multi-level storage of the memory into linear copy transport.

[0067] Furthermore, the weights of the four-dimensional data matrix need to be Perform shape transposition processing to obtain the weight of the four-dimensional data matrix after shape transposition processing, which can be .for In the lowest two dimensions and It is already continuous, so the original slice copy and transfer of matrix weight data is converted to linear copy and transfer.

[0068] S260: Obtain data to be processed, and load the target large language model corresponding to the improved large language model reasoning running framework.

[0069] S270. In the pre-filling stage, a pre-filling data processing operation is performed on the data to be processed to obtain pre-filling stage processing completed data.

[0070] S280. In the generation phase, all activation tensor data and key-value cache data in the pre-filling phase processing completion data are respectively subjected to generation phase processing to obtain generation phase processing completion data.

[0071] Optionally, after performing shape transposition processing on the four-dimensional data matrix weights to obtain the four-dimensional data matrix weights after shape transposition processing, it also includes: for the matrix weight data, according to the size of the block column copied each time, the four-dimensional data matrix weights after shape transposition processing are uniformly divided into blocks, and the matrix multiplication weight data after uniform blocking are sequentially stored in different blocks corresponding to the target distributed memory.

[0072] In this embodiment, for the matrix weight data, according to the above-mentioned matrix multiplication weight data rearrangement method and shape transposition, the four-dimensional data matrix weight after the shape transposition processing is obtained, and it is necessary to distribute it to the target distributed memory. Therefore, it is necessary to firstly perform uniform block processing on the four-dimensional data matrix weight after the shape transposition processing according to the size of the block column of each copy, assuming that the four-dimensional data matrix weight after the shape transposition processing can be divided into 4 blocks. Assume that the target distributed memory includes 4 blocks, namely bank0, bank1, bank2 and bank3. Therefore. The matrix multiplication weight data after uniform block division can be stored in sequence in bank0, bank1, bank2 and bank3 corresponding to the target distributed memory.

[0073] The advantage of this setting is that it can ensure the load balancing of each storage and ensure that each slice is complete in the storage without the need for additional splicing operations.

[0074] Optionally, after performing shape transposition processing on the four-dimensional data matrix weights to obtain the four-dimensional data matrix weights after shape transposition processing, it also includes: dividing the key-value cache data blocks into odd-layer key-value cache data and even-layer key-value cache data according to the key-value cache row data; for the key-value cache data, performing data storage allocation processing through a pre-set key-value cache data distributed storage method according to the odd-layer key-value cache data and the even-layer key-value cache data, and generating a key-value cache data storage allocation result.

[0075] In this embodiment, the key-value cache data includes key-value cache row data and key-value cache column data; each layer in the target large language model includes a key-value cache data block; the target distributed memory includes multiple thread blocks, and each thread block includes multiple sub-memories.

[0076] For example, key-value cache data can be represented as ,in, Cache row data for key values; Cache column data for key values.

[0077] Specifically, for key-value cache data, Often needs to remain intact and cannot be split, and within the same thread block, also needs to be maintained Dimensionality is complete. So, The dimensions are evenly divided into various memories within the same thread block, and key-value cache data of different levels are evenly distributed to different thread blocks.

[0078] Specifically, the data storage allocation processing is performed according to the odd-numbered key-value cache data and the even-numbered key-value cache data through a pre-set key-value cache data distributed storage method to generate a key-value cache data storage allocation result, including: according to the key-value cache data distributed storage method, the odd-numbered key-value cache data blocks are allocated to the first thread block in the target distributed memory, and according to the key-value cache column data, the odd-numbered key-value cache data blocks are evenly allocated to different sub-memories corresponding to the first thread block; the even-numbered key-value cache data blocks are allocated to the second thread block in the target distributed memory, and according to the key-value cache column data, the even-numbered key-value cache data blocks are evenly allocated to different sub-memories corresponding to the second thread block, and the key-value cache data storage allocation result is generated.

[0079] In this embodiment, it is first necessary to divide the key-value cache data blocks into odd-numbered key-value cache data and even-numbered key-value cache data according to the key-value cache row data. Assume that each layer in the target large language model includes a key-value cache data block, and assume that it can include 3 layers, that is, 3 key-value cache data blocks (key-value cache data block 1, key-value cache data block 2, and key-value cache data block 3). Assume that the target distributed memory includes a first thread block and a second thread block; wherein the first thread block includes sub-memory 1 and sub-memory 2; the second thread block includes sub-memory 3 and sub-memory 4.

[0080] Specifically, the key-value cache data block 1 and the key-value cache data block 3 may be divided into odd-numbered layers of key-value cache data; and the key-value cache data block 2 may be divided into even-numbered layers of key-value cache data.

[0081] Further, according to the key-value cache column data, the key-value cache data block 1 and the key-value cache data block 3 can be divided into two blocks, namely, the key-value cache data block 11 and the key-value cache data block 12; the key-value cache data block 31 and the key-value cache data block 32. The key-value cache data block 1 can be allocated to the first thread block, and the key-value cache data block 11 corresponding to the key-value cache data block 1 can be allocated to the sub-memory 1; and the key-value cache data block 12 corresponding to the key-value cache data block 1 can be allocated to the sub-memory 2.

[0082] Similarly, the key-value cache data block 3 can be allocated to the first thread block, the key-value cache data block 31 corresponding to the key-value cache data block 3 can be allocated to the sub-memory 1, and the key-value cache data block 32 corresponding to the key-value cache data block 3 can be allocated to the sub-memory 2. The same allocation method is used for the key-value cache data block 2, which will not be repeated here.

[0083] The technical solution of the embodiment of the present invention is to obtain matrix weight data in the process of transferring the tensor data from the third-level cache to the second-level cache before tensor data processing and calculation; obtain the target operator slicing method for the matrix weight data through a preset matrix multiplication weight data rearrangement method; process the matrix weight data according to the row size of each copy and the column size of each copy, and calculate the number of row dimension copies and the number of column dimension copies respectively; generate four-dimensional data matrix weights according to the number of row dimension copies, the row size of each copy and the column size of each copy and the column size of each copy and the column size; perform shape transposition processing on the four-dimensional data matrix weight to obtain the four-dimensional data matrix weight after the shape transposition processing is completed, and transfer the processed four-dimensional data matrix weight within the multi-level storage of the memory. The type is converted to linear copy transfer; the data to be processed is obtained, and the target large language model corresponding to the improved large language model reasoning operation framework is loaded and processed; wherein the improved large language model reasoning operation framework includes a large language model loading stage, a pre-filling stage and a generation stage; data processing based on the improved large language model reasoning operation framework includes data processing in a pre-filling stage and data processing in multiple generation stages; the processing data in the generation stage includes activation tensor data and key-value cache data; in the pre-filling stage, the data to be processed is pre-filled with data processing operations to obtain pre-filling stage processing completion data; in the generation stage, all activation tensor data and key-value cache data in the pre-filling stage processing completion data are respectively processed in the generation stage to obtain generation stage processing completion data. The problem of the decline in overall reasoning performance and uneven load between memories due to memory affinity and data shape being unfriendly to hardware is solved, and the load balancing of memories under the large language model is realized, the reasoning performance of the model is improved, and the text throughput is higher and faster.

[0084] Embodiment 3

[0085] Figure 3 A structural diagram of a data processing device based on an improved large language model reasoning operation framework provided in the third embodiment of the present invention. The data processing device based on an improved large language model reasoning operation framework provided in this embodiment can be implemented by software and / or hardware, and can be configured in a terminal device or a server to implement a data processing method based on an improved large language model reasoning operation framework in the embodiment of the present invention. Figure 3 As shown, the device includes: a data acquisition module 310 for processing, a pre-filling phase processing completion data determination module 320, and a generation phase processing completion data determination module 330.

[0086] The data acquisition module 310 to be processed is used to acquire the data to be processed and load the target large language model corresponding to the improved large language model reasoning operation framework;

[0087] The improved large language model inference operation framework includes a large language model loading stage, a pre-filling stage and a generation stage; data processing based on the improved large language model inference operation framework includes data processing in a pre-filling stage once and data processing in multiple generation stages; the processed data in the generation stage includes activation tensor data and key-value cache data;

[0088] The pre-filling stage processing completed data determination module 320 is used to perform a pre-filling data processing operation on the data to be processed in the pre-filling stage to obtain pre-filling stage processing completed data;

[0089] The generation phase processing completion data determination module 330 is used to perform generation phase processing on all activation tensor data and key-value cache data in the pre-filling phase processing completion data in the generation phase to obtain generation phase processing completion data.

[0090] The technical solution of the embodiment of the present invention obtains the data to be processed and loads the target large language model corresponding to the improved large language model reasoning operation framework; wherein the improved large language model reasoning operation framework includes a large language model loading stage, a pre-filling stage and a generation stage; data processing based on the improved large language model reasoning operation framework includes data processing in a pre-filling stage and data processing in multiple generation stages; the processing data in the generation stage includes activation tensor data and key-value cache data; in the pre-filling stage, the data to be processed is pre-filled with data processing operations to obtain pre-filling stage processing completion data; in the generation stage, all activation tensor data and key-value cache data in the pre-filling stage processing completion data are respectively processed in the generation stage to obtain generation stage processing completion data. The problem of the overall reasoning performance decline and uneven load between memories due to the memory affinity and data shape being unfriendly to the hardware is solved, and the load balancing of the memory under the large language model is realized, the reasoning performance of the model is improved, and the text throughput is higher and faster.

[0091] Based on the above embodiments, the pre-filling stage processing completed data determination module 320 can be specifically used to: trim and optimize the data to be processed in the pre-filling stage in the improved large language model inference running framework on the last dimension in the word vector dimension, and calculate the word vector dimension through a multi-layer perceptron and a fully connected layer to obtain the pre-filling stage processing completed data.

[0092] On the basis of the above embodiments, it can also be specifically used for: for the activation tensor data in the generation stage, when the generation stage processing calculation is performed, the storage space remains constant, and the storage space required for the next round of generation stage processing calculation will reuse the storage space of the previous round; for the key-value cache data in the generation stage, when the generation stage processing calculation is performed, the maximum storage space matching the key-value cache data is obtained, and the generation stage processing calculation of the key-value cache data is performed according to the maximum storage space.

[0093] On the basis of the above embodiments, it also includes a transport type conversion module, which can be specifically used to: before the data to be processed is obtained and the target large language model corresponding to the improved large language model reasoning running framework is loaded and processed, before the tensor data is processed and calculated, in the process of transporting the tensor data from the third-level cache to the second-level cache, the matrix weight data is obtained; wherein the multi-level storage of the memory includes a first-level cache, a second-level cache and a third-level cache; the data copy speeds corresponding to the second-level cache and the third-level cache are respectively less than the first-level cache; through a preset matrix multiplication weight data rearrangement method, a target operator slicing method is obtained for the matrix weight data; Among them, the target operator slicing method includes the row size of each copy and the column size of each copy; the matrix weight data is processed according to the row size of each copy and the column size of each copy, and the number of row dimension copies and the number of column dimension copies are calculated respectively; the four-dimensional data matrix weight is generated according to the number of row dimension copies, the row size of each copy, the number of column dimension copies, and the column size of each copy; the four-dimensional data matrix weight is transposed to obtain the four-dimensional data matrix weight after the shape transposition processing, and the transportation type of the processed four-dimensional data matrix weight in the multi-level storage of the memory is converted into linear copy transportation.

[0094] On the basis of the above embodiments, it can also be specifically used for: after the four-dimensional data matrix weights are subjected to shape transposition processing to obtain the four-dimensional data matrix weights after shape transposition processing, for the matrix weight data, according to the size of the block column of each copy, the four-dimensional data matrix weights after shape transposition processing are uniformly divided into blocks, and the matrix multiplication weight data after uniform blocking are sequentially stored in different blocks corresponding to the target distributed memory.

[0095] On the basis of the above embodiments, it can also be specifically used for: after the shape transposition processing is performed on the four-dimensional data matrix weight to obtain the four-dimensional data matrix weight after the shape transposition processing is completed, wherein the key-value cache data includes key-value cache row data and key-value cache column data; each layer in the target large language model includes a key-value cache data block; the target distributed memory includes multiple thread blocks, and each thread block includes multiple sub-memories; according to the key-value cache row data, the key-value cache data blocks are divided into odd-layer key-value cache data and even-layer key-value cache data; for the key-value cache data, according to the odd-layer key-value cache data and the even-layer key-value cache data, data storage allocation processing is performed through a pre-set key-value cache data distributed storage method to generate a key-value cache data storage allocation result.

[0096] On the basis of the above embodiments, the data storage allocation processing is performed according to the odd-layer key-value cache data and the even-layer key-value cache data through a pre-set key-value cache data distributed storage method to generate a key-value cache data storage allocation result. It can also be specifically used for: allocating odd-layer key-value cache data blocks in the first thread block in the target distributed memory according to the key-value cache data distributed storage method, and evenly distributing the odd-layer key-value cache data blocks to different sub-memories corresponding to the first thread block according to the key-value cache column data; allocating even-layer key-value cache data blocks in the second thread block in the target distributed memory, and evenly distributing the even-layer key-value cache data blocks to different sub-memories corresponding to the second thread block according to the key-value cache column data, and generating the key-value cache data storage allocation result.

[0097] A data processing device based on an improved large language model reasoning operation framework provided in an embodiment of the present invention can execute a data processing method based on an improved large language model reasoning operation framework provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0098] Embodiment 4

[0099] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement the fourth embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0100] like Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0101] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0102] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a data processing method based on an improved large language model reasoning operation framework.

[0103] In some embodiments, a data processing method based on an improved large language model reasoning operation framework may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data processing method based on the improved large language model reasoning operation framework described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute a data processing method based on the improved large language model reasoning operation framework in any other appropriate manner (for example, by means of firmware).

[0104] The method includes: obtaining data to be processed, and loading and processing a target large language model corresponding to an improved large language model reasoning operation framework; wherein the improved large language model reasoning operation framework includes a large language model loading stage, a pre-filling stage and a generation stage; data processing based on the improved large language model reasoning operation framework includes data processing in a pre-filling stage once and data processing in multiple generation stages; the processed data in the generation stage includes activation tensor data and key-value cache data; in the pre-filling stage, pre-filling data processing operations are performed on the data to be processed to obtain pre-filling stage processing completion data; in the generation stage, all activation tensor data and key-value cache data in the pre-filling stage processing completion data are respectively processed in the generation stage to obtain generation stage processing completion data.

[0105] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0107] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0108] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0109] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0110] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0111] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0112] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0113] Embodiment 5

[0114] Embodiment 5 of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions are used to execute a data processing method based on an improved large language model reasoning operation framework when executed by a computer processor, the method comprising: obtaining data to be processed, and loading and processing the target large language model corresponding to the improved large language model reasoning operation framework; wherein the improved large language model reasoning operation framework comprises a large language model loading stage, a pre-filling stage and a generation stage; data processing based on the improved large language model reasoning operation framework comprises data processing in a pre-filling stage once and data processing in multiple generation stages; the processed data in the generation stage comprises activation tensor data and key-value cache data; in the pre-filling stage, a pre-filling data processing operation is performed on the data to be processed to obtain pre-filling stage processing completion data; in the generation stage, all activation tensor data and key-value cache data in the pre-filling stage processing completion data are respectively subjected to generation stage processing to obtain generation stage processing completion data.

[0115] Of course, the computer-readable storage medium provided in an embodiment of the present invention, whose computer executable instructions are not limited to the method operations described above, can also execute related operations in data processing based on an improved large language model reasoning operation framework provided in any embodiment of the present invention.

[0116] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0117] It is worth noting that in the above-mentioned embodiment of data processing based on the improved large language model inference operation framework, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0118] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data processing method based on an improved large language model reasoning operation framework, characterized in that: include: Obtain the data to be processed, and load the target large language model corresponding to the improved large language model reasoning running framework; The improved large language model inference operation framework includes a large language model loading stage, a pre-filling stage and a generation stage; data processing based on the improved large language model inference operation framework includes data processing in a pre-filling stage once and data processing in multiple generation stages; the processed data in the generation stage includes activation tensor data and key-value cache data; In the pre-filling stage, a pre-filling data processing operation is performed on the data to be processed to obtain pre-filling stage processing completed data; In the generation phase, all activation tensor data and key-value cache data in the pre-filling phase processing completed data are respectively processed in the generation phase to obtain the generation phase processing completed data; Before obtaining the data to be processed and loading the target large language model corresponding to the improved large language model reasoning running framework, the method further includes: Before the tensor data is processed and calculated, in the process of transferring the tensor data from the third-level cache to the second-level cache, obtaining matrix weight data; The multi-level storage of the memory includes a first-level cache, a second-level cache, and a third-level cache; the data copy speeds corresponding to the second-level cache and the third-level cache are both lower than those of the first-level cache; Obtaining a target operator slicing method for the matrix weight data through a preset matrix multiplication weight data rearrangement method; Wherein, the target operator slicing method includes copying the row size of the slices each time and copying the column size of the slices each time; Process the matrix weight data according to the row size of each copy and the column size of each copy, and calculate the number of row dimension copies and the number of column dimension copies respectively; Generate the weight of the four-dimensional data matrix according to the number of row dimension copies, the row size of each copy, the number of column dimension copies, and the column size of each copy; Performing shape transposition processing on the four-dimensional data matrix weights to obtain the four-dimensional data matrix weights after the shape transposition processing, and converting the transport type of the processed four-dimensional data matrix weights within the multi-level storage of the memory into linear copy transport; After the four-dimensional data matrix weights are subjected to shape transposition processing to obtain the four-dimensional data matrix weights after shape transposition processing, the method further includes: The key-value cache data includes key-value cache row data and key-value cache column data; each layer in the target large language model includes a key-value cache data block; according to the key-value cache row data, the key-value cache data block is divided into odd-numbered layer key-value cache data and even-numbered layer key-value cache data; For the key-value cache data, data storage allocation processing is performed according to the odd-numbered key-value cache data and the even-numbered key-value cache data by using a preset key-value cache data distributed storage method to generate a key-value cache data storage allocation result.

2. The method according to claim 1, characterized in that: In the pre-filling stage, the pre-filling data processing operation is performed on the data to be processed to obtain pre-filling stage processing completed data, including: The data to be processed in the pre-filling stage in the improved large language model reasoning operation framework is trimmed and optimized on the last dimension in the word vector dimension, and is calculated on the word vector dimension through a multi-layer perceptron and a fully connected layer to obtain the pre-filling stage processed data.

3. The method according to claim 1, characterized in that Also includes: For the activation tensor data in the generation phase, the storage space remains constant during the generation phase processing calculation, and the storage space required for the next round of generation phase processing calculation will reuse the storage space of the previous round; For the key-value cache data in the generation phase, when performing generation phase processing calculations, a maximum storage space matching the key-value cache data is obtained, and generation phase processing calculations of the key-value cache data are performed based on the maximum storage space.

4. The method according to claim 1, characterized in that After the four-dimensional data matrix weights are subjected to shape transposition processing to obtain the four-dimensional data matrix weights after shape transposition processing, the method further includes: For the matrix weight data, the four-dimensional data matrix weight after the shape transposition processing is evenly divided into blocks according to the size of the block column copied each time, and the matrix multiplication weight data after the even block division is sequentially stored in different blocks corresponding to the target distributed memory, and the target distributed memory includes multiple thread blocks, and each thread block includes multiple sub-memories.

5. The method according to claim 1, characterized in that The step of performing data storage allocation processing according to the odd-numbered key-value cache data and the even-numbered key-value cache data by using a preset key-value cache data distributed storage method to generate a key-value cache data storage allocation result includes: According to the key-value cache data distributed storage method, the odd-numbered-layer key-value cache data blocks are allocated in a first thread block in a target distributed memory, and according to the key-value cache column data, the odd-numbered-layer key-value cache data blocks are evenly allocated to different sub-memories corresponding to the first thread block; The even-numbered layer key-value cache data blocks are allocated in the second thread block in the target distributed memory, and according to the key-value cache column data, the even-numbered layer key-value cache data blocks are evenly allocated to different sub-memories corresponding to the second thread block, and the key-value cache data storage allocation result is generated.

6. A data processing device based on an improved large language model reasoning operation framework, characterized in that: include: A data acquisition module to be processed, used to acquire the data to be processed and load the target large language model corresponding to the improved large language model reasoning operation framework; The improved large language model inference operation framework includes a large language model loading stage, a pre-filling stage and a generation stage; data processing based on the improved large language model inference operation framework includes data processing in a pre-filling stage once and data processing in multiple generation stages; the processed data in the generation stage includes activation tensor data and key-value cache data; A pre-filling stage processing completed data determination module is used to perform a pre-filling data processing operation on the data to be processed in the pre-filling stage to obtain pre-filling stage processing completed data; A generation phase processing completed data determination module is used to perform generation phase processing on all activation tensor data and key-value cache data in the pre-filling phase processing completed data in the generation phase to obtain generation phase processing completed data; The device also includes a transport type conversion module, which is used to obtain matrix weight data in the process of transporting the tensor data from the third-level cache to the second-level cache before the tensor data processing calculation; The multi-level storage of the memory includes a first-level cache, a second-level cache, and a third-level cache; the data copy speeds corresponding to the second-level cache and the third-level cache are both lower than those of the first-level cache; Obtaining a target operator slicing method for the matrix weight data through a preset matrix multiplication weight data rearrangement method; Wherein, the target operator slicing method includes copying the row size of the slices each time and copying the column size of the slices each time; Process the matrix weight data according to the row size of each copy and the column size of each copy, and calculate the number of row dimension copies and the number of column dimension copies respectively; Generate the weight of the four-dimensional data matrix according to the number of row dimension copies, the row size of each copy, the number of column dimension copies, and the column size of each copy; Performing shape transposition processing on the four-dimensional data matrix weights to obtain the four-dimensional data matrix weights after the shape transposition processing, and converting the transport type of the processed four-dimensional data matrix weights within the multi-level storage of the memory into linear copy transport; After the four-dimensional data matrix weights are subjected to shape transposition processing to obtain the four-dimensional data matrix weights after shape transposition processing, the method further includes: The key-value cache data includes key-value cache row data and key-value cache column data; each layer in the target large language model includes a key-value cache data block; according to the key-value cache row data, the key-value cache data block is divided into odd-numbered layer key-value cache data and even-numbered layer key-value cache data; For the key-value cache data, data storage allocation processing is performed according to the odd-numbered key-value cache data and the even-numbered key-value cache data by using a preset key-value cache data distributed storage method to generate a key-value cache data storage allocation result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements a data processing method based on an improved large language model reasoning operation framework as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a data processing method based on an improved large language model reasoning operation framework as described in any one of claims 1 to 5 when executed.

Images