Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: acquiring a reasoning request set of a large language model, wherein the reasoning request set comprises lexical element sequences corresponding to a plurality of reasoning requests respectively; a plurality of lexical element sequences in the reasoning request set are blocked, a plurality of sequence block groups corresponding to the reasoning request set are obtained, each sequence block group comprises a plurality of sequence blocks, and the difference value between the lengths of the sequence blocks in each sequence block group is smaller than a preset difference value threshold value; splicing the sequence blocks in each sequence block group to obtain a matrix corresponding to each sequence block group; the matrixes corresponding to the multiple sequence block groups are processed through an attention kernel function, attention features of the multiple reasoning requests are obtained, and the attention features are used for the reasoning process of the large language model. By dividing the lexical element sequence into the sequence block groups with the aligned lengths, a large amount of invalid calculation is avoided, and the calculation power utilization rate and the data processing efficiency are improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a data processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] Large language models have been widely used in various scenarios today, but in the small batch and multi-concurrent scenario, the data processing speed of the large language model is often limited and decreased. In this scenario, a large number of requests containing a small amount of data will be input into the large language model in a short time, and the model needs to frequently schedule resources to respond to different requests, which is difficult to efficiently process.

[0003] At present, the common processing method is to fill the short sequence with meaningless placeholder tokens to the maximum sequence length of the batch through padding operation, to realize sequence length alignment, and then uniformly execute matrix operation to complete batch analysis of inference requests.

[0004] However, this method has obvious defects. In order to realize sequence length alignment, a large number of meaningless placeholder tokens are introduced, which in turn produces a large amount of invalid calculation, resulting in serious waste of computing power, low computing power utilization rate and data processing efficiency. SUMMARY

[0005] The present disclosure provides a data processing method, device, electronic equipment and storage medium, which can avoid serious waste of computing power and improve the computing power utilization rate and data processing efficiency. The technical solutions of the present disclosure are as follows:

[0006] According to an aspect of an embodiment of the present disclosure, a data processing method is provided, which comprises:

[0007] Obtaining an inference request set of a large language model, the inference request set comprising token sequences corresponding to a plurality of inference requests respectively, and the token sequence corresponding to any inference request being obtained by encoding the inference request;

[0008] Dividing the plurality of token sequences in the inference request set into blocks respectively to obtain a plurality of sequence block groups corresponding to the inference request set, each sequence block group comprising a plurality of sequence blocks, and the difference between the lengths of the sequence blocks in each sequence block group being less than a preset difference threshold;

[0009] Splicing the sequence blocks in each sequence block group to obtain a matrix corresponding to each sequence block group;

[0010] Processing the matrices corresponding to the plurality of sequence block groups through an attention kernel function to obtain attention features of the plurality of inference requests, the attention features being used in the inference process of the large language model.

[0011] According to another aspect of the embodiments of the present disclosure, a data processing apparatus is provided, the apparatus comprising:

[0012] The acquisition unit is configured to acquire a set of inference requests of a large language model, the set of inference requests comprising a plurality of token sequences corresponding to a plurality of inference requests respectively, and any token sequence corresponding to an inference request being obtained by encoding the inference request;

[0013] The block unit is configured to perform blocking on the plurality of token sequences in the set of inference requests respectively to obtain a plurality of sequence block groups corresponding to the set of inference requests, each sequence block group comprising a plurality of sequence blocks, and a difference between lengths of the sequence blocks in each sequence block group being less than a preset difference threshold;

[0014] The splicing unit is configured to perform splicing on the sequence blocks in each sequence block group to obtain a matrix corresponding to each sequence block group;

[0015] The processing unit is configured to perform processing on the matrices corresponding to the plurality of sequence block groups by an attention kernel function to obtain attention features of the plurality of inference requests, the attention features being used in an inference process of the large language model.

[0016] In some embodiments, the block unit comprises:

[0017] The first division sub-unit is configured to divide, for any token sequence, the token sequence into a preset number of first sequence blocks corresponding to a length range to which the token sequence belongs based on the length range;

[0018] The first adding sub-unit is configured to add the preset number of first sequence blocks in a sequence block group corresponding to the length range.

[0019] In some embodiments, the first adding sub-unit is configured to pad the preset number of first sequence blocks to obtain a preset number of second sequence blocks, the length of the second sequence blocks being equal to a preset length corresponding to the length range; and add the preset number of second sequence blocks in the sequence block group corresponding to the length range.

[0020] In some embodiments, the block unit comprises:

[0021] The second division sub-unit is further configured to determine, for any token sequence, first sequence blocks and second sequence blocks in the token sequence based on a length range to which the token sequence belongs, the length of the first sequence blocks being equal to a preset length corresponding to the length range, and the second sequence blocks comprising a remaining part of the token sequence other than the first sequence blocks;

[0022] A second adding sub-unit configured to add the first type of sequence block and the second type of sequence block in the sequence block group corresponding to the length range.

[0023] In some embodiments, the second adding sub-unit is configured to add the first type of sequence block in the sequence block group corresponding to the length range; pad the second type of sequence block to obtain a third type of sequence block, the length of the third type of sequence block being equal to a preset length corresponding to the length range, and add the third type of sequence block in the sequence block group corresponding to the length range.

[0024] In some embodiments, the splicing unit is configured to, for any sequence block group, pad each sequence block in the sequence block group to obtain each intermediate sequence block in the sequence block group, the length of the intermediate sequence block being equal to the maximum length of the sequence blocks in the sequence block group; and splice each intermediate sequence block in the sequence block group to obtain the matrix corresponding to the sequence block group.

[0025] In some embodiments, the processing unit comprises:

[0026] An obtaining sub-unit configured to obtain a double-index mapping table, the double-index mapping table being used to indicate the relationship between the storage address of a sequence block and the position of the sequence block in the corresponding matrix;

[0027] A determining sub-unit configured to determine, by the attention kernel function, the sequence block corresponding to each thread in the matrix corresponding to the plurality of sequence block groups based on the double-index mapping table;

[0028] A processing sub-unit configured to call each thread to process the corresponding sequence block respectively to obtain the attention features of the plurality of inference requests.

[0029] In some embodiments, the double-index mapping table is used to indicate the mapping relationship between the storage address of a sequence block and a sequence block identifier, the sequence block identifier being the identifier of the sequence block in the corresponding sequence block group.

[0030] The double-index mapping table is also used to indicate the mapping relationship between the mask bit of a sequence block and the matrix offset, the mask bit being used to indicate whether the token in the sequence block is valid data or invalid data, and the matrix offset being used to indicate the position of the token in the corresponding matrix.

[0031] In some embodiments, the determining sub-unit is configured to determine, based on the double-index mapping table, a sequence block identifier and a matrix offset corresponding to each thread; and for any thread, based on the sequence block identifier and the matrix offset corresponding to the thread, determine a sequence block corresponding to the thread in a matrix in which a sequence block group indicated by the sequence block identifier is located.

[0032] In some embodiments, the processing sub-unit is configured to perform processing on the corresponding sequence block by each thread respectively to obtain an attention feature of each sequence block; and for each token sequence of an inference request, combine the attention features of the sequence blocks corresponding to the token sequence to obtain an attention feature of the inference request.

[0033] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, which comprises:

[0034] one or more processors;

[0035] a memory for storing the program code executable by the processor;

[0036] wherein the processor is configured to execute the program code to implement the above data processing method.

[0037] According to another aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, when instructions in the computer readable storage medium are executed by a processor of an electronic device, the electronic device can execute the above data processing method.

[0038] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, comprising program code, which when executed by a processor, implements the above data processing method.

[0039] The embodiments of the present disclosure provide a data processing method, by dividing a token sequence into blocks and grouping the blocks according to similar lengths, a large amount of meaningless padding is avoided, thereby avoiding a large amount of invalid calculation and improving the utilization rate of computing resources. On this basis, each length-aligned sequence block group is processed in a matrix manner, so that the subsequent processing process using an attention kernel function can be better implemented in parallel, thereby improving the data processing efficiency and improving the inference speed of a large language model when processing multiple requests.

[0040] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure without imposing undue limitations on the disclosure.

[0042] Figure 1 is a schematic diagram of an implementation environment of a data processing method according to an example embodiment.

[0043] Figure 2 is a flowchart of a data processing method according to an example embodiment.

[0044] Figure 3 is a flowchart of another data processing method according to an example embodiment.

[0045] Figure 4 is a block diagram of a data processing apparatus according to an example embodiment.

[0046] Figure 5 is a block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0047] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings.

[0048] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. The implementation described in the following example embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0049] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the inference request set involved in the present disclosure is obtained under sufficient authorization.

[0050] Figure 1 is a schematic diagram of an implementation environment of a data processing method according to an example embodiment. Referring to Figure 1 , the implementation environment specifically includes a terminal 101 and a server 102. The terminal 101 can be connected to the server 102 through a wireless network or a wired network.

[0051] In some embodiments, the terminal 101 can be at least one of a smartphone, a smart watch, a desktop computer, a laptop computer, and the like. The terminal 101 can be installed and run with an application program having data processing capability. A user can log in to the application program through the terminal 101 to obtain services provided by the application program. The application program is associated with the server 102, and the server 102 provides background services to the terminal 101. The terminal 101 can be referred to as one of a plurality of terminals, and the following exemplary embodiments are exemplified by the terminal 101. Those skilled in the art can know that the number of the terminals can be more or less. For example, the terminals can be several, or the terminals can be tens or hundreds, or more, and the number of the terminals and the types of the devices are not limited in the embodiments of the present disclosure.

[0052] In some embodiments, the server 102 is a stand-alone physical server, and can also be a server cluster or a distributed system composed of a plurality of physical servers, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, and the like. That is, the server 102 is at least one of a server, a plurality of servers, a cloud computing platform, and a virtualization center. Optionally, the number of the servers can be more or less, and the embodiments of the present disclosure do not limit this. Of course, the server 102 can also include other functional servers to provide more comprehensive and diversified services.

[0053] In some embodiments, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or the server 102 and the terminal 101 cooperatively compute in a distributed computing architecture. The server 102 can be connected to the terminal 101 and other terminals through a wireless network or a wired network, and optionally, the number of the servers can be more or less, and the embodiments of the present disclosure do not limit this.

[0054] Large language models (LLMs) are widely used in various scenarios today, but in small batch multi-concurrent scenarios such as application programming interface (API) services and real-time interactions, the data processing speed of the large language model is often limited and decreased. In such scenarios, a large number of requests containing a small amount of data are input into the large language model in a short period of time, and the model needs to frequently schedule resources to respond to different requests, making it difficult to efficiently process.

[0055] Currently, the commonly used processing method is dynamic batching, which groups different lengths of inference requests in a short period of time into multiple batches, fills different lengths of sequences in the same batch with meaningless token, and supplements short sequences to the longest sequence length of the batch to achieve sequence length alignment, so that the data meets the requirement of fixed dimensions for matrix operation, and then performs matrix operation on the whole batch of sequences to complete batch processing. However, this method has obvious defects, which will be described in detail below.

[0056] First, the computing power utilization rate is low. Different lengths of sequences in the batch need to be padded according to the longest sequence in the batch. To achieve sequence length alignment, a large number of meaningless tokens are introduced. These padding parts will participate in subsequent operations (such as attention calculation and linear layer operation) but do not produce effective results, thereby generating a large amount of redundant calculation, such as more than 40% of invalid calculation, resulting in serious waste of computing power, such as 30-60% of wasted computing power. For example, when the sequence length difference in the batch is greater than 2 times, the floating-point operations per second (FLOPs) utilization rate is less than 55%, reducing the utilization rate of computing resources.

[0057] Second, sparse computation is inefficient. Taking grouped-query attention (GQA) as an example, the non-uniform sparse attention matrix generated by this scheme is non-continuous, which destroys the memory continuity. The efficient computation of GPU / TPU relies on the reading of continuous memory blocks, and the sparse elements in the non-uniform sparse attention matrix are scattered in non-continuous memory addresses, which causes the hardware to frequently jump between non-continuous memory addresses to read data when accessing the video memory, i.e. frequent skipping, resulting in a large amount of additional memory scheduling overhead, such as an increase of 70% in video memory access overhead, reducing the overall efficiency.

[0058] In addition, parallelism is insufficient. Traditional dynamic batch processing methods cannot fully utilize fine-grained parallelism when processing inference tasks of large language models. Streaming multiprocessors (SMs) in GPUs can efficiently perform parallel computation, but traditional methods cannot finely distribute tasks to each streaming multiprocessor, resulting in low utilization of streaming multiprocessors, such as less than 65%, which affects the overall performance of large language models.

[0059] Finally, GQA performance degrades. The sparse attention matrix in this scheme has randomness when accessed, which cannot effectively utilize the bandwidth of dynamic random access memory (DRAM), resulting in waste of bandwidth resources, such as a 45% decrease in bandwidth utilization.

[0060] Currently, there is no good solution to the fragmentation problem of dynamic batch processing. Based on this, the present disclosure proposes a method for inference acceleration of large language models, which is suitable for multi-batch and multi-concurrent scenarios, dynamically reorganizes the inference request set of large language models and optimizes sparse matrices, and is suitable for GPU / TPU inference deployment of the Transformer architecture. The method proposed in this application is introduced below.

[0061] Figure 2 is a flowchart of a data processing method according to an exemplary embodiment, as shown in Figure 2 The method is performed by an electronic device and includes the following steps:

[0062] In step 201, the electronic device obtains an inference request set of a large language model, the inference request set including a plurality of word token sequences corresponding to a plurality of inference requests respectively, and any word token sequence corresponding to an inference request is obtained by encoding the inference request.

[0063] In the embodiments of the present disclosure, the word token sequences in the inference request set correspond to respective inference requests of the large language model, such as question and answer inference requests, text generation inference requests, translation inference requests, and summary inference requests. Exemplarily, the inference request is a natural language segment in a text scenario, and the word token sequence is an ordered list composed of word tokens. The sequence length refers to the number of word tokens in the word token sequence. In some embodiments, the inference request set is a set of inference requests in a batch in a dynamic batch processing manner.

[0064] In step 202, the electronic device divides the plurality of word token sequences in the inference request set respectively to obtain a plurality of sequence block groups corresponding to the inference request set, each sequence block group including a plurality of sequence blocks, and the difference between the lengths of the sequence blocks in each sequence block group is less than a preset difference threshold.

[0065] In the embodiments of the present disclosure, each word sequence is split to obtain a sequence block corresponding to each word sequence, that is, a sequence fragment. The number of sequence blocks corresponding to each word sequence is one or more, which is not limited in the embodiments of the present disclosure. The length of a sequence block refers to the number of word sequences contained in the sequence block.

[0066] The sequence blocks corresponding to the plurality of word sequences in the inference request are integrated into a plurality of sequence block groups, that is, a sequence fragment set. The lengths of the sequence blocks in each sequence block group are close, and the lengths of the sequence blocks in different sequence block groups are greatly different. For example, the lengths of the sequence blocks in one sequence block group are close to T / 2, and the lengths of the sequence blocks in another sequence block group are close to T. This makes the lengths of the sequence blocks in each sequence block group consistent or have little difference, so that there is no need for padding or a small amount of padding when splicing into a matrix, reducing invalid calculation, and the dimensions of the matrix corresponding to each sequence block are fixed, so that the hardware resources can perform parallel calculation on the matrices of the same dimension in batches, improving the data processing efficiency.

[0067] Exemplarily, the plurality of sequence block groups correspond to the same preset difference threshold value; or each sequence block group corresponds to a respective preset difference threshold value, and the preset difference threshold values corresponding to different sequence block groups can be different. The preset difference threshold value can be adjusted according to actual needs, which is not limited in the embodiments of the present disclosure. In some embodiments, the sequence blocks in the sequence block group do not need to be padded, and adaptive padding is performed when the matrix is spliced subsequently.

[0068] In step 203, the electronic device splices the sequence blocks in each sequence block group to obtain a matrix corresponding to each sequence block group.

[0069] In the embodiments of the present disclosure, the matrix refers to a dense matrix obtained by splicing the sequence blocks. Compared with the sparse attention matrix in the traditional way, the non-zero elements corresponding to the effective word in the matrix account for a high proportion, and only a small amount of placeholder words for padding are included, and the zero values in the matrix are fewer.

[0070] In the traditional dynamic batch processing mode, since each word sequence is padded to the length of the longest sequence in a batch, this results in a non-uniform sparse matrix obtained subsequently, and there are a large number of zero values corresponding to meaningless words in the matrix, which causes the hardware to frequently jump between discontinuous memory addresses when accessing the video memory to read data, causing serious negative effects on the request analysis process of the large language model. In the present disclosure, since the lengths of the sequence blocks in the sequence block group are very close, even if padding is performed, there are only a small amount of placeholder words, and most of the elements in the dense matrix obtained by splicing the sequence blocks in the same group are effective values, that is, most of the elements in the matrix are non-zero values, which greatly improves the data processing efficiency during subsequent matrix operation.

[0071] In step 204, the electronic device processes the matrix corresponding to the plurality of sequence block groups through an attention kernel function to obtain attention features of the plurality of inference requests, which are used in the inference process of the large language model.

[0072] In the embodiments of the present disclosure, the attention kernel function is a function used to process the attention mechanism in the large language model. The attention kernel function processes the attention matrix by determining the correlation degree between different elements. By processing the matrix corresponding to each sequence block group through the attention kernel function, the corresponding attention features can be obtained. Due to the memory continuity and high effective data proportion of the dense matrix, the attention kernel function can improve the computing efficiency through continuous memory access, avoiding the waste of computing power caused by skipping invalid data in the traditional sparse attention matrix processing.

[0073] For example, the attention features can be input into the decoder (such as the decoder layer of the Transformer) of the large language model to generate the final output of the inference request, such as the dialogue answer, generated text, translation result, text summary, etc. The attention features are the key intermediate data for connecting feature extraction and result generation.

[0074] The embodiments of the present disclosure provide a data processing method, which avoids a large amount of meaningless padding by blocking and grouping the token sequences according to similar lengths, thereby avoiding a large amount of invalid calculation and improving the utilization rate of computing resources. On this basis, each length-aligned sequence block group is processed in the form of a matrix, so that the subsequent processing process using the attention kernel function can be better implemented in parallel, thereby improving the data processing efficiency and the inference speed of the large language model when processing multiple requests.

[0075] In some embodiments, the plurality of token sequences in the inference request set are respectively blocked to obtain a plurality of sequence block groups corresponding to the inference request set, including:

[0076] For any token sequence, the token sequence is divided into a preset number of first sequence blocks corresponding to the length range to which the token sequence belongs.

[0077] The preset number of first sequence blocks are added to the sequence block group corresponding to the length range.

[0078] In the embodiments of the present disclosure, the token sequence is divided into a corresponding number of sequence blocks according to the length range and is grouped into the corresponding sequence block group, which ensures that the lengths of the sequence blocks in the same group are consistent and makes the subsequent matrix splicing and attention calculation more targeted, reduces the calculation redundancy caused by mixing different length sequences, and is beneficial to subsequent parallel calculation, thereby improving the computing efficiency.

[0079] In some embodiments, the preset number of first sequence blocks are added in the sequence block group corresponding to the length range, including:

[0080] The preset number of first sequence blocks are padded to obtain a preset number of second sequence blocks, and the length of the second sequence block is equal to the preset length corresponding to the length range;

[0081] The preset number of second sequence blocks are added in the sequence block group corresponding to the length range.

[0082] In the embodiments of the present disclosure, the padding of the sequence blocks unifies the lengths of the sequence blocks in the same group, which eliminates the length difference of the sequence blocks in the same group, facilitates subsequent sequence block splicing and matrix parallel operation, avoids additional overhead caused by uneven data length when the attention kernel function is processed, reduces the waste of computing resources, and ensures the data processing efficiency.

[0083] In some embodiments, the plurality of token sequences in the inference request set are respectively divided into blocks to obtain a plurality of sequence block groups corresponding to the inference request set, including:

[0084] For any token sequence, based on the length range to which the token sequence belongs, the first type of sequence block and the second type of sequence block in the token sequence are determined, the length of the first type of sequence block is equal to the preset length corresponding to the length range, and the second type of sequence block includes the remaining part of the token sequence except the first type of sequence block;

[0085] The first type of sequence block and the second type of sequence block are added in the sequence block group corresponding to the length range.

[0086] In the embodiments of the present disclosure, by dividing the first type of sequence block of fixed length and the second type of sequence block containing the remaining part, and being classified into the corresponding sequence block group, subsequent sequence block padding can be more efficiently performed, thereby adapting to the subsequent parallel computing requirements, and improving the data processing efficiency.

[0087] In some embodiments, the first type of sequence block and the second type of sequence block are added in the sequence block group corresponding to the length range, including:

[0088] The first type of sequence block is added in the sequence block group corresponding to the length range;

[0089] The second type of sequence block is padded to obtain a third type of sequence block, and the length of the third type of sequence block is equal to the preset length corresponding to the length range, and the third type of sequence block is added in the sequence block group corresponding to the length range.

[0090] In the embodiments of the present disclosure, the length of the second type of sequence block after padding is consistent with the first type of sequence block, which makes the sequence blocks in the same group adapt to matrix splicing and parallel processing, avoids additional overhead caused by uneven data length when the attention kernel function is processed, reduces waste of computing resources, and ensures data processing efficiency.

[0091] In some embodiments, the sequence blocks in each sequence block group are spliced to obtain a matrix corresponding to each sequence block group, including:

[0092] For any sequence block group, each sequence block in the sequence block group is padded to obtain each intermediate sequence block in the sequence block group, and the length of the intermediate sequence block is equal to the maximum length of the sequence blocks in the sequence block group;

[0093] Each intermediate sequence block in the sequence block group is spliced to obtain a matrix corresponding to the sequence block group.

[0094] In the embodiments of the present disclosure, by padding the sequence blocks to the maximum length in the group to obtain the intermediate sequence blocks, and then splicing them into a matrix, it is ensured that the lengths of all sequence blocks in the same group are consistent, which facilitates better adaptation to the parallel computing architecture of the hardware, enables the attention kernel function to perform continuous and efficient batch processing on the matrix, reduces invalid computation and memory access overhead caused by chaotic data dimensions, improves the attention feature extraction efficiency, and thus improves the overall inference efficiency.

[0095] In some embodiments, the matrices corresponding to the plurality of sequence block groups are processed by the attention kernel function to obtain the attention features of the plurality of inference requests, including:

[0096] A double-index mapping table is obtained, which is used to indicate the relationship between the storage address of the sequence block and the position of the sequence block in the corresponding matrix;

[0097] Based on the double-index mapping table, the sequence block corresponding to each thread is determined in the matrices corresponding to the plurality of sequence block groups by the attention kernel function;

[0098] Each thread is called to process the corresponding sequence block to obtain the attention features of the plurality of inference requests.

[0099] In the embodiments of the present disclosure, the double-index mapping table enables the thread to quickly locate the corresponding sequence block, reduces data transmission and redundant operations, improves data processing efficiency, and different threads can simultaneously locate different sequence blocks in the matrix, so that each thread can process multiple sequence blocks in parallel, improves the utilization rate of hardware parallelism, and further improves the data processing speed of the large language model inference process.

[0100] In some embodiments, the double-index mapping table is used to indicate a mapping relationship between a storage address of a sequence block and a sequence block identifier, the sequence block identifier being an identifier of the sequence block in a corresponding sequence block group.

[0101] The double-index mapping table is also used to indicate a mapping relationship between a mask bit of the sequence block and a matrix offset, the mask bit being used to indicate whether a token in the sequence block is valid data or invalid data, and the matrix offset being used to indicate a position of the token in the sequence block in a corresponding matrix.

[0102] In the embodiments of the present disclosure, the double-index mapping table is used to associate the sequence block storage address with the identifier, and the mask bit with the matrix offset, so as to facilitate quick positioning of the sequence block. Such multi-dimensional mapping not only guarantees the accuracy of data processing, but also reduces invalid calculation overhead, and improves the efficiency and accuracy of attention feature extraction.

[0103] In some embodiments, based on the double-index mapping table, in the matrices corresponding to the plurality of sequence block groups, a sequence block corresponding to each thread is determined, including:

[0104] Based on the double-index mapping table, a sequence block identifier and a matrix offset corresponding to each thread are determined.

[0105] For any thread, based on the sequence block identifier and the matrix offset corresponding to the thread, a sequence block corresponding to the thread is determined in a matrix in which the sequence block group indicated by the sequence block identifier is located.

[0106] In the embodiments of the present disclosure, the sequence block identifier is associated with the sequence block group to which it belongs, and the matrix offset indicates the specific position of the sequence block in the matrix, which enables the thread to quickly locate the corresponding sequence block, ensures that each thread can timely obtain the data to be processed, reduces thread waiting time and memory access delay, and improves the stability and efficiency of multi-thread parallel computing.

[0107] In some embodiments, each thread is called to process the corresponding sequence block, to obtain attention features of the plurality of reasoning requests, including:

[0108] Each thread is called to process the corresponding sequence block, to obtain attention features of each sequence block.

[0109] For a token sequence of each reasoning request, the attention features of the sequence block corresponding to the token sequence are merged, to obtain the attention features of each reasoning request.

[0110] In the embodiments of the present disclosure, the attention features of each sequence block are obtained by multi-threading, and the features of the corresponding sequence block are merged according to the reasoning request, to ensure that the attention features of each reasoning request can completely reflect the semantic association of the corresponding token sequence, and to guarantee the accuracy of the subsequent large language model reasoning result.

[0111] The above Figure 2 The flow chart of the data processing method of the present disclosure is shown, and the data processing scheme provided by the present disclosure is further described below. Figure 3 The flow chart of another data processing method according to an exemplary embodiment is shown, referring to Figure 3 The method is performed by an electronic device, including the following steps:

[0112] In step 301, the electronic device obtains a set of inference requests of a large language model, the set of inference requests including a plurality of word token sequences corresponding to the plurality of inference requests respectively, and any word token sequence corresponding to an inference request is obtained by encoding the inference request.

[0113] In the embodiment of the present disclosure, the principle and step 201 of the electronic device obtaining the set of inference requests of the large language model are the same, and the embodiment of the present disclosure will not be described here.

[0114] If the electronic device is implemented as a background server of the large language model, the electronic device obtains the set of inference requests from an associated business server, or receives the inference requests sent by the client. If the electronic device is implemented as a terminal configured with a large language model application, the electronic device obtains the set of inference requests generated by local user operation.

[0115] In step 302, the electronic device respectively blocks the plurality of word token sequences in the set of inference requests to obtain a plurality of sequence block groups corresponding to the set of inference requests, each sequence block group including a plurality of sequence blocks, and the difference between the lengths of the sequence blocks in each sequence block group is less than a preset difference threshold.

[0116] In the embodiment of the present disclosure, the principle of the electronic device obtaining a plurality of sequence block groups (Block Group) by blocking is the same as the above-mentioned step 202, and the embodiment of the present disclosure will not be described here.

[0117] In some embodiments, different preset manners are adopted for division when the length ranges to which the token sequences belong are different. For example, different blocking strategies are adopted for long sequences and short sequences. Illustratively, the plurality of token sequences in the inference request set are arranged in ascending order of sequence length, and the determination threshold T of long and short sequences is set to 1 / 4 of the maximum length of the token sequences in the inference request set. For token sequences with a length less than the determination threshold, i.e., short sequences, the first preset manner is adopted to split them into a plurality of sequence blocks; for token sequences with a length not less than the determination threshold, i.e., long sequences, the second preset manner is adopted to keep them as a whole. This realizes dynamic adaptation of token sequence blocking. On this basis, the sequence blocks obtained after blocking the token sequences in the inference request set are recombined, such as two sequence block groups. The lengths of the sequence blocks in each sequence block group are close, and the lengths of the sequence blocks in one of the sequence block groups are close to T / 2, and the lengths of the sequence blocks in the other sequence block group are close to T.

[0118] The above uses examples to illustrate the adoption of different blocking strategies for token sequences in different length ranges. The detailed blocking strategies for dividing token sequences are introduced below by way of way one and way two.

[0119] Way one: divide token sequences into a corresponding number of sequence blocks according to length ranges, and assign them to corresponding sequence block groups. Correspondingly, for any token sequence, based on the length range to which the token sequence belongs, the token sequence is divided into a preset number of first sequence blocks corresponding to the length range; and the preset number of first sequence blocks are added to the sequence block group corresponding to the length range. This ensures that the lengths of the sequence blocks in the same group are consistent, and also makes the subsequent matrix splicing and attention calculation more targeted, reduces the computational redundancy caused by mixing different length sequences, is conducive to subsequent parallel calculation, and improves the calculation efficiency.

[0120] In some embodiments, the sequence blocks are padded to unify the lengths of the sequence blocks in the same group. Correspondingly, the preset number of first sequence blocks are padded to obtain a preset number of second sequence blocks, and the lengths of the second sequence blocks are equal to the preset length corresponding to the length range; and the preset number of second sequence blocks are added to the sequence block group corresponding to the length range. This eliminates the length difference of the sequence blocks in the same group, and the padding operation required when splicing matrices is completed in advance, facilitating subsequent sequence block splicing and matrix parallel operation, avoiding additional overheads when the attention kernel function is processed due to uneven data length, reducing waste of computing resources, and ensuring data processing efficiency.

[0121] The second way is to divide the first type of sequence block with a fixed length and the second type of sequence block containing the remaining part, and to be classified into the corresponding sequence block group. Accordingly, for any token sequence, based on the length range to which the token sequence belongs, the first type of sequence block and the second type of sequence block in the token sequence are determined, the length of the first type of sequence block is equal to the preset length corresponding to the length range, and the second type of sequence block includes the remaining part of the first type of sequence block in the token sequence; the first type of sequence block and the second type of sequence block are added in the sequence block group corresponding to the length range. This facilitates more efficient sequence block padding when splicing the matrix subsequently, thereby adapting to the parallel computing requirement and improving the data processing efficiency.

[0122] In some embodiments, the length of the second type of sequence block after padding is consistent with that of the first type of sequence block. Accordingly, the first type of sequence block is added in the sequence block group corresponding to the length range; the second type of sequence block is padded to obtain a third type of sequence block, the length of the third type of sequence block is equal to the preset length corresponding to the length range, and the third type of sequence block is added in the sequence block group corresponding to the length range. This completes the padding operation in advance when splicing the matrix, so that the sequence blocks in the same group adapt to matrix splicing and parallel processing, avoids additional overhead due to uneven data length when the attention kernel function is processed, reduces the waste of computing resources, and ensures the data processing efficiency.

[0123] In step 303, the electronic device pads each sequence block in the sequence block group to obtain each intermediate sequence block in the sequence block group, and the length of the intermediate sequence block is equal to the maximum length of the sequence blocks in the sequence block group.

[0124] In the embodiments of the present disclosure, in order to adapt to the requirement of matrix parallel computing, each sequence block in the same group is padded to the same length, which facilitates better adaptation to the parallel computing architecture of hardware, enables the attention kernel function to perform continuous and efficient batch processing on the matrix, reduces invalid computation and memory access overhead caused by data dimension disorder, and improves the attention feature extraction efficiency.

[0125] In step 304, the electronic device splices each intermediate sequence block in the sequence block group to obtain a matrix corresponding to the sequence block group.

[0126] In the embodiments of the present disclosure, each intermediate sequence block in each sequence block group is spliced by row, and each sequence block group corresponds to a continuous dense matrix (Dense Block Matrix).

[0127] In one embodiment, CUDA unified memory is enabled to manage memory migration. In a conventional manner, sequence blocks can be stored in scattered memory and cannot be accessed continuously. In this manner, sequence block data is scheduled to a continuous memory region, avoiding the overhead of data copying. When GPU threads need to read data of a sequence block, they can read in bulk from continuous memory addresses without skipping other scattered addresses, reducing memory access latency and improving parallel computing efficiency.

[0128] The above steps 303 and 304 are an exemplary manner of splicing sequence blocks in each sequence block group to obtain a matrix corresponding to each sequence block group, which does not constitute a limitation.

[0129] In step 305, the electronic device determines a sequence block corresponding to each thread in the matrix corresponding to the plurality of sequence block groups based on the dual-index mapping table through an attention kernel function, the dual-index mapping table being used to indicate a relationship between a storage address of the sequence block and a position of the sequence block in the corresponding matrix.

[0130] In the embodiments of the present disclosure, the dual-index mapping table enables threads to quickly locate corresponding sequence blocks without data copying, reduces data transmission and redundant operations, improves data processing efficiency, and enables different threads to simultaneously locate different sequence blocks in the matrix, so that each thread can process multiple sequence blocks in parallel, can fully utilize multi-core hardware resources, improves hardware parallelism utilization, and further improves the data processing speed of the large language model inference process.

[0131] In some embodiments, the dual-index mapping table is used to indicate a mapping relationship between a storage address of the sequence block and a sequence block identifier; the dual-index mapping table is also used to indicate a mapping relationship between a mask bit of the sequence block and a matrix offset. The storage address of the sequence block is a physical address actually stored by the sequence block, indicating an actual storage location in the video memory. The sequence block identifier is an identifier of the sequence block in the corresponding sequence block group, also referred to as a logical block ID for ease of description, which is a number of the sequence block in the sequence block group to which the sequence block belongs. The mask bit is used to indicate that a token in the sequence block is valid data or invalid data, which is a binary bit sequence used to mark valid tokens in the sequence block. Each bit corresponds to a token in the sequence block. If the bit is 1, the corresponding token is valid data, and vice versa. The matrix offset is used to indicate a position of a token in the sequence block in the corresponding matrix.

[0132] By mapping the sequence block identifier and the storage address, the corresponding sequence block can be quickly found according to the storage address, and the group to which the sequence block belongs can be quickly located. By the mask bit and the matrix offset, the valid data can be quickly identified, and the position of the valid data in the matrix can be determined according to the matrix offset, so that the attention kernel function is only calculated for the valid data, avoiding the interference of invalid data on the result and reducing the invalid calculation overhead, thereby improving the data processing efficiency.

[0133] In some embodiments, the thread quickly locates the corresponding sequence block through the sequence block identifier and the matrix offset. Accordingly, the sequence block identifier and the matrix offset corresponding to each thread are determined based on the double-index mapping table; for any thread, the sequence block corresponding to the thread is determined in the matrix in which the sequence block group indicated by the sequence block identifier is located, based on the sequence block identifier and the matrix offset corresponding to the thread. This ensures that each thread can timely obtain the data to be processed, reduces the thread waiting time and memory access delay, realizes efficient matching of the thread and the sequence block, and improves the efficiency of multi-thread parallel computing.

[0134] Illustratively, the sequence block identifier and the matrix offset are determined based on the index of the current thread through the double-index mapping table; and the dense data block in the matrix, i.e., the sequence block data processed by the thread, is determined based on the matrix offset and the address of the matrix. This can quickly locate the address of the sequence block in the matrix without extracting from the sparse matrix or skipping the invalid values filled, can quickly perform attention calculation on the continuously stored dense data block, and improves the data processing efficiency.

[0135] In step 306, the electronic device calls each thread to process the corresponding sequence block, to obtain the attention features of the plurality of inference requests.

[0136] In the embodiments of the present disclosure, the attention features are used in the subsequent inference process of the large language model. Illustratively, the attention features can be input into the decoder of the large language model to generate the final output of the inference request, such as a dialogue answer, generated text, translation result, text summary, etc.

[0137] In some embodiments, the features of the corresponding sequence blocks are merged according to the inference request. Accordingly, each thread is called to process the corresponding sequence block, to obtain the attention features of each sequence block; and for the token sequence of each inference request, the attention features of the sequence blocks corresponding to the token sequence are merged, to obtain the attention features of each inference request. This ensures that the attention features of each inference request can completely reflect the semantic association of the corresponding token sequence, and guarantees the accuracy of the subsequent inference result of the large language model.

[0138] The steps 305 and 306 are an exemplary manner of processing the matrices corresponding to the plurality of sequence block groups by the attention kernel function to obtain the attention features of the plurality of inference requests, and do not constitute a limitation.

[0139] The data processing method provided in the embodiments of the present disclosure avoids a large amount of meaningless padding by blocking and grouping the token sequences according to similar lengths, thereby avoiding a large amount of invalid calculation and improving the utilization rate of computing resources. On this basis, the sequence block groups of each length are processed in a matrix form, so that the subsequent processing process using the attention kernel function can be better parallelized, thereby improving the data processing efficiency and improving the inference speed of the large language model when processing multiple requests. The following will be introduced from the following aspects.

[0140] First, dynamic blocking and reorganization are performed in the method, the length difference of the sequence blocks in each sequence block group after the inference request set is split is very small, and the padding part when the matrix is spliced is also very small. Compared with the traditional method, a large amount of padding is avoided, and the redundant padding is reduced by 82%. This avoids a large amount of meaningless calculation and improves the utilization rate of computing resources. For example, when the length variance of the token sequence is greater than 2, the utilization rate of FLOPs is improved from 55% to 95%.

[0141] Second, matrix dense packaging is performed in the method, the sequence block groups are rearranged into a continuous dense matrix, the memory locality is improved, and the memory jump reading overhead caused by non-continuous access is reduced. This improves the inference speed of the large language model, and compared with the traditional method, the inference speedup ratio is improved by 3.8 times, and the utilization rate of the stream processor is also improved, such as from 65% to 92%.

[0142] In addition, GQA sparse optimization is performed in the method, a double-index mapping table is established for the sequence blocks, the effective calculation area can be quickly located without reorganizing the matrix, the data can be accessed in a continuous memory mode, zero-copy calculation is realized, the DRAM bandwidth utilization rate is improved, such as reducing the bandwidth occupation by 57%, and the attention feature extraction speed is also improved, such as reducing the attention calculation delay by 1 / 4.1.

[0143] All the optional technical solutions described above can be combined to form optional embodiments of the present disclosure, which will not be described one by one here.

[0144] Figure 4 is a block diagram of a data processing device according to an exemplary embodiment. As shown in Figure 4 The device includes an acquisition unit 401, a blocking unit 402, a splicing unit 403, and a processing unit 404.

[0145] The acquisition unit 401 is configured to acquire an inference request set of a large language model, the inference request set including a plurality of token sequences corresponding to a plurality of inference requests respectively, and any token sequence corresponding to an inference request being obtained by encoding the inference request;

[0146] The blocking unit 402 is configured to perform blocking on the plurality of token sequences in the inference request set respectively to obtain a plurality of sequence block groups corresponding to the inference request set, each sequence block group including a plurality of sequence blocks, and a difference between lengths of the sequence blocks in each sequence block group being less than a preset difference threshold;

[0147] The splicing unit 403 is configured to perform splicing on the sequence blocks in each sequence block group to obtain a matrix corresponding to each sequence block group.

[0148] The processing unit 404 is configured to perform processing on the matrices corresponding to the plurality of sequence block groups by an attention kernel function to obtain attention features of the plurality of inference requests, the attention features being used in an inference process of the large language model.

[0149] In some embodiments, the blocking unit 402 includes:

[0150] The first division subunit is configured to divide, for any token sequence, the token sequence into a preset number of first sequence blocks corresponding to a length range to which the token sequence belongs based on the length range.

[0151] The first adding subunit is configured to add the preset number of first sequence blocks in the sequence block group corresponding to the length range.

[0152] In some embodiments, the first adding subunit is configured to pad the preset number of first sequence blocks to obtain a preset number of second sequence blocks, the length of the second sequence block being equal to a preset length corresponding to the length range; and add the preset number of second sequence blocks in the sequence block group corresponding to the length range.

[0153] In some embodiments, the blocking unit 402 includes:

[0154] The second division subunit is further configured to determine, for any token sequence, first sequence blocks and second sequence blocks in the token sequence based on a length range to which the token sequence belongs, the length of the first sequence block being equal to a preset length corresponding to the length range, and the second sequence block including a remaining part of the token sequence excluding the first sequence block.

[0155] The second adding subunit is configured to add the first sequence blocks and the second sequence blocks in the sequence block group corresponding to the length range.

[0156] In some embodiments, the second adding subunit is configured to perform adding the first type of sequence block in the sequence block group corresponding to the length range; padding the second type of sequence block to obtain a third type of sequence block, the length of the third type of sequence block being equal to the preset length corresponding to the length range, and adding the third type of sequence block in the sequence block group corresponding to the length range.

[0157] In some embodiments, the splicing unit 403 is configured to perform, for any sequence block group, padding each sequence block in the sequence block group to obtain each intermediate sequence block in the sequence block group, the length of the intermediate sequence block being equal to the maximum length of the sequence blocks in the sequence block group; and splicing each intermediate sequence block in the sequence block group to obtain the matrix corresponding to the sequence block group.

[0158] In some embodiments, the processing unit 404 includes:

[0159] The obtaining subunit is configured to perform obtaining a double-index mapping table, the double-index mapping table being used to indicate the relationship between the storage address of the sequence block and the position of the sequence block in the corresponding matrix;

[0160] The determining subunit is configured to perform determining, by using an attention kernel function, the sequence block corresponding to each thread in the matrix corresponding to the plurality of sequence block groups based on the double-index mapping table.

[0161] The processing subunit is configured to perform calling each thread to process the corresponding sequence block respectively to obtain the attention features of the plurality of inference requests.

[0162] In some embodiments, the double-index mapping table is used to indicate the mapping relationship between the storage address of the sequence block and the sequence block identifier, the sequence block identifier being the identifier of the sequence block in the corresponding sequence block group.

[0163] The double-index mapping table is also used to indicate the mapping relationship between the mask bit of the sequence block and the matrix offset, the mask bit being used to indicate whether the token in the sequence block is valid data or invalid data, and the matrix offset being used to indicate the position of the token in the sequence block in the corresponding matrix.

[0164] In some embodiments, the determining subunit is configured to perform determining, based on the double-index mapping table, the sequence block identifier and the matrix offset corresponding to each thread; and for any thread, determining the sequence block corresponding to the thread in the matrix in which the sequence block group indicated by the sequence block identifier is located based on the sequence block identifier and the matrix offset corresponding to the thread.

[0165] In some embodiments, the processing subunit is configured to perform calling respective threads to process corresponding sequence blocks respectively to obtain attention features of each sequence block; and merging the attention features of the sequence blocks corresponding to the token sequence of each inference request to obtain the attention feature of each inference request.

[0166] The embodiments of the present disclosure provide a data processing apparatus. By dividing the token sequence into blocks and grouping the blocks according to similar lengths, a large amount of meaningless padding is avoided, a large amount of invalid calculation is avoided, and the utilization rate of computing resources is improved. On this basis, each length-aligned sequence block group is processed in a matrix manner, so that the subsequent processing process using the attention kernel function can be better parallelized, the data processing efficiency is improved, and the inference speed of the large language model when processing multiple requests is improved.

[0167] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure.

[0168] It should be noted that the data processing apparatus provided in the above embodiments is only exemplified by the division of the above functional units. In actual applications, the above functions can be completed by different functional units according to needs, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the data processing apparatus and the data processing method provided in the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be described here.

[0169] As for the data processing apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0170] Figure 5 is a block diagram of an electronic device according to an exemplary embodiment. Generally, the electronic device 500 includes a processor 501 and a memory 502.

[0171] The processor 501 can include one or more processing cores, such as a 4-core processor, a 5-core processor, and the like. The processor 501 can be implemented in at least one of a hardware manner such as a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 501 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 501 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content required to be displayed by the display screen. In some embodiments, the processor 501 can also include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0172] The memory 502 can include one or more computer-readable storage media that can be non-transitory. The memory 502 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 502 is used to store at least one program code for being executed by the processor 501 to implement the data processing method provided by the method embodiments in the present disclosure.

[0173] In some embodiments, the electronic device 500 can also optionally include a peripheral device interface 503 and at least one peripheral device. The processor 501, the memory 502, and the peripheral device interface 503 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 503 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 504, a display screen 505, a camera assembly 506, an audio circuit 507, and a power supply 508.

[0174] The peripheral interface 503 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 501 and the memory 502. In some embodiments, the processor 501, the memory 502 and the peripheral interface 503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 501, the memory 502 and the peripheral interface 503 can be implemented on a separate chip or circuit board, and the present embodiments are not limited in this regard.

[0175] The radio frequency circuit 504 is configured to receive and send RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 504 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 504 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 504 can communicate with other electronic devices through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 504 can also include NFC (Near Field Communication) related circuitry, and the present disclosure is not limited in this regard.

[0176] The display screen 505 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 505 is a touch display screen, the display screen 505 is further configured to capture touch signals on or above the surface of the display screen 505. The touch signals can be input to the processor 501 as control signals for processing. In this case, the display screen 505 can also be configured to provide virtual controls and / or virtual keyboards, also known as soft controls and / or soft keyboards. In some embodiments, the display screen 505 can be one, configured on the front panel of the electronic device 500; in other embodiments, the display screen 505 can be at least two, respectively configured on different surfaces of the electronic device 500 or in a folding design; in yet other embodiments, the display screen 505 can be a flexible display screen, configured on a curved surface or a folding surface of the electronic device 500. Even, the display screen 505 can also be configured in an irregular shape other than a rectangle, i.e., a notched screen. The display screen 505 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0177] The camera assembly 506 is configured to capture images or videos. Optionally, the camera assembly 506 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is configured on the front panel of the electronic device, and the rear-facing camera is configured on the back of the electronic device. In some embodiments, the rear-facing camera is at least two, respectively configured as any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function by fusing the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function by fusing the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 506 can further include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0178] The audio circuit 507 can include a microphone and a speaker. The microphone is used to collect sound waves of a user and an environment, and convert the sound waves into an electrical signal input to the processor 501 for processing, or input to the radio frequency circuit 504 to realize voice communication. The microphone can be multiple for the purpose of stereo sound collection or noise reduction, and arranged at different parts of the electronic device 500 respectively. The microphone can also be an array microphone or an omnidirectional collection type microphone. The speaker is used to convert an electrical signal from the processor 501 or the radio frequency circuit 504 into sound waves. The speaker can be a conventional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert an electrical signal into a sound wave audible to humans, but also convert an electrical signal into an inaudible sound wave to humans for ranging purposes, etc. In some embodiments, the audio circuit 507 can also include a headphone jack.

[0179] The power supply 508 is used to supply power to various components in the electronic device 500. The power supply 508 can be alternating current, direct current, disposable batteries or rechargeable batteries. When the power supply 508 includes rechargeable batteries, the rechargeable batteries can support wired charging or wireless charging. The rechargeable batteries can also be used to support fast charging technology.

[0180] Those skilled in the art can understand that the structure shown in the above description is not a limitation on the electronic device 500, and the electronic device 500 can include more or fewer components than those shown in the figure, or combine certain components, or use different component arrangements. Figure 5 The structure shown in the above description is not a limitation on the electronic device 500, and the electronic device 500 can include more or fewer components than those shown in the figure, or combine certain components, or use different component arrangements.

[0181] In an exemplary embodiment, a computer readable storage medium including instructions, such as the memory 502 including instructions, is also provided, which can be executed by the processor 501 of the electronic device 500 to complete the above data processing method. Alternatively, the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0182] In an exemplary embodiment, a computer program product is also provided, which includes program code that is executed by a processor to implement the above data processing method.

[0183] Those skilled in the art will readily understand other implementations of the present disclosure after considering the description and practice of the application disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptive changes of the present disclosure that follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the art that are not disclosed in the present disclosure. The specification and examples are only considered as exemplary, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0184] It should be understood that the present disclosure is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized by, The method comprises: obtaining a set of inference requests of a large language model, the set of inference requests comprising a plurality of word token sequences corresponding to the plurality of inference requests respectively, any word token sequence corresponding to an inference request being obtained by encoding the inference request; dividing the plurality of word token sequences in the set of inference requests into a plurality of sequence block groups respectively, each sequence block group comprising a plurality of sequence blocks, and a difference between lengths of the sequence blocks in each sequence block group being less than a preset difference threshold; splicing the sequence blocks in each sequence block group to obtain a matrix corresponding to each sequence block group; processing the matrices corresponding to the plurality of sequence block groups by using an attention kernel function to obtain attention features of the plurality of inference requests, the attention features being used in an inference process of the large language model.

2. The data processing method according to claim 1, characterized in that, The method comprises: for any word token sequence, dividing the word token sequence into a preset number of first sequence blocks corresponding to a length range to which the word token sequence belongs; adding the preset number of first sequence blocks in a sequence block group corresponding to the length range.

3. The data processing method according to claim 2, characterized in that, The method comprises: padding the preset number of first sequence blocks to obtain a preset number of second sequence blocks, the second sequence blocks having a length equal to a preset length corresponding to the length range; adding the preset number of second sequence blocks in the sequence block group corresponding to the length range.

4. The data processing method of claim 1, wherein, The method comprises: for any word token sequence, determining first sequence blocks and second sequence blocks in the word token sequence based on a length range to which the word token sequence belongs, the first sequence blocks having a length equal to a preset length corresponding to the length range, and the second sequence blocks comprising a remaining part of the word token sequence excluding the first sequence blocks; adding the first sequence blocks and the second sequence blocks in a sequence block group corresponding to the length range.

5. The data processing method according to claim 4, characterized in that, The method comprises: adding the first sequence blocks in the sequence block group corresponding to the length range; padding the second sequence blocks to obtain third sequence blocks, the third sequence blocks having a length equal to the preset length corresponding to the length range, and adding the third sequence blocks in the sequence block group corresponding to the length range.

6. The data processing method of claim 1, wherein, The method comprises: for any sequence block group, padding each sequence block in the sequence block group to obtain each intermediate sequence block in the sequence block group, the intermediate sequence blocks having a length equal to a maximum length of the sequence blocks in the sequence block group; splicing each intermediate sequence block in the sequence block group to obtain a matrix corresponding to the sequence block group.

7. The data processing method of claim 1, wherein, The attention kernel function is used to process the matrices corresponding to the plurality of sequence block groups to obtain attention features of the plurality of reasoning requests, including: obtaining a double-index mapping table, the double-index mapping table being used to indicate a relationship between a storage address of a sequence block and a position of the sequence block in a corresponding matrix; determining, based on the double-index mapping table, a sequence block corresponding to each thread in the matrices corresponding to the plurality of sequence block groups by using the attention kernel function; calling each thread to process the corresponding sequence block to obtain the attention features of the plurality of reasoning requests.

8. The data processing method according to claim 7, characterized in that, The double-index mapping table is used to indicate a mapping relationship between a storage address of a sequence block and a sequence block identifier, the sequence block identifier being an identifier of the sequence block in a corresponding sequence block group; The double-index mapping table is also used to indicate a mapping relationship between a mask bit of a sequence block and a matrix offset, the mask bit being used to indicate that a token in the sequence block is valid data or invalid data, and the matrix offset being used to indicate a position of the token in the sequence block in a corresponding matrix.

9. The data processing method according to claim 8, characterized in that, The double-index mapping table is also used to indicate a mapping relationship between a mask bit of a sequence block and a matrix offset, the mask bit being used to indicate that a token in the sequence block is valid data or invalid data, and the matrix offset being used to indicate a position of the token in the sequence block in a corresponding matrix. The double-index mapping table is also used to indicate a mapping relationship between a mask bit of a sequence block and a matrix offset, the mask bit being used to indicate that a token in the sequence block is valid data or invalid data, and the matrix offset being used to indicate a position of the token in the sequence block in a corresponding matrix. The double-index mapping table is also used to indicate a mapping relationship between a mask bit of a sequence block and a matrix offset, the mask bit being used to indicate that a token in the sequence block is valid data or invalid data, and the matrix offset being used to indicate a position of the token in the sequence block in a corresponding matrix.

10. The data processing method according to claim 7, characterized in that, The double-index mapping table is also used to indicate a mapping relationship between a mask bit of a sequence block and a matrix offset, the mask bit being used to indicate that a token in the sequence block is valid data or invalid data, and the matrix offset being used to indicate a position of the token in the sequence block in a corresponding matrix. The double-index mapping table is also used to indicate a mapping relationship between a mask bit of a sequence block and a matrix offset, the mask bit being used to indicate that a token in the sequence block is valid data or invalid data, and the matrix offset being used to indicate a position of the token in the sequence block in a corresponding matrix. The device includes:

11. A data processing apparatus, characterized by an obtaining unit configured to obtain a reasoning request set of a large language model, the reasoning request set including a plurality of token sequences corresponding to a plurality of reasoning requests respectively, and a token sequence corresponding to any reasoning request being obtained by encoding the reasoning request; a blocking unit configured to block the plurality of token sequences in the reasoning request set respectively to obtain a plurality of sequence block groups corresponding to the reasoning request set, each sequence block group including a plurality of sequence blocks, and a difference between lengths of the sequence blocks in each sequence block group being less than a preset difference threshold; a concatenating unit configured to concatenate the sequence blocks in each sequence block group to obtain a matrix corresponding to each sequence block group; a processing unit configured to process the matrices corresponding to the plurality of sequence block groups by using an attention kernel function to obtain attention features of the plurality of reasoning requests, the attention features being used in a reasoning process of the large language model. The electronic device includes:

12. An electronic device, comprising: one or more processors; a memory for storing program code executable by the processor; wherein the processor is configured to execute the program code to implement the data processing method of any one of claims 1 to 10. ​ 13. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the data processing method as claimed in any one of claims 1 to 10.

14. A computer program product comprising program code which, when executed by a processor, implements the data processing method as claimed in any one of claims 1 to 10.

Images