Data Processing Method, Apparatus, Device, and Storage Medium for Large Models

Abstract

The present application provides a data processing method, device, equipment and storage medium for a large model, wherein the method includes: generating an input feature matrix according to a prompt word input by a user; determining at least one thread block corresponding to the input feature matrix, the operation type of each thread block and the target basic computing unit to which each thread block belongs according to the processing type of each input feature sequence in the input feature matrix; allocating each input feature sequence to each thread block according to the operation type of each thread block and the processing type of each input feature sequence; running each thread block in parallel to obtain the operation result corresponding to the input feature matrix; and obtaining the output result of the large model based on the operation result corresponding to each input feature matrix. The present application can realize dynamic scheduling of resource allocation at the kernel function level of the GPU, so that an SM unit can process pre-filling and decoding tasks at the same time, avoiding idleness or repeated calculations, thereby significantly improving the inference performance of the large model.

Description

Technical Field

[0001] The present application relates to the technical field of large-scale model reasoning optimization, and more specifically, to a data processing method, apparatus, device, and storage medium for a large-scale model. Background Art

[0002] In large-scale model inference, each request undergoes two processing stages: prefill and decoding. Prefill calculates attention and generates a key-value cache, while decoding generates output through autoregression. Prefill is computationally intensive, while decoding is memory intensive. Efficiently scheduling these two stages to maximize GPU utilization is a key challenge.

[0003] The current common large-model inference architecture uses Chunked Prefill technology based on high-bandwidth memory. It splits prompts of varying lengths into blocks of consistent lengths (chunks) for prefilling, and then inserts the decoding requirements of other prefilled prompts.

[0004] However, the attention module used in technologies like Chunked Prefill can only be used for separate prefill or decode calculations. When executing, the GPU still needs to process them sequentially and must wait until the current chunk is completely prefilled before decoding can begin. This makes it impossible to achieve true parallelism and thus performance improvement. Summary of the Invention

[0005] The purpose of this application is to provide a large-scale model data processing method, device, equipment and storage medium to address the shortcomings of the above-mentioned existing technologies, so as to solve the performance limitations of the existing technologies.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for processing data of a large model, the method comprising:

[0008] Generating at least one input feature matrix of the large model according to the prompt word input by the user, wherein each input feature matrix includes at least one input feature sequence;

[0009] Determining, according to a processing type of each input feature sequence in the input feature matrix, at least one thread block corresponding to the input feature matrix, an operation type of each thread block, and a target basic computing unit to which each thread block belongs, wherein the processing type includes: a pre-filling processing type or a decoding processing type, and the operation type includes: a pre-filling operation type or a decoding operation type;

[0010] Allocate each input feature sequence to each thread block according to the operation type of each thread block and the processing type of each input feature sequence;

[0011] Running each thread block in parallel to obtain a calculation result corresponding to the input feature matrix;

[0012] Based on the calculation results corresponding to each of the input feature matrices, the output result of the large model is obtained.

[0013] In a second aspect, another embodiment of the present application provides a large model data processing device, the device comprising:

[0014] A generation module, configured to generate at least one input feature matrix of the large model according to the prompt word input by the user, wherein each input feature matrix includes at least one input feature sequence;

[0015] a determination module, configured to determine, based on a processing type of each input feature sequence in the input feature matrix, at least one thread block corresponding to the input feature matrix, an operation type of each thread block, and a target basic computing unit to which each thread block belongs, wherein the processing type includes a pre-filling processing type or a decoding processing type, and the operation type includes a pre-filling operation type or a decoding operation type;

[0016] an allocation module, configured to allocate each input feature sequence to each thread block according to the operation type of each thread block and the processing type of each input feature sequence;

[0017] An operation module is used to run each thread block in parallel to obtain a calculation result corresponding to the input feature matrix;

[0018] The output module is used to obtain the output result of the large model based on the operation results corresponding to each of the input feature matrices.

[0019] In the third aspect, another embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of any method described in the first aspect above.

[0020] In a fourth aspect, another embodiment of the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any method described in the first aspect are executed.

[0021] The beneficial effects of the present application are as follows: generating at least one input feature matrix of a large model through a prompt word input by a user; and determining at least one thread block corresponding to the input feature matrix, the operation type of each thread block, and the target basic computing unit to which each thread block belongs according to the processing type of each input feature sequence in the input feature matrix; assigning each input feature sequence to each thread block according to the operation type of each thread block and the processing type of each input feature sequence; running each thread block in parallel to obtain the operation result corresponding to the input feature matrix; and obtaining the output result of the large model based on the operation result corresponding to each input feature matrix, which can realize dynamic scheduling of resource allocation at the kernel function level of the GPU, allowing the same SM unit to process pre-filling and decoding tasks at the same time, so that input feature sequences with different processing types in the input feature matrix can be executed in parallel, reducing waiting time, improving overall throughput, ensuring that graphics processor resources are fully utilized, avoiding idle or repeated calculations, and thus significantly improving the reasoning performance of the large model. In addition, it can better adapt to inputs of different lengths, avoid additional padding operations caused by non-divisible sequence lengths, give full play to the capabilities of graphics processor hardware, and enhance adaptability to different application scenarios. At the same time, it can also reduce the number of global memory accesses and achieve efficient memory access. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 This is a schematic diagram of a Chunked prefill technology in the prior art;

[0024] Figure 2 A schematic diagram of the architecture of a graphics processor provided in an embodiment of the present application;

[0025] Figure 3 A schematic structural diagram of a large model provided in an embodiment of the present application;

[0026] Figure 4 A flow chart of a data processing method for a large model provided in an embodiment of the present application;

[0027] Figure 5A schematic diagram of a data processing method for a large model provided in an embodiment of the present application;

[0028] Figure 6 A schematic flow chart of determining at least one thread block corresponding to an input feature matrix, the operation type of each thread block, and the target basic computing unit to which each thread block belongs in the data processing method for a large model provided in an embodiment of the present application;

[0029] Figure 7 A schematic flow chart of determining the number of first thread blocks whose operation type is a pre-filling operation type and the number of second thread blocks whose operation type is a decoding operation type in the data processing method of a large model provided in an embodiment of the present application;

[0030] Figure 8 Another flowchart diagram of determining at least one thread block corresponding to an input feature matrix, the operation type of each thread block, and the target basic computing unit to which each thread block belongs in the data processing method for a large model provided in an embodiment of the present application;

[0031] Figure 9 A schematic diagram of a flow chart for determining the target basic computing unit to which each thread block belongs and the operation type of each thread block in the data processing method for a large model provided in an embodiment of the present application;

[0032] Figure 10 A schematic diagram of a large-scale data processing device provided in an embodiment of the present application;

[0033] Figure 11 A schematic diagram of the electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0035] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0036] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0037] The current common large-model inference architecture uses chunked prefill technology based on high-bandwidth memory. This technology splits prompts of varying lengths into chunks of consistent length for prefilling (prefill), and then inserts decoding requirements for other prefilled prompts. Prefill refers to the process by which the model sees input text and generates intermediate results (such as key and value caches), while decoding refers to the process by which the model gradually generates output based on previously generated results.

[0038] However, although the attention module used in technologies such as Chunked Prefill can divide inputs of different lengths into small chunks to optimize calculations, it can only be used for separate prefill or decode calculations. When executing, the GPU still needs to wait until the current chunk is completely prefilled before decoding can begin. In other words, prefilling must be completed before decoding can begin, which makes true parallelism impossible. When the context length increases, such as in applications such as RAG or Agent, it is difficult to achieve performance improvements.

[0039] For example, Figure 1 This is a schematic diagram of a Chunked prefill technology in the prior art, refer to Figure 1 As shown in the figure, Chunked prefill cannot simultaneously calculate the prefill chunk and decode in one SM unit. Since the length of the input feature sequence seq is usually not divisible by the total number of threads, the remaining seq must be re-filled (padding) to participate in a new operation.

[0040] For example, consider a GPU with two Streaming Multiprocessors (SMs), each capable of executing up to two threads. Given a sequence seq with a prefill length of 5 and a decode length of 3, the chunked prefill process requires three passes to produce the final result.

[0041] This means that, in existing techniques, if the input sequence length is not divisible by the number of threads, the remaining portion needs to be padded, resulting in additional computational overhead. Furthermore, if the input text lengths are uneven (for example, some prompts are very long, while others are very short), GPU resources can be unevenly allocated, with some computing units on the GPU sitting idle. Furthermore, during the decoding phase, the model needs to frequently load data from the GPU's global memory, increasing latency.

[0042] Based on the above problems, the embodiment of the present application proposes a data processing method for a large model, which generates at least one input feature matrix of the large model through a prompt word input by the user; and according to the processing type of each input feature sequence in the input feature matrix, determines at least one thread block corresponding to the input feature matrix, the operation type of each thread block, and the target basic computing unit to which each thread block belongs; according to the operation type of each thread block and the processing type of each input feature sequence, assigns each input feature sequence to each thread block; runs each thread block in parallel to obtain the operation result corresponding to the input feature matrix; based on the operation result corresponding to each input feature matrix, obtains the output result of the large model, so that input feature sequences with different processing types in the input feature matrix can be executed in parallel, reducing waiting time, improving overall throughput, ensuring that graphics processor resources are fully utilized, avoiding idle or repeated calculations, and thus significantly improving the reasoning performance of the large model. In addition, it can better adapt to inputs of different lengths, avoid additional padding operations caused by non-divisible sequence lengths, and give full play to the capabilities of graphics processor hardware.

[0043] First, the relevant architecture involved in the data processing method of the large model provided in the embodiment of the present application is explained.

[0044] Figure 2 A schematic diagram of the architecture of a graphics processor provided in an embodiment of the present application, referring to Figure 2 As shown, the graphics processor provided in the embodiment of the present application includes multiple basic computing units (Streaming Multiprocessor, referred to as SM), global memory (referred to as Global mem) and a second-level cache (referred to as L2 cache).

[0045] Global memory is the GPU's main memory, accessible to all warps and thread blocks. The L2 cache stores copies of data recently accessed from global memory and prefetches data that will be accessed soon.

[0046] Among them, a basic computing unit serves as an independent computing module, and each basic computing unit includes multiple integer and floating-point operation units (CUDA cores for short), multiple matrix operation units (Tensor cores for short), L1 cache and on-chip shared memory.

[0047] Among them, multiple thread blocks can be executed simultaneously on a basic computing unit. For example, the basic computing unit can dynamically pull thread blocks from the GPU global scheduling queue until the basic computing unit reaches the thread block capacity limit or lacks resources.

[0048] Among them, a thread block includes multiple warps. Warps in the same thread block can share data and collaborate to complete tasks.

[0049] For example, a warp typically contains 32 threads, which is the basic granularity of GPU scheduling. The GPU scheduler schedules the warp to the CUDA core of the SM for execution.

[0050] Illustratively, each thread has independent registers for storing local variables and intermediate results.

[0051] For example, during the calculation process of large model inference, data is first loaded from global memory to shared memory or L1 cache, then transferred to registers, and then calculated using CUDA cores or Tensor cores to obtain calculation results.

[0052] For example, the parameters of a large model (e.g., weight matrix) are typically stored in global memory. During the inference process of a large model in the data processing method for a large model provided in this application, some weights can be loaded into L2 cache or shared memory using tile technology to reduce the overhead of frequent accesses to global memory.

[0053] For example, the KV cache (Key-Value Cache) is the core data structure of the autoregressive decoding stage, used to store historical context information of the generated sequence. The contents of the KV cache can be first loaded from global memory into the L2 cache and then further migrated to shared memory or registers to accelerate the calculation of the attention mechanism.

[0054] For example, the intermediate results of the attention mechanism (such as the calculation results of Query, Key, and Value) can be stored in shared memory for sharing by threads in the same thread block.

[0055] For example, the data of the input sequence and the output sequence are stored in the global memory and loaded into the shared memory or registers on demand during the inference process.

[0056] It can be understood that in the pre-filling stage of large-model reasoning, the SM unit is responsible for calculating the Query, Key, and Value of the input sequence and generating the initial KV cache. In the decoding stage, the SM unit generates the output sequence word by word, calculates the Query of one time step each time, and interacts with the historical KV cache. During the large-model reasoning process, by executing the data processing method of the large model provided in the embodiment of the present application, the GPU can simultaneously perform pre-filling and decoding calculations in a single SM unit, which not only improves the utilization of hardware resources, but also reduces redundant calculations and memory access delays, thereby significantly improving the efficiency of large language model reasoning.

[0057] Optionally, the large model may be a large model based on the Transformer architecture. The structure of the large model involved in the data processing method of the large model provided in the embodiment of the present application is exemplarily described below.

[0058] Figure 3 A schematic diagram of the structure of the large model provided in the embodiment of the present application, referring to Figure 3 As shown in the figure, the large model includes a text embedding layer (Text Embedding for short) and N processing layers connected in sequence, and obtains the final text prediction result (Text Prediction), where N is a positive integer.

[0059] Among them, Text Embedding is the input layer of the model, which converts the input text (such as words or tokens) into a high-dimensional vector representation. Each word is mapped to a vector of fixed dimension. Specifically, the positional encoding can be used to preserve the position information of the word.

[0060] Among them, each processing layer includes two normalization layers (Layer Norm for short), a pre-projection layer (Preprojection for short), a pre-fill attention mechanism (Prefill Attention for short), a decoding attention mechanism (decodeAttention for short), and a feedforward network (Feed Forward for short).

[0061] For example, the input text is first converted into an embedding vector, normalized and pre-projected by the normalization layer and the pre-projection layer, and then enters the attention mechanism (pre-filling or decoding). After the attention output, it passes through the feedforward network and is repeatedly processed in multiple layers. Finally, the probability distribution of the predicted next token is output to obtain the final text prediction result (Text Prediction).

[0062] The following describes in detail the data processing method of the large model provided in the embodiments of the present application in combination with multiple embodiments.

[0063] Figure 4 A flow chart of a data processing method for a large model provided in an embodiment of the present application, referring to Figure 4 As shown, the execution subject of the method can be any electronic device with processing capabilities, and the method includes:

[0064] S401. Generate at least one input feature matrix of a large model according to a prompt word input by a user.

[0065] It is understood that the large model can generate natural language answers based on the prompt words input by the user. Figure 3 As shown in the figure, the input of each processing layer of the large model will be dynamically updated as the processing progresses. The normalization layer and pre-projection layer of each processing layer can process the input of the processing layer into an input feature matrix for the operation of the attention mechanism.

[0066] Optionally, at the beginning of the inference process, the large model can first input the prompt word entered by the user into the text embedding layer to convert the prompt word entered by the user into a token sequence that the large model can understand, and add special tags to the token sequence to identify the task type or sequence boundary. At the same time, the token sequence is truncated to the maximum supported length of the model, or the token sequence is padded (Padding) to a fixed length and a special Padding tag is used to represent the padding part.

[0067] Optionally, after obtaining the Token sequence, the Token sequence can be input into the first processing layer, and the normalization layer and pre-projection layer of the first processing layer convert the token sequence into a vector and perform position encoding to obtain the input feature matrix of the first processing layer.

[0068] Alternatively, the input feature matrix of the first processing layer can be fed into the attention mechanism of the first processing layer of the large model for processing. The results of the attention mechanism operation are fed forward and then passed layer by layer to obtain the input feature matrices of other processing layers of the large model. The number of input feature matrices of the large model can be the same as the number of processing layers of the large model.

[0069] That is to say, referring to Figure 3 As shown, the input feature matrix can be the output of the pre-projection layer of a processing layer, that is, the set of inputs of the pre-filled attention mechanism or the decoding attention mechanism.

[0070] For example, taking any input feature matrix as an example, the input feature matrix can be obtained by normalizing and pre-projecting the input of the corresponding processing layer, and the input of the corresponding processing layer can be obtained by performing residual connection on the output of the previous processing layer.

[0071] Each input feature matrix includes at least one input feature sequence.

[0072] Exemplarily, the matrix shape of the input feature matrix can be expressed as: [sequence length, feature dimension]. An input feature matrix can be, for example, a sentence, and the multiple input feature sequences in the input feature matrix can be, for example, vectors of multiple tokens in the sentence.

[0073] S402: Determine, according to the processing type of each input feature sequence in the input feature matrix, at least one thread block corresponding to the input feature matrix, the operation type of each thread block, and the target basic computing unit to which each thread block belongs.

[0074] It can be understood that, taking the reasoning process of any processing layer in the large model reasoning process as an example, during the reasoning process of the processing layer, the input feature sequence in the input feature matrix can perform the operation of the pre-filling attention mechanism in the processing layer, or the operation of the decoding attention mechanism.

[0075] In this process, we can first determine the number of thread blocks required for each input feature sequence in the input feature matrix to perform the attention mechanism operation, the operation type of each thread block, and the target basic computing unit to which each thread block belongs during execution, so that each input feature sequence can be executed in parallel in each target computing unit.

[0076] The processing type includes: a pre-filling processing type or a decoding processing type, and the operation type includes: a pre-filling operation type or a decoding operation type.

[0077] Exemplarily, the pre-filling processing type means that the input feature sequence needs to be processed by the pre-filling attention mechanism, the pre-filling operation type means that the pre-filling attention mechanism is operated, the decoding processing type means that the input feature sequence needs to be processed by the decoding attention mechanism, and the decoding operation type means that the decoding attention mechanism is operated.

[0078] For example, depending on the processing type of the input feature sequence, it can be determined whether the input feature sequence needs to perform the operation of the pre-filling attention mechanism or the decoding attention mechanism, thereby determining the number and operation type of each thread block that performs the operation of the attention mechanism, and dynamically determining the target basic computing unit to which each thread block belongs through the GPU hardware scheduler.

[0079] For example, Figure 5 A schematic diagram of a data processing method for a large model provided in an embodiment of the present application, referring to Figure 5 As shown, taking the inference process of any processing layer in the large model inference process as an example, in the inference process of this processing layer, assuming that the input feature matrix is an 8*12 matrix, and the processing type of the first 5 input feature sequences in the input feature matrix is a pre-filling processing type, and the processing type of the last 3 input feature sequences is a decoding processing type, by executing this step, it can be determined that 8 thread blocks are required to execute the operation of the attention mechanism of the input feature matrix, and it is determined that the operation type of 5 thread blocks among the 8 thread blocks is the pre-filling processing type, and the operation type of 3 thread blocks is the decoding operation type, and it can be determined that the target basic computing units to which these 8 thread blocks belong are SM0 and SM1 respectively.

[0080] Optionally, the above S402 can be deployed in each processing layer of the large model in the form of a kernel function during specific implementation. For example, it can be deployed in the form of a kernel function after the pre-projection layer of each processing layer, or it can be deployed at the end of the pre-projection layer of each processing layer.

[0081] For example, when the attention mechanism is Multi-Head Self-Attention (MHA), the input data can be divided into multiple "heads" for independent calculation and then spliced together. That is, when calculating the blocks, it is divided by a single head and finally multiplied by the number of heads (num_head).

[0082] S403 : Allocate each input feature sequence to each thread block according to the operation type of each thread block and the processing type of each input feature sequence.

[0083] Optionally, continue with reference to Figure 5As shown, after determining each thread block and the operation type of each thread block, each input feature sequence input can be matched according to the processing type of each input feature sequence and the operation type of each thread block to assign each input feature sequence to the corresponding thread block.

[0084] S404: Run each thread block in parallel to obtain a calculation result corresponding to the input feature matrix.

[0085] Optionally, after each input feature sequence is assigned to each thread block, each thread block can be run in parallel so that each input feature sequence can perform the operation of the pre-filling attention mechanism or the operation of the decoding attention mechanism in each target basic computing unit, simultaneously and separately in its own thread block, thereby saving the time required for repeated calculations due to the inability to divide the length of the input feature sequence evenly, and improving the computing efficiency.

[0086] For example, flashattention can be called to calculate the operation result corresponding to the input feature matrix.

[0087] Among them, the operation result corresponding to the input feature matrix can be understood as the set of sub-operation results obtained after the operation of the pre-filled attention mechanism is performed in parallel by each input feature sequence in the input feature matrix during the inference process of the processing layer, or after the operation of the decoding attention mechanism.

[0088] S405. Based on the calculation results corresponding to each input feature matrix, the output result of the large model is obtained.

[0089] Optionally, taking the inference process of any processing layer in the large model inference process as an example, after obtaining the operation results corresponding to the input feature matrix of the processing layer, the subsequent pre-projection processing, normalization and feedforward processing can be continued on the operation results corresponding to the input feature matrix in the processing layer, and the output result of the processing layer can be obtained. Based on the output result of the processing layer, combined with the subsequent processing layers of the processing layer, the operation results corresponding to each input feature matrix are obtained, thereby obtaining the output result of the large model.

[0090] In this embodiment, at least one input feature matrix of a large model is generated based on a prompt word input by the user; and based on the processing type of each input feature sequence in the input feature matrix, at least one thread block corresponding to the input feature matrix, the operation type of each thread block, and the target basic computing unit to which each thread block belongs are determined; based on the operation type of each thread block and the processing type of each input feature sequence, each input feature sequence is assigned to each thread block; each thread block is run in parallel to obtain the operation result corresponding to the input feature matrix; based on the operation result corresponding to each input feature matrix, the output result of the large model is obtained, which can realize dynamic scheduling of resource allocation at the kernel function level of the GPU, allowing the same SM unit to process pre-filling and decoding tasks at the same time, so that input feature sequences with different processing types in the input feature matrix can be executed in parallel, reducing waiting time, improving overall throughput, ensuring that GPU resources are fully utilized, avoiding idle or repeated calculations, and thus significantly improving the reasoning performance of the large model. In addition, it can better adapt to inputs of different lengths, avoid additional padding operations caused by non-divisible sequence lengths, give full play to the capabilities of the GPU hardware, and enhance adaptability to different application scenarios. At the same time, it can also reduce the number of global memory accesses and achieve efficient memory access.

[0091] In one possible implementation, Figure 6 A flow chart of determining at least one thread block corresponding to an input feature matrix, the operation type of each thread block, and the target basic computing unit to which each thread block belongs in the data processing method of the large model provided in the embodiment of the present application, with reference to Figure 6 As shown, the above S402 includes:

[0092] S601: Determine, according to the processing type of each input feature sequence in the input feature matrix, the number of first thread blocks whose operation type is a pre-filling operation type and the number of second thread blocks whose operation type is a decoding operation type.

[0093] Optionally, the number of first thread blocks whose operation type is the pre-filling operation type and the number of second thread blocks whose operation type is the decoding type may be calculated according to the processing type of each input feature sequence in the input feature matrix.

[0094] Exemplarily, the number of input feature sequences whose processing type is the pre-filled processing type in the input feature matrix can be determined, and based on the number of input feature sequences whose processing type is the pre-filled processing type and the preset number of threads of each thread block, the number of first thread blocks whose operation type is the pre-filled operation type can be determined.

[0095] Exemplarily, the number of input feature sequences whose processing type is decoding processing type in the input feature matrix can be determined, and based on the number of input feature sequences whose processing type is decoding processing type and the preset number of threads of each thread block, the number of second thread blocks whose operation type is decoding type can be determined.

[0096] For example, continue to refer to Figure 5 As shown, taking the inference process of any processing layer in the large model inference process as an example, in the inference process of this processing layer, assuming that the input feature matrix is an 8*12 matrix, and the processing type of the first 5 input feature sequences in the input feature matrix is a pre-filling processing type, and the processing type of the last 3 input feature sequences is a decoding processing type, by executing this step, it can be determined that the number of first thread blocks whose operation type is a pre-filling operation type is 5, and the number of second thread blocks whose operation type is a decoding type is 3.

[0097] S602: Determine, based on the first number of thread blocks and the second number of thread blocks, at least one thread block corresponding to the input feature matrix, an operation type of each thread block, and a target basic computing unit to which each thread block belongs.

[0098] Optionally, after determining the number of first thread blocks and the number of second thread blocks, the thread blocks required for the input feature matrix during the attention mechanism operation can be determined based on the number of first thread blocks and the number of second thread blocks, that is, at least one thread block corresponding to the input feature matrix, and the operation type of each thread block can be determined.

[0099] Optionally, after the thread blocks and the operation types of the thread blocks are determined, the target basic computing unit to which each thread block belongs may be dynamically determined by the GPU hardware scheduler.

[0100] For example, continue to refer to Figure 5 As shown, at this time, the processing type of the first 5 input feature sequences in the input feature matrix is the pre-filling processing type, and the processing type of the last 3 input feature sequences is the decoding processing type. Then, it can be determined that at least one thread block corresponding to the input feature matrix is 5 thread blocks whose operation type is the pre-filling operation type and 3 thread blocks whose operation type is the decoding operation type, and the target basic computing units to which each thread block belongs can be dynamically determined as SM0 and SM1 through the GPU hardware scheduler.

[0101] By inputting the processing type of each input feature sequence in the feature matrix, the number of first thread blocks with the pre-filling operation type and the number of second thread blocks with the decoding operation type are determined. Based on the number of first thread blocks and the number of second thread blocks, at least one thread block corresponding to the input feature matrix, the operation type of each thread block, and the target basic computing unit to which each thread block belongs are determined. This can improve the utilization rate of each basic computing unit in the GPU and reduce the resource waste caused by sequence filling, thereby significantly improving the inference performance of large models.

[0102] In one possible implementation, Figure 7 A flow chart of determining the number of first thread blocks of pre-filled operation type and the number of second thread blocks of decoding operation type in the data processing method of the large model provided in the embodiment of the present application, with reference to Figure 7 As shown, in the above S601, determining the number of first thread blocks whose operation type is a pre-filling operation type and the number of second thread blocks whose operation type is a decoding type according to the processing type of each input feature sequence in the input feature matrix includes:

[0103] S701: Determine the number of input feature sequences with a processing type of pre-filling type in an input feature matrix, and calculate the number of first thread blocks with an operation type of pre-filling type according to the number of input feature sequences with a processing type of pre-filling type.

[0104] Optionally, the number of input feature sequences whose processing type is the pre-filled type in the input feature matrix can be determined first, and the number of first thread blocks whose operation type is the pre-filled operation type can be calculated based on the number of input feature sequences whose processing type is the pre-filled type, the preset number of threads of each thread block, and the number of heads in the multi-head attention mechanism.

[0105] For example, the number of first thread blocks prefill_blocks whose operation type is the prefill operation type may be calculated with reference to the following formula (1):

[0106] (1)

[0107] Among them, prefill_blocks is the number of first thread blocks whose operation type is prefill operation type, is the number of heads in the multi-head attention mechanism, prefill_blockM is the number of threads in the thread block whose operation type is the prefill operation type, and prefill_seqlen is the number of input feature sequences whose processing type is the prefill type.

[0108] S702: Determine the number of input feature sequences with a decoding processing type in the input feature matrix, and calculate the number of second thread blocks with a decoding operation type according to the number of input feature sequences with a decoding processing type.

[0109] Optionally, the number of input feature sequences whose processing type is decoding type in the input feature matrix can be determined first, and the number of second thread blocks whose operation type is decoding operation type can be calculated based on the number of input feature sequences whose processing type is decoding type, the preset number of threads of each thread block, and the number of heads in the multi-head attention mechanism.

[0110] For example, the number of second thread blocks decode_blocks whose operation type is decoding operation type can be calculated by referring to the following formula (2):

[0111] (2)

[0112] Wherein, decode_blocks is the number of second thread blocks whose operation type is decoding operation type, is the number of heads in the multi-head attention mechanism, decode_blockM is the number of threads in the thread block whose operation type is the decoding operation type, and decode_seqlen is the number of input feature sequences whose processing type is the decoding type.

[0113] In one possible implementation, Figure 8 Another flow chart for determining at least one thread block corresponding to an input feature matrix, the operation type of each thread block, and the target basic computing unit to which each thread block belongs in the data processing method of the large model provided in the embodiment of the present application, with reference to Figure 8 As shown, the above S602 includes:

[0114] S801: Determine, based on the first number of thread blocks and the second number of thread blocks, at least one thread block corresponding to an input feature matrix and a ratio of pre-filled processing types to decoding processing types in the input feature matrix.

[0115] Optionally, at least one thread block corresponding to the input feature matrix may be determined according to the first number of thread blocks and the second number of thread blocks.

[0116] In one example, continue with reference to Figure 5As shown, taking the inference process of any processing layer in the large model inference process as an example, in the inference process of the processing layer, through the above S601, it can be determined that the number of first thread blocks whose operation type is the pre-filling operation type is 5, and the number of second thread blocks whose operation type is the decoding type is 3, then the number of first thread blocks and the number of second thread blocks are summed, and it is determined that the number of thread blocks corresponding to the input feature matrix is 8, and 8 thread blocks can be obtained by starting the kernel (Kernel) and specifying the dimension of the grid (Grid), that is, the number of first thread blocks and the number of second thread blocks can be summed, and the thread blocks corresponding to the input feature matrix can be obtained through the kernel startup configuration.

[0117] In another example, taking the inference process of any processing layer in the large model inference process as an example, in the inference process of the processing layer, assuming that it is determined that the number of first thread blocks of the pre-filling operation type is 10, and the number of second thread blocks of the decoding type is 6, then the number of first thread blocks and the number of second thread blocks can be summed to determine that the number of thread blocks corresponding to the input feature matrix is 16, and 16 thread blocks can be obtained by starting the kernel and specifying the dimension of the grid.

[0118] Optionally, the Euclidean algorithm may be used to obtain the greatest common divisor of the number of first thread blocks and the number of second thread blocks, and the ratio of the prefill processing type to the decoding processing type in the input feature matrix (prefill_blocks_min:decode_blocks_min) may be obtained by performing the simplest decomposition.

[0119] S802: Calculate an indicator value corresponding to the input feature matrix according to the ratio of the pre-filling processing type to the decoding processing type in the input feature matrix.

[0120] The indicator value is used to indicate the sum of two values in the ratio values of the two operation types in the input feature matrix.

[0121] Optionally, the minimum value prefill_blocks_min of the prefill processing type and the minimum value decode_blocks_min of the decoding processing type can be determined according to the ratio of the prefill processing type to the decoding processing type in the input feature matrix, and the indicator value corresponding to the input feature matrix can be calculated.

[0122] Exemplarily, the sum of the minimum value prefill_blocks_min of the pre-filling processing type and the minimum value decode_blocks_min of the decoding processing type may be calculated as the indicator value tag corresponding to the input feature matrix.

[0123] S803: Determine the target basic computing unit to which each thread block belongs and the operation type of each thread block according to the indication value corresponding to the input feature matrix.

[0124] Optionally, after obtaining the indication value corresponding to the input feature matrix, the target basic computing unit to which each thread block belongs and the operation type of each thread block can be determined by combining the indication value tag corresponding to the input feature matrix with the basic computing unit allocated by the GPU hardware scheduler.

[0125] In one possible implementation, Figure 9 A flow chart of determining the target basic computing unit to which each thread block belongs and the operation type of each thread block in the data processing method of the large model provided in the embodiment of the present application, with reference to Figure 9 As shown, the above S803 includes:

[0126] S901: traverse each thread block, and for the traversed current thread block, use the current basic computing unit allocated to it as the target basic computing unit to which the current thread block belongs.

[0127] It can be understood that after obtaining multiple thread blocks, the target basic computing unit and operation type to which each thread block belongs can be determined in turn.

[0128] Optionally, for the multiple thread blocks obtained by traversal, the current basic computing unit allocated to the current thread block is used as the target basic computing unit to which the current thread block belongs.

[0129] Among them, the current basic computing unit allocated can be obtained through the GPU hardware scheduler. That is to say, the process of allocating thread blocks to basic computing units can dynamically allocate thread blocks to basic computing units through the GPU hardware scheduler, thereby improving operating efficiency and avoiding resource waste.

[0130] S902: Obtain the number of executed thread blocks of the current basic computing unit, and determine the operation type of the current thread block according to the indication value corresponding to the input feature matrix and the number of executed thread blocks.

[0131] Optionally, after determining the current basic computing unit to which the current thread block belongs, the number of times count of executed thread blocks of the current basic computing unit may be obtained.

[0132] The number of executed thread blocks is used to indicate how many times the task has been executed on the current basic computing unit.

[0133] Optionally, the operation type of the current thread block is determined according to the indication value corresponding to the input feature matrix, the number of times the thread block has been executed, and a preset threshold.

[0134] Exemplarily, a modulo operation is performed on the number of executed thread blocks count and the indication value tag, and the result of the modulo operation op is compared with the minimum value prefill_blocks_min of the pre-fill processing type. If the result of the modulo operation op is less than the minimum value prefill_blocks_min of the pre-fill processing type, it is determined that the operation type of the current thread block is the pre-fill operation type. If the result of the modulo operation is greater than the minimum value prefill_blocks_min of the pre-fill processing type, it is determined that the operation type of the current thread block is the decoding operation type.

[0135] For example, it can be implemented by referring to the following formula (3):

[0136] (3)

[0137] in, is the result of the modulo operation, is the number of thread blocks executed, Indicates the value.

[0138] By inputting the indicator value corresponding to the feature matrix and the number of executed thread blocks, the operation type of the current thread block is determined, so that the operation type of the thread block can be performed in a specified proportion, ensuring that the task distribution on each basic computing unit meets the expectations and achieving dynamic balance of tasks. At the same time, it can fully utilize each basic computing unit, improve hardware utilization and resource utilization, reduce memory access overhead, and help to quickly obtain calculation results in efficient parallel computing scenarios.

[0139] In a possible implementation, the above S403 allocates each input feature sequence to each thread block according to the operation type of each thread block and the processing type of each input feature sequence, including:

[0140] The first input feature sequence of the pre-filled processing type and the second input feature sequence of the decoding processing type in the input feature matrix are traversed respectively.

[0141] Optionally, the first input feature sequence of the pre-filling processing type and the second input feature sequence of the decoding processing type in the input feature matrix may be traversed separately.

[0142] For the current first input feature sequence traversed, determine whether there is a thread block of pre-filled operation type in the current target basic computing unit. If so, assign the current first input feature sequence to the thread block of the pre-filled operation type. Otherwise, take the next target basic computing unit of the current target basic computing unit as the new current target basic computing unit and perform thread block allocation.

[0143] It can be understood that after determining at least one thread block corresponding to the input feature matrix, the operation type of each thread block, and the target basic computing unit to which each thread block belongs, the first input feature sequence of the pre-filled processing type in the input feature matrix can be assigned to the thread block whose operation type is the pre-filled operation type by traversal.

[0144] Optionally, for the current first input feature sequence traversed, check whether there is a thread block of pre-filled operation type in the current target basic computing unit. If so, assign the current first input feature sequence to the thread block of the pre-filled operation type; otherwise, take the next target basic computing unit of the current target basic computing unit as the new current target basic computing unit and perform thread block allocation.

[0145] The next target basic computing unit of the current target basic computing unit may be understood as any target basic computing unit except the current target basic computing unit.

[0146] For the current second input feature sequence traversed, determine whether there is a thread block of the decoding operation type in the current target basic computing unit. If so, assign the current second input feature sequence to the thread block of the decoding operation type. Otherwise, use the next target basic computing unit of the current target basic computing unit as the new current target basic computing unit and perform thread block allocation.

[0147] It can be understood that after determining at least one thread block corresponding to the input feature matrix, the operation type of each thread block, and the target basic computing unit to which each thread block belongs, the second input feature sequence of the decoding processing type in the input feature matrix can be assigned to the thread block with the operation type of the decoding operation type by traversal.

[0148] Optionally, for the current second input feature sequence traversed, check whether there is a thread block of the decoding operation type in the current target basic computing unit. If so, assign the current second input feature sequence to the thread block of the decoding operation type; otherwise, use the next target basic computing unit of the current target basic computing unit as the new current target basic computing unit and perform thread block allocation.

[0149] The next target basic computing unit of the current target basic computing unit may be understood as any target basic computing unit except the current target basic computing unit.

[0150] In a possible implementation, the above S404 runs each thread block in parallel to obtain the operation result corresponding to the input feature matrix, including:

[0151] When the current thread block is running, the model weight information is read from the shared storage space corresponding to the target basic computing unit to which the current thread block belongs, and calculations are performed based on the model weight information to obtain the calculation results of the input feature sequence corresponding to the current thread block.

[0152] Optionally, when the current thread block is running, the model weight information can be read from the shared storage space corresponding to the target basic computing unit to which the current thread block belongs, and calculations can be performed based on the model weight information to obtain the calculation results of the input feature sequence corresponding to the current thread block.

[0153] The model weight information includes: pre-filled weight values and decoded weight values.

[0154] Among them, reference Figure 2 As shown in FIG, the shared storage space corresponding to the target basic computing unit can be understood as the L1 cache and on-chip shared memory in the SM unit.

[0155] For example, before the inference process of a large model begins, the frequently accessed model weights can be copied from the global memory to the on-chip shared memory of the SM, so that the thread blocks running in the target basic computing unit can read the model weight information from the on-chip shared memory and perform matrix operations, thereby reducing the shared memory access latency and improving the performance of attention calculations.

[0156] Based on the same inventive concept, the embodiments of the present application also provide a data processing device for a large model corresponding to the data processing method for a large model. Since the principle of solving the problem by the device in the embodiments of the present application is similar to the data processing method for a large model mentioned above in the embodiments of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0157] Reference Figure 10 As shown, Figure 10 A schematic diagram of a data processing device for a large model provided in an embodiment of the present application, the device comprising: a generation module 1001, a determination module 1002, an allocation module 1003, an operation module 1004, and an output module 1005;

[0158] A generation module 1001 is configured to generate at least one input feature matrix of a large model based on a prompt word input by a user, wherein each input feature matrix includes at least one input feature sequence;

[0159] A determination module 1002 is configured to determine, based on a processing type of each input feature sequence in the input feature matrix, at least one thread block corresponding to the input feature matrix, an operation type of each thread block, and a target basic computing unit to which each thread block belongs, wherein the processing type includes a pre-filling processing type or a decoding processing type, and the operation type includes a pre-filling operation type or a decoding operation type;

[0160] An allocation module 1003 is configured to allocate each input feature sequence to each thread block according to the operation type of each thread block and the processing type of each input feature sequence;

[0161] The running module 1004 is used to run each thread block in parallel to obtain the operation result corresponding to the input feature matrix;

[0162] The output module 1005 is used to obtain the output result of the large model based on the operation results corresponding to each input feature matrix.

[0163] Optionally, the determination module 1002 is specifically configured to:

[0164] Determining, according to a processing type of each input feature sequence in the input feature matrix, the number of first thread blocks whose operation type is a pre-filling operation type and the number of second thread blocks whose operation type is a decoding type;

[0165] At least one thread block corresponding to the input feature matrix, an operation type of each thread block, and a target basic computing unit to which each thread block belongs are determined according to the first thread block number and the second thread block number.

[0166] Optionally, the determination module 1002 is specifically configured to:

[0167] Determine the number of input feature sequences of the pre-filling processing type in the input feature matrix, and calculate the number of first thread blocks of the pre-filling operation type according to the number of input feature sequences of the pre-filling processing type;

[0168] The number of input feature sequences with a processing type of decoding type in the input feature matrix is determined, and the number of second thread blocks with an operation type of decoding operation type is calculated according to the number of input feature sequences with a processing type of decoding type.

[0169] Optionally, the determination module 1002 is specifically configured to:

[0170] Determining, based on the first number of thread blocks and the second number of thread blocks, at least one thread block corresponding to the input feature matrix and a ratio of pre-filled processing types to decoded processing types in the input feature matrix;

[0171] According to the ratio of the pre-filled processing type to the decoded processing type in the input feature matrix, an indicator value corresponding to the input feature matrix is calculated, where the indicator value is used to indicate the sum of two values in the ratio values of the two operation types in the input feature matrix;

[0172] According to the indication value corresponding to the input feature matrix, the target basic computing unit to which each thread block belongs and the operation type of each thread block are determined.

[0173] Optionally, the determination module 1002 is specifically configured to

[0174] Traverse each thread block, and for the current thread block traversed, use the current basic computing unit allocated to it as the target basic computing unit to which the current thread block belongs;

[0175] The number of executed thread blocks of the current basic computing unit is obtained, and the operation type of the current thread block is determined according to the indication value corresponding to the input feature matrix and the number of executed thread blocks.

[0176] Optionally, the allocation module 1003 is specifically configured to

[0177] traverse the first input feature sequence of the pre-filled processing type and the second input feature sequence of the decoding processing type in the input feature matrix respectively;

[0178] For the traversed current first input feature sequence, determine whether there is a thread block of the pre-filled operation type in the current target basic computing unit; if so, assign the current first input feature sequence to the thread block of the pre-filled operation type; otherwise, use the next target basic computing unit of the current target basic computing unit as the new current target basic computing unit and perform thread block assignment;

[0179] For the current second input feature sequence traversed, determine whether there is a thread block of the decoding operation type in the current target basic computing unit. If so, assign the current second input feature sequence to the thread block of the decoding operation type. Otherwise, use the next target basic computing unit of the current target basic computing unit as the new current target basic computing unit and perform thread block allocation.

[0180] Optionally, the running module 1004 is specifically configured to:

[0181] When the current thread block is running, the model weight information is read from the shared storage space corresponding to the target basic computing unit to which the current thread block belongs, and calculations are performed based on the model weight information to obtain the calculation results of the input feature sequence corresponding to the current thread block, wherein the model weight information includes: pre-filled weight values and decoding weight values.

[0182] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0183] The present application also provides an electronic device, such as Figure 11 As shown, Figure 11The electronic device structure diagram provided in the embodiment of the present application includes: a processor 1101, a memory 1102, and optionally, a bus 1103. The memory 1102 stores machine-readable instructions executable by the processor 1101 (for example, Figure 10 The device generates the execution instructions corresponding to the module 1001, the determination module 1002, the allocation module 1003, the operation module 1004 and the output module 1005, etc.), when the electronic device is running, the processor 1101 communicates with the memory 1102 through the bus 1103, and when the machine-readable instructions are executed by the processor 1101, the steps of the data processing method of the above-mentioned large model are executed.

[0184] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned large model data processing method are executed.

[0185] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0186] In addition, the functional units in the various embodiments of the present application can be integrated into a single processing unit, each unit can exist physically separately, or two or more units can be integrated into a single unit. If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0187] The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the protection scope of the present application.

Claims

1. A data processing method for a large model, characterized in that: The method comprises: Generating at least one input feature matrix of the large model according to the prompt word input by the user, wherein each input feature matrix includes at least one input feature sequence; Determining, according to the processing type of each input feature sequence in the input feature matrix, the number of first thread blocks whose operation type is a pre-filling operation type and the number of second thread blocks whose operation type is a decoding type; Determining, based on the first number of thread blocks and the second number of thread blocks, at least one thread block corresponding to the input feature matrix, an operation type of each thread block, and a target basic computing unit to which each thread block belongs, wherein the processing type includes: a pre-filling processing type or a decoding processing type, and the operation type includes: a pre-filling operation type or a decoding operation type; Allocate each input feature sequence to each thread block according to the operation type of each thread block and the processing type of each input feature sequence; Running each thread block in parallel to obtain a calculation result corresponding to the input feature matrix; Based on the calculation results corresponding to each of the input feature matrices, the output result of the large model is obtained.

2. The data processing method of the large model according to claim 1, characterized in that: The determining, according to the processing type of each input feature sequence in the input feature matrix, the number of first thread blocks whose operation type is a pre-filling operation type and the number of second thread blocks whose operation type is a decoding type includes: Determining the number of input feature sequences of the pre-filled processing type in the input feature matrix, and calculating the number of first thread blocks of the pre-filled operation type according to the number of input feature sequences of the pre-filled processing type; The number of input feature sequences with a processing type of decoding type in the input feature matrix is determined, and the number of second thread blocks with an operation type of decoding operation type is calculated according to the number of input feature sequences with a processing type of decoding type.

3. The data processing method of the large model according to claim 1, characterized in that: The determining, based on the first number of thread blocks and the second number of thread blocks, at least one thread block corresponding to the input feature matrix, an operation type of each thread block, and a target basic computing unit to which each thread block belongs includes: Determining, based on the first number of thread blocks and the second number of thread blocks, at least one thread block corresponding to the input feature matrix and a ratio of pre-filled processing types to decoded processing types in the input feature matrix; calculating, based on a ratio of a pre-filled processing type to a decoded processing type in the input feature matrix, an indicator value corresponding to the input feature matrix, the indicator value being used to indicate a sum of two values in a ratio of two operation types in the input feature matrix; According to the indication value corresponding to the input feature matrix, the target basic computing unit to which each thread block belongs and the operation type of each thread block are determined.

4. The data processing method of the large model according to claim 3, characterized in that: The step of determining the target basic computing unit to which each thread block belongs and the operation type of each thread block according to the indication value corresponding to the input feature matrix includes: traversing each of the thread blocks, and for the traversed current thread block, using the current basic computing unit allocated thereto as the target basic computing unit to which the current thread block belongs; The number of executed thread blocks of the current basic computing unit is obtained, and the operation type of the current thread block is determined according to the indication value corresponding to the input feature matrix and the number of executed thread blocks.

5. The data processing method of the large model according to claim 1, characterized in that: The allocating each input feature sequence to each thread block according to the operation type of each thread block and the processing type of each input feature sequence includes: traversing the first input feature sequence of the pre-filled processing type and the second input feature sequence of the decoding processing type in the input feature matrix respectively; For the traversed current first input feature sequence, determine whether there is a thread block of the pre-filled operation type in the current target basic computing unit; if so, assign the current first input feature sequence to the thread block of the pre-filled operation type; otherwise, use the next target basic computing unit of the current target basic computing unit as the new current target basic computing unit and perform thread block assignment; For the traversed current second input feature sequence, determine whether there is a thread block of the decoding operation type in the current target basic computing unit. If so, assign the current second input feature sequence to the thread block of the decoding operation type. Otherwise, use the next target basic computing unit of the current target basic computing unit as the new current target basic computing unit and perform thread block allocation.

6. The data processing method of the large model according to claim 1, characterized in that: The parallel execution of the thread blocks to obtain the operation result corresponding to the input feature matrix includes: When the current thread block is running, the model weight information is read from the shared storage space corresponding to the target basic computing unit to which the current thread block belongs, and calculations are performed based on the model weight information to obtain the calculation results of the input feature sequence corresponding to the current thread block, wherein the model weight information includes: pre-filled weight values and decoding weight values.

7. A data processing device for a large model, characterized in that: The device comprises: A generation module, configured to generate at least one input feature matrix of the large model according to the prompt word input by the user, wherein each input feature matrix includes at least one input feature sequence; a determination module configured to determine, based on a processing type of each input feature sequence in the input feature matrix, the number of first thread blocks whose operation type is a pre-filling operation type and the number of second thread blocks whose operation type is a decoding type; and determine, based on the number of first thread blocks and the number of second thread blocks, at least one thread block corresponding to the input feature matrix, the operation type of each thread block, and a target basic computing unit to which each thread block belongs, wherein the processing type includes: a pre-filling processing type or a decoding processing type, and the operation type includes: a pre-filling operation type or a decoding operation type; an allocation module, configured to allocate each input feature sequence to each thread block according to the operation type of each thread block and the processing type of each input feature sequence; An operation module is used to run each thread block in parallel to obtain a calculation result corresponding to the input feature matrix; The output module is used to obtain the output result of the large model based on the operation results corresponding to each of the input feature matrices.

8. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-readable instructions to perform the steps of the large model data processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the large model data processing method according to any one of claims 1 to 6.

Images