Large model data processing method and device, equipment and medium

Abstract

The invention provides a large model data processing method, device, equipment and medium, which are applied to a chip, the chip comprises a PNM, an SIP and a DMA, a producer-consumer model is arranged between the SIP and the PNM, and the SIP and the PNM dynamically and alternately play a producer and a consumer. According to the invention, the DMA is responsible for loading data from the external memory to the shared cache; in a pre-filling stage and a generation stage of the large model, a calculation task is allocated to a PNM unit and an SIP unit; the PNM and the SIP dynamically and alternately play roles of a producer and a consumer; through the cooperative working mode, the computing resources of the chip are fully utilized, and the processing efficiency and the throughput are improved. Therefore, in the end side application of a large model, cooperative work among the PNM, the SIP and the DMA can be effectively coordinated, the problems of storage bandwidth and capacity are solved, and the reasoning efficiency and performance of the model are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data processing method and device for a large model, equipment and medium. BACKGROUND

[0002] With the development of large models, their parameter quantity and computational complexity are increasing. Large models need to frequently access a large number of parameters (weights, biases, etc.) and data during operation, and the total memory capacity of an end-side device is much smaller than that of a data center server, making it difficult to accommodate all the parameters and data of a large model. Therefore, an effective data processing method for a large model is urgently needed to solve the storage bandwidth and capacity problems faced by large models in end-side applications. SUMMARY

[0003] In view of the above problems, the present application is proposed in order to provide a data processing method, device, equipment and medium for a large model that overcomes the above problems or at least partially solves the above problems.

[0004] To achieve the above object and other related objects, the present application provides a data processing method for a large model, applied to a chip, the chip comprising a PNM, a SIP and a DMA, a producer-consumer model being provided between the SIP and the PNM, and the two dynamically alternately playing the role of producer and consumer, the method comprising:

[0005] The DMA loads model parameters and initial data from an external memory to a shared cache;

[0006] In the pre-filling stage, the PNM obtains a query vector, a key vector and a value vector corresponding to each token of an input sequence from the HBM, and performs data type conversion processing on the key vector and the value vector to obtain a converted key vector, a converted value vector, and places the converted key vector and the converted value vector in the HBM or the shared cache; the SIP performs attention calculation on the query vector corresponding to each token, the converted key vector and the converted value vector to generate an updated context representation of each token; the SIP obtains output layer weights from the HBM, and performs matrix multiplication calculation on the output layer weights and the context representation to obtain an output representation of each token;

[0007] In the generation phase, the PNM obtains a current query vector, a current key vector and a current value vector corresponding to a current input token from the HBM, and performs matrix multiplication calculation on the current query vector and the current key vector to obtain an attention score; the SIP performs rotation position encoding processing on the attention score to obtain an encoded attention score; the PNM performs matrix multiplication calculation on the encoded attention score and the current value vector to obtain a context vector of the current input token; the SIP performs softmax calculation on the context vector of the current input token to obtain a probability weight vector; the SIP obtains an output layer weight from the HBM, and performs matrix multiplication calculation on the output layer weight and the probability weight vector to obtain an inference result of the current input token.

[0008] Optionally, before the step of loading the model parameters and the data from the external storage to the shared cache by the DMA, the method further comprises:

[0009] The intra-loop and extra-loop parameters of the PNM are pre-configured and declared outside the loop.

[0010] The intra-loop and extra-loop parameters of the DMA are pre-configured and declared outside the loop or inside the loop.

[0011] Optionally, before the step of obtaining, by the PNM in the pre-population phase, the query vector, the key vector and the value vector corresponding to each token of the input sequence from the HBM, the method further comprises:

[0012] performing tokenization and vectorization processing on the prompt word input by the user to obtain an input sequence;

[0013] calculating the query vector, the key vector and the value vector corresponding to each token of the input sequence, and transmitting the query vector, the key vector and the value vector corresponding to each token to the HBM.

[0014] Optionally, after the step of obtaining, by the PNM, the query vector, the key vector and the value vector corresponding to each token of the input sequence from the HBM, and performing conversion processing on the key vector and the value vector to obtain a converted key vector and a converted value vector, the method further comprises:

[0015] The PNM checks whether the task queue in the shared cache is full;

[0016] If the task queue is not full, the PNM packs the converted key vector and the converted value vector into a new task and puts the new task into the task queue, so that the SIP takes out the new task from the task queue and uses the converted key vector and the converted value vector carried in the new task to perform subsequent attention calculation.

[0017] Optionally, after the step of the PNM checking whether the task queue in the shared cache is full, the method further comprises:

[0018] If the task queue is full, the PNM performs conditional waiting until the task queue has a vacancy.

[0019] Optionally, after the step of the SIP performing the rotation position encoding processing on the attention score to obtain an encoded attention score, the method further comprises:

[0020] The SIP checks whether the task queue in the shared cache is full.

[0021] If the task queue is not full, the SIP packs the encoded attention score into a new task and puts the new task into the task queue, so that the PNM takes the new task from the task queue and uses the encoded attention score carried in the new task to perform subsequent matrix multiplication calculation.

[0022] Optionally, after the step of the SIP checking whether the task queue in the shared cache is full, the method further comprises:

[0023] If the task queue is full, the SIP performs conditional waiting until the task queue has a vacancy.

[0024] In a second aspect, the present application further provides a data processing apparatus of a large model, applied to a chip, the chip comprising a PNM, a SIP and a DMA, a producer-consumer model being arranged between the SIP and the PNM, and the two dynamically alternately playing the roles of a producer and a consumer, the apparatus comprising:

[0025] A loading module, configured to load model parameters and initial data from an external memory to a shared cache by the DMA;

[0026] A pre-filling module, configured to, in a pre-filling stage, acquire, by the PNM, a query vector, a key vector and a value vector corresponding to each token of an input sequence from an HBM, perform data type conversion processing on the key vector and the value vector to obtain a converted key vector and a converted value vector, and place the converted key vector and the converted value vector in the HBM or the shared cache; perform, by the SIP, attention calculation on the query vector corresponding to each token, the converted key vector and the converted value vector to generate an updated context representation of each token; acquire, by the SIP, an output layer weight from the HBM, and perform matrix multiplication calculation on the output layer weight and the context representation to obtain an output representation of each token;

[0027] The generating module is configured to, in a generating stage, acquire, from the HBM, a current query vector, a current key vector and a current value vector corresponding to a current input token, perform matrix multiplication calculation on the current query vector and the current key vector to obtain an attention score, perform rotation position encoding processing on the attention score by the SIP to obtain an encoded attention score, perform matrix multiplication calculation on the encoded attention score and the current value vector by the PNM to obtain a context vector of the current input token, perform softmax calculation on the context vector of the current input token by the SIP to obtain a probability weight vector, acquire, from the HBM, an output layer weight, and perform matrix multiplication calculation on the output layer weight and the probability weight vector to obtain an inference result of the current input token.

[0028] In a third aspect, the present application provides an electronic device, comprising a memory and a processor; the memory is configured to store a computer program; the processor is configured to execute the computer program stored in the memory, so that the electronic device executes the steps of the data processing method of the large model as described above.

[0029] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by an electronic device to implement the steps of the data processing method of the large model as described above.

[0030] The above one or more technical solutions provided by the present application can have the following advantages or at least achieve the following technical effects:

[0031] The present application is responsible for loading data from an external memory to a shared cache through DMA; in the pre-filling stage and the generating stage of the large model, the computing tasks are allocated to the PNM and the SIP two units; the PNM and the SIP dynamically alternate the roles of producer and consumer; through this cooperative working mode, the computing resources of the chip are fully utilized, and the processing efficiency and throughput are improved. Thus in the end-side application of the large model, the cooperative work among the PNM, the SIP and the DMA can be effectively coordinated, the storage bandwidth and capacity problems can be solved, and the inference efficiency and performance of the model can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 A flowchart of a data processing method of a large model in an embodiment of the present application is shown;

[0033] Figure 2 A computing flowchart during pre-filling in an embodiment of the present application is shown;

[0034] Figure 3 A computing flowchart during generation in an embodiment of the present application is shown;

[0035] Figure 4 A diagram showing a producer-consumer model generating SIP task parameters in an embodiment of the present application;

[0036] Figure 5 A diagram showing a producer-consumer model in which a common thread is the producer in accordance with the present application in an embodiment of the present application; Figure 4

[0037] Figure 6 A diagram showing a producer-consumer model in which a common thread is the producer in accordance with the present application in an embodiment of the present application; Figure 4

[0038] Figure 7 A diagram showing a producer-consumer model generating PNM task parameters in an embodiment of the present application;

[0039] Figure 8 A diagram showing a producer-consumer model in which a common thread is the producer in accordance with the present application in an embodiment of the present application; Figure 7

[0040] Figure 9 A diagram showing a producer-consumer model in which a common thread is the producer in accordance with the present application in an embodiment of the present application; Figure 7

[0041] Figure 10 A diagram showing a function module of a data processing device of a large model in an embodiment of the present application;

[0042] Figure 11 A diagram showing an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The present application is herein described, by way of example only, with reference to certain embodiments thereof. It is to be understood that variations and modifications of the embodiments can be made based on the description set out herein, without departing from the scope of the application. The embodiments and features of the application are combinable where not mutually exclusive. For the avoidance of doubt, the following embodiments and features of the embodiments can be combined with each other where compatible.

[0044] ​​​​It is to be noted that the drawings provided in the following embodiments only schematically illustrate the basic concepts of the present application, and only the components related to the present application are shown in the drawings, rather than being drawn according to the number, shape and size of the components in actual implementation, and the type, number and proportion of each component in actual implementation can be arbitrarily changed, and the component layout type can also be more complex.

[0045] In the following description, numerous specific details are discussed in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to one of ordinary skill in the art that the embodiments of the present application can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid obscuring the embodiments of the present application.

[0046] The terms "first", "second", and the like in the description and the claims of the present disclosure and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0047] Unless otherwise specified, the term "a plurality of" means two or more.

[0048] In the present disclosure, the character " / " represents an "or" relationship between the preceding and following objects. For example, A / B means A or B.

[0049] The term "and / or" is a description of the association relationship between objects, which means that there can be three relationships. For example, A and / or B means that there are three relationships of A or B, or A and B.

[0050] Before the present application is further described, the terms and terminology used in the present application are explained, and the terms and terminology used in the present application are applicable to the following explanations:

[0051] <1> Producer-Consumer Model, used to solve the problem of data transmission and cooperation between multiple threads (or processes). The core idea is to use a "buffer" (or "message queue") as an intermediate medium, and let the "producer" thread be responsible for generating data and putting it into the buffer, and the "consumer" thread be responsible for taking data from the buffer and processing, so as to realize the decoupling and efficient cooperation of production and consumption.

[0052] Producer (Producer): responsible for generating data (such as calculation results, user requests, task instructions, etc.), and putting the data into the buffer. When the buffer is full, the producer needs to wait (block) until there is free space in the buffer before continuing to produce.

[0053] Consumer: responsible for taking data from the buffer and processing it (e.g., parsing requests, executing tasks, storing results, etc.). When the buffer is empty, the consumer needs to wait (block) until the buffer has new data before continuing to consume.

[0054] Buffer: as an "intermediary" between producers and consumers, it is usually a finite-size queue (FIFO) or array. It is used to balance the processing speed of producers and consumers (e.g., when the producer generates data faster than the consumer processes it, the buffer temporarily stores the data; otherwise, the consumer waits), and to decouple the two (no need to interact directly, only communicate through the buffer).

[0055] The technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings in the embodiments of the present application.

[0056] Please refer to Figure 1 An embodiment of the present application provides a data processing method of a large model, applied to a chip, the chip comprising a PNM, a SIP and a DMA, a producer-consumer model is arranged between the SIP and the PNM, and the SIP and the PNM dynamically alternate to play the roles of producers and consumers, and the method can comprise the following steps S10-S30:

[0057] Step S10, the DMA loads model parameters and initial data from an external storage to a shared cache.

[0058] The chip comprises a PNM (Processing Near Memory), a SIP (System in Package) and a DMA (Direct Memory Access).

[0059] The SIP and the PNM adopt a producer-consumer model, and the SIP and the PNM dynamically alternate to play the roles of producers and consumers. This alternating mode reflects the data dependency relationship between different operations in the pipeline: when an operation unit (such as the PNM or the SIP) completes a certain calculation step and generates the data required for the next step, it plays the role of a producer; when another operation unit needs to use this data to perform the next calculation, it plays the role of a consumer.

[0060] For example, in some operations, the PNM is the producer and the SIP is the consumer; while in other operations, the SIP is the producer and the PNM is the consumer.

[0061] DMA (Direct Memory Access), used to load model parameters (such as Q, K, V projection matrix, output layer weight) and data (such as token embedding of input sequence) in external memory (such as main memory) into the shared cache (such as L1) inside the chip before the large model starts execution, providing the basis for subsequent SIP and PNM calculations. The use of DMA reduces the memory access burden.

[0062] Model parameters can refer to parameters required by the large model during runtime (such as pre-filling stage, generation stage), such as output layer weight, bias, and other possible parameters.

[0063] Shared cache can refer to various types of on-chip memory, such as L1 cache; please refer to Figure 4 and Figure 7 L1 in

[0064] In specific implementations, before the large model starts execution, DMA establishes an efficient data transmission path between relatively slow external memory (such as main memory) and high-speed shared cache (such as L1 cache) according to pre-set configuration parameters (such as source address, target address, transmission length, etc.), to batch transfer the required parameters and initial input data of the model to the shared cache, reducing the number of subsequent model calculations (such as matrix multiplication, activation function, etc.) to slow main memory (i.e. reducing the delay of accessing slow storage), significantly improving computing performance.

[0065] Step S20, in the pre-filling stage, PNM obtains the query vector, key vector and value vector corresponding to each token of the input sequence from the HBM, and performs data type conversion processing on the key vector and the value vector to obtain the converted key vector, the converted value vector, and places the converted key vector and the converted value vector in the HBM or the shared cache; SIP performs attention calculation on the query vector corresponding to each token, the converted key vector and the converted value vector to generate the updated context representation of each token; SIP obtains the output layer weight from the HBM and performs matrix multiplication calculation on the output layer weight and the context representation to obtain the output representation of each token.

[0066] HBM (High Bandwidth Memory) is an advanced, stackable DRAM (Dynamic Random Access Memory) technology designed specifically for high-performance computing (HPC) and graphics processing (GPU), which can be used to solve the bandwidth and energy efficiency bottlenecks encountered by traditional memory (such as GDDR). Please refer to Figure 2 and Figure 3 L3 in

[0067] Please refer toFigure 2 Pre-fill stage:

[0068] PNM (Producer) to get Q, K, V vectors corresponding to each token in the input sequence from HBM; and to perform data type conversion processing (such as applying K, V projection matrix) on K and V vectors, to generate converted K and V, and then to write converted K and V into HBM or shared cache.

[0069] SIP (Consumer) to get Q vector, and converted K and V vectors generated by PNM, from HBM or shared cache, and then to perform attention calculation on Q vector, and converted K and V vectors, to generate updated context representation for each token (usually corresponding to the result of Attention(Q, K, V)); and then to write updated context representation for each token into HBM or shared cache.

[0070] SIP (Consumer) to get output layer weight from shared cache, and then to perform matrix multiplication calculation on updated context representation for each token and output layer weight, to get output representation for each token.

[0071] In the pre-fill stage, PNM first plays the role of Producer (processing K, V vectors for each token), and then plays the role of Consumer (processing output of SIP and finally calculating output representation for each token). SIP plays the role of Consumer (processing converted K, V vectors output by PNM), and then its output is consumed by SIP. This embodies the dynamic alternating roles.

[0072] In a specific implementation, in the pre-fill stage, PNM gets query vector (Q), key vector (K) and value vector (V) corresponding to each token in the input sequence from HBM, and then can perform data type conversion processing on key vector and value vector corresponding to each token to get converted key vector and converted value vector, and write them into HBM or shared cache; SIP can get query vector, converted key vector and converted value vector corresponding to each token generated by PNM from HBM or shared cache, and perform attention calculation to generate updated context representation for each token, and write updated context representation for each token into shared cache; SIP gets output layer weight from HBM or shared cache, and performs matrix multiplication calculation on output layer weight and context representation to get output representation for each token, and writes output representation for each token into HBM or shared cache. Thus, through HBM or shared cache, PNM and SIP can efficiently pass and share data, so that data can be quickly accessed and processed by PNM and SIP.

[0073] Step S30, in the generation stage, the PNM obtains the current query vector, the current key vector and the current value vector corresponding to the current input token from the HBM, and performs matrix multiplication calculation on the current query vector and the current key vector to obtain an attention score; the SIP performs rotation position encoding processing on the attention score to obtain an encoded attention score; the PNM performs matrix multiplication calculation on the encoded attention score and the current value vector to obtain a context vector of the current input token; the SIP performs softmax calculation on the context vector of the current input token to obtain a probability weight vector; the SIP obtains the output layer weight from the HBM, and performs matrix multiplication calculation on the output layer weight and the probability weight vector to obtain the inference result of the current input token.

[0074] Wherein, please refer to Figure 3 , the generation stage:

[0075] The PNM (producer) obtains the Q, K and V vectors of the current input token from the HBM, and performs matrix multiplication calculation on the current Q and K vectors to obtain an attention score, which can be written into the HBM or shared cache for subsequent steps.

[0076] The SIP (consumer) obtains the attention score generated by the PNM from the HBM or shared cache, and performs rotation position encoding (RoPE) processing on the attention score to generate an encoded attention score, which can be written into the HBM or shared cache for subsequent steps. At this time, the SIP plays the role of consumer in this step, consumes the output of the PNM, and becomes the producer of the next step.

[0077] The PNM (consumer) obtains the encoded attention score generated by the SIP from the shared cache, and performs matrix multiplication calculation on the encoded attention score and the current value vector to generate a context vector of the current input token, which can be written into the HBM or shared cache. At this time, the PNM plays the role of consumer in this step, consumes the output of the SIP, and becomes the producer of the next step.

[0078] The SIP (consumer) obtains the context vector of the current input token generated by the PNM from the HBM or shared cache, and performs softmax calculation on the context vector of the current input token to generate a probability weight vector, which can be written into the HBM or shared cache. At this time, the SIP plays the role of consumer in this step, consumes the output of the PNM, and becomes the producer of the next step.

[0079] The SIP (consumer) obtains the output layer vector from the HBM, and performs matrix multiplication calculation on the probability weight vector and the output layer weight to generate the inference result of the current input token, and then can write the inference result into the shared cache. At this time, the SIP plays the role of consumer in this step, consumes the output of the SIP, and completes the processing of the current token.

[0080] In the generation phase, the producer-consumer role alternation is more obvious: PNM→SIP→PNM→SIP→SIP. Each step depends on the output of the previous step, and PNM and SIP alternately perform calculation and result delivery.

[0081] In a specific implementation, in the generation phase, the PNM can obtain the current query vector, the current key vector and the current value vector corresponding to the current input token from the HBM, and then can perform matrix multiplication calculation on the current query vector and the current key vector to obtain the attention score, and write it into the HBM or the shared cache. The SIP can obtain the attention score of the current input token from the HBM or the shared cache, and rotate the position encoding processing to obtain the encoded attention score, and write the encoded attention score into the HBM or the shared cache. The PNM obtains the encoded attention score from the HBM or the shared cache, and performs matrix multiplication calculation on the encoded attention score and the current value vector to obtain the context vector of the current input token, and writes the context vector of the current input token into the HBM or the shared cache. The SIP can obtain the context vector of the current input token from the HBM or the shared cache and perform softmax calculation to obtain the probability weight vector. Finally, the SIP obtains the probability weight vector and the output layer weight from the HBM, and performs matrix multiplication calculation on the output layer weight and the probability weight vector to obtain the inference result of the current input token, and writes the inference result of the current input token into the HBM or the shared cache. Thus, through the HBM or the shared cache, the PNM and the SIP can efficiently deliver and share data, so that the data can be quickly accessed and processed by the PNM and the SIP.

[0082] Further, in an embodiment, before step S10, the method can further include S40:

[0083] Step S40, pre-configure the loop-in and loop-out parameters of the PNM, and declare outside the loop;

[0084] Pre-configure the loop-in and loop-out parameters of the DMA, and declare outside the loop or declare inside the loop.

[0085] The configuration parameters for DMA can include: channel address (specifying the source and destination addresses of DMA transfer), priority (setting the priority of DMA transfer to ensure that high-priority transfers are executed first), data transfer direction (specifying data transfer from peripheral to memory or from memory to peripheral), loop mode (configuring whether DMA automatically restarts after the transfer is completed), and data transfer amount (setting the amount of data transferred each time).

[0086] PNM configuration parameters can include: number of loops, loop conditions (setting the termination conditions of the loop, such as data processing completion or reaching a specific number of iterations), data buffer size, and data processing logic (defining the specific data processing logic of PNM inside the loop, such as data transformation and matrix operations).

[0087] In the specific implementation, before the DMA loads data from external storage into the shared cache, the SIP can initiate the configuration of the loop-inside and loop-outside parameters of PNM and DMA. The SIP pre-configures the DMA loop-inside and loop-outside parameters once, declaring them either outside or inside the loop. These parameters can include channel address, priority, data transfer direction, loop mode, and data transfer amount. After configuration, the SIP can trigger the DMA to start executing the data transfer task within the loop via a specific command or interface (e.g., the DMA loads model parameters and initial data from external storage into the shared cache according to the pre-configured parameters). Similarly, the SIP pre-configures the PNM loop-inside and loop-outside parameters once, declaring them outside the loop. These parameters can include loop count, loop condition, data buffer size, and data processing logic. After configuration, the SIP can trigger the PNM to start executing the data processing task within the loop via a specific command or interface (e.g., the PNM retrieves data from the shared cache and processes it according to the pre-configured parameters). Through the parameter configuration and triggering mechanism initiated by the SIP, the work of DMA and PNM can be fully scheduled and optimized, thereby adapting to different data processing needs and improving the inference effect and performance of large models.

[0088] Furthermore, in one embodiment, before step S20, the method may further include steps S50 to S60:

[0089] Step S50: The prompt words input by the user are segmented and vectorized to obtain the input sequence;

[0090] Step S60: For each token in the input sequence, calculate its corresponding query vector, key vector, and value vector, and transmit the query vector, key vector, and value vector corresponding to each token to HBM.

[0091] The prompt words can refer to the text entered by the user into the large model, which is used to guide the large model to generate a specific type of response.

[0092] The input sequence refers to the sequence of user input, which is usually formed after tokenization and vectorization. This sequence is typically composed of a series of tokens (such as a word, a character, or other forms of text fragments).

[0093] In the specific implementation, after receiving the prompt words input by the user, a word segmenter can be used to segment the prompt words, and the segmented prompt words can be vectorized (e.g., embedded) to obtain the input sequence. For each token in the input sequence, its corresponding query vector, key vector, and value vector (linear transformation matrix) are calculated, and the calculated query vector, key vector, and value vector corresponding to each token are transmitted to HBM so that subsequent PNM and SIP can access and use this data. Thus, the large model can effectively process the prompt words input by the user and generate the corresponding output.

[0094] In this embodiment, DMA is responsible for loading data from external memory into the shared cache. During the pre-filling and generation phases of the large model, computational tasks are assigned to both the PNM and SIP units. The PNM and SIP dynamically alternate between acting as producer and consumer. This collaborative approach fully utilizes the chip's computing resources, improving processing efficiency and throughput. Therefore, in edge applications of large models, the collaborative work between the PNM, SIP, and DMA can be effectively coordinated, resolving storage bandwidth and capacity issues and improving model inference efficiency and performance.

[0095] Based on the foregoing embodiments, a second embodiment of the data processing method for the large model of the present invention is proposed. In this embodiment, after step S20, in which PNM obtains the query vector, key vector, and value vector corresponding to each token of the input sequence from HBM, and performs transformation processing on the key vector and the value vector to obtain the transformed key vector and the transformed value vector, the following steps A10 to A20 may be included:

[0096] Step A10: PNM checks whether the task queue in the shared cache is full;

[0097] The shared cache serves as a buffer between PNM and SIP, storing tasks to be processed (e.g., containing the transformed key vector and the transformed value vector).

[0098] Please see Figures 4-6 , Figure 4 It demonstrates the relationship between scalar threads (producers) and common threads (consumers), as well as the management of task queues; Figure 5The scalar thread is described in detail as a producer in a loop, including checking if the task queue is full (checktask:CFGSIP full) and generating new tasks (gen task:CFGSIPparams). Figure 6 It describes the loop operations of the common thread as a consumer, including getting tasks (get task: CFGSIP params) and running tasks (run task: attention).

[0099] from Figures 4-6 As can be seen, PNM is the producer, which is responsible for generating tasks and putting them into the task queue; SIP is the consumer, which is responsible for retrieving tasks from the task queue and executing them.

[0100] In the specific implementation, during the pre-filling stage, PNM (acting as the producer at this time) transforms the key vector and value vector to obtain the transformed key vector and value vector, and writes them into the shared cache. Then, before PNM generates a new task (such as gen task:CFGSIP params), it also checks whether the current task queue in the shared cache is full.

[0101] Step A20: If the task queue is not full, PNM packages the transformed key vector and the transformed value vector into a new task and puts it into the task queue, so that SIP can retrieve the new task from the task queue and use the transformed key vector and the transformed value vector carried in the new task to perform subsequent attention calculations.

[0102] In practical implementation, when the current task queue is not full, PNM can package the transformed key vector and value vector into a new task and put it into the task queue. Then, SIP can retrieve the new task from the shared cache task queue and use the transformed key vector and value vector carried in the new task to perform subsequent attention calculations. Thus, through the task queue, PNM and SIP in the producer-consumer model can work in parallel to a certain extent. PNM can prepare the next task while SIP processes the previous task, effectively balancing the processing speed between PNM and SIP and avoiding PNM wasting computing resources waiting for queue space.

[0103] Furthermore, in one embodiment, after step A10, the following step A30 may also be included:

[0104] Step A30: If the task queue is full, PNM will conditionally wait until there is space in the task queue.

[0105] In its implementation, when the current task queue is full, PNM conditionally waits until an empty slot becomes available in the queue. This avoids wasting computational resources.

[0106] In this embodiment, PNM checks whether the task queue in the shared cache is full. If the task queue is not full, PNM packages the transformed key vector and the transformed value vector into a new task and puts it into the task queue. SIP then retrieves the new task from the task queue and uses the transformed key vector and transformed value vector carried in the new task to perform subsequent attention calculations. Thus, through the task queue, an efficient and balanced collaborative relationship can be established between PNM and SIP, effectively balancing the processing speed between PNM and SIP and preventing PNM from wasting computing resources waiting for queue space.

[0107] Based on the foregoing embodiments, a third embodiment of the data processing method for the large model of the present invention is proposed. In this embodiment, after step S20, in which SIP performs rotational position encoding processing on the attention score to obtain the encoded attention score, the following steps B10 to B20 may be further included:

[0108] Step B10: SIP checks whether the task queue in the shared cache is full.

[0109] Please refer to Figures 7-9 , Figure 7 This demonstrates the relationship between the scalar thread and the common thread. The scalar thread (producer) is responsible for producing tasks, while the common thread is responsible for consuming tasks. Tasks are placed in the states: valid slots, and specific tasks include CGFPNMA params, CGFPNMB params, and CGFPNMC params. Figure 8 The scalar thread is described in detail as a producer in a loop, including checking if the task queue is full (check task:CFGPNMA full) and generating new tasks (gen task:CFGPNMAparams). Figure 9 It describes the loop operations of the common thread as a consumer, including waiting for a task (CFGPNA params), running a task (PNMAwithparams), getting a task (CFGPNMAparams), conditional waiting (CFGPNC params), and attention.

[0110] from Figures 7-9 As can be seen, SIP is the producer, responsible for generating tasks and putting them into the task queue; PNM is the consumer, responsible for retrieving tasks from the task queue and executing them.

[0111] In the specific implementation, SIP performs rotational position encoding on the attention score to obtain the encoded attention score, and writes it into the shared buffer (at this time, SIP acts as the producer). Then, before SIP generates new tasks (such as CFPPNMAparams, CFPPNMB params, CFPPNMC params), it also checks whether the corresponding task queue (such as CFPPNMAparams, CFPPNMB params, CFPPNMC params) is full.

[0112] In step B20, if the task queue is not full, SIP packages the encoded attention score into a new task and puts it into the task queue so that PNM can retrieve the new task from the task queue and use the encoded attention score carried in the new task to perform subsequent matrix multiplication calculations.

[0113] In its implementation, if the task queue is not full, SIP packages the encoded attention score into a new task and adds it to the task queue. PNM can then retrieve the new task from the queue and use the encoded attention score carried in the new task to perform subsequent matrix multiplication calculations (e.g., run task:PNMAwith params). This ensures the orderly generation and consumption of tasks, avoids task queue overflow, and guarantees efficient computation.

[0114] Furthermore, in one embodiment, after step B10, the following step B30 may also be included:

[0115] In step B30, if the task queue is full, SIP will conditionally wait until there is space in the task queue.

[0116] In this embodiment, if the task queue is full, SIP will conditionally wait until there is space in the task queue, thus avoiding the waste of computing resources.

[0117] In this embodiment, SIP checks whether the task queue in the shared cache is full. If the task queue is not full, SIP packages the encoded attention score into a new task and puts it into the task queue, so that PNM can retrieve the new task from the task queue and use the encoded attention score carried in the new task to perform subsequent matrix multiplication calculations. This ensures the orderly generation and consumption of tasks, avoids task queue overflow, and ensures efficient computation.

[0118] Based on the same inventive concept, the fourth embodiment of the present invention also provides a large-model data processing apparatus corresponding to the large-model data processing method of the foregoing embodiments. Since the principle by which the apparatus in the fourth embodiment solves the problem is similar to the large-model data processing method of the foregoing embodiments, the implementation of the apparatus can be referred to the implementation of the method, and repeated details will not be elaborated further. Please refer to... Figure 10 This invention discloses a large-scale data processing device for a chip, which includes a PNM (Producer-Consumer Model), a SIP (System-in-Package), and a DMA (Distribution-Directed Memory). A producer-consumer model is established between the SIP and the PNM, and the two dynamically alternate between acting as producers and consumers. The device may include:

[0119] Loading module 10 is used for DMA to load model parameters and initial data from external memory into a shared cache;

[0120] The pre-filling module 20 is used in the pre-filling stage to: PNM obtains the query vector, key vector, and value vector corresponding to each token of the input sequence from HBM; performs data type conversion on the key vector and the value vector to obtain the converted key vector and the converted value vector; and places the converted key vector and the converted value vector in HBM or a shared cache. SIP performs attention calculation on the query vector, the converted key vector, and the converted value vector corresponding to each token to generate an updated context representation for each token. SIP obtains the output layer weights from HBM and performs matrix multiplication calculation on the output layer weights and the context representation to obtain the output representation for each token.

[0121] The generation module 30 is configured to, during the generation phase, have the PNM obtain the current query vector, current key vector, and current value vector corresponding to the current input token from the HBM, and perform matrix multiplication on the current query vector and the current key vector to obtain an attention score; the SIP performs rotational position encoding on the attention score to obtain an encoded attention score; the PNM performs matrix multiplication on the encoded attention score and the current value vector to obtain a context vector of the current input token; the SIP performs softmax calculation on the context vector of the current input token to obtain a probability weight vector; and the SIP obtains the output layer weights from the HBM and performs matrix multiplication on the output layer weights and the probability weight vector to obtain the inference result of the current input token.

[0122] In addition, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described data processing method for large models.

[0123] Figure 11 This is a schematic block diagram of the electronic device provided in an embodiment of this application. Figure 11 As shown, the electronic device includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the electronic device are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 11 The general will label all buses as bus systems.

[0124] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0125] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0126] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the electronic device 400. Examples of this data include: any executable program for operation on the electronic device 400, such as the operating system 4021 and application programs 4022; the operating system 4021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 4022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The data processing method for the large model provided in this embodiment of the invention can be included in the application program 4022.

[0127] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the large-scale model data processing method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0128] In an exemplary embodiment, the electronic device 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0129] In summary, this invention utilizes DMA to load data from external memory into the shared cache; during the pre-filling and generation phases of large models, computational tasks are allocated to both the PNM and SIP units; the PNM and SIP dynamically alternate between acting as producer and consumer; this collaborative approach fully utilizes the chip's computational resources, improving processing efficiency and throughput. Therefore, in edge applications of large models, it effectively coordinates the collaborative work between the PNM, SIP, and DMA, resolving storage bandwidth and capacity issues, and improving model inference efficiency and performance.

[0130] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A data processing method for a large model, characterized in that, Applied to a chip, the chip including a PNM, a SIP, and a DMA, wherein a producer-consumer model is established between the SIP and the PNM, and the two dynamically alternate between acting as producers and consumers, the method includes: DMA loads model parameters and initial data from external memory into a shared cache; In the pre-filling phase, PNM obtains the query vector, key vector, and value vector corresponding to each token of the input sequence from HBM, and performs data type conversion on the key vector and the value vector to obtain the converted key vector and the converted value vector, and places the converted key vector and the converted value vector in HBM or the shared cache; SIP performs attention calculation on the query vector, the converted key vector, and the converted value vector corresponding to each token to generate the updated context representation of each token; SIP obtains the output layer weights from HBM, and performs matrix multiplication calculation on the output layer weights and the context representation to obtain the output representation of each token; During the generation phase, PNM obtains the current query vector, current key vector, and current value vector corresponding to the current input token from HBM, and performs matrix multiplication on the current query vector and the current key vector to obtain an attention score; SIP performs rotational position encoding on the attention score to obtain an encoded attention score; PNM performs matrix multiplication on the encoded attention score and the current value vector to obtain the context vector of the current input token; SIP performs softmax calculation on the context vector of the current input token to obtain a probability weight vector; SIP obtains the output layer weights from HBM, and performs matrix multiplication on the output layer weights and the probability weight vector to obtain the inference result of the current input token.

2. The method according to claim 1, characterized in that, Prior to the step of loading model parameters and data from external storage into the shared cache by the DMA, the method further includes: Pre-configure the loop parameters of PNM and declare them outside the loop; Pre-configure the loop parameters for DMA, and declare them either outside or inside the loop.

3. The method according to claim 1, characterized in that, Before the step in the pre-filling stage where PNM obtains the query vector, key vector, and value vector corresponding to each token of the input sequence from HBM, the method further includes: The prompts input by the user are segmented and vectorized to obtain the input sequence; For each token in the input sequence, calculate its corresponding query vector, key vector, and value vector, and transmit the query vector, key vector, and value vector corresponding to each token to HBM.

4. The method according to claim 1, characterized in that, After the step of obtaining the query vector, key vector, and value vector corresponding to each token of the input sequence from the HBM, and transforming the key vector and the value vector to obtain the transformed key vector and the transformed value vector, the method further includes: PNM checks whether the task queue in the shared cache is full; If the task queue is not full, PNM packages the transformed key vector and the transformed value vector into a new task and puts it into the task queue, so that SIP can retrieve the new task from the task queue and use the transformed key vector and the transformed value vector carried in the new task to perform subsequent attention calculations.

5. The method according to claim 4, characterized in that, After the step of PNM checking whether the task queue in the shared cache is full, the method further includes: If the task queue is full, PNM will conditionally wait until there is space in the task queue.

6. The method according to claim 1, characterized in that, After the step of SIP performing rotational position encoding on the attention score to obtain the encoded attention score, the method further includes: SIP checks whether the task queue in the shared cache is full; If the task queue is not full, SIP packages the encoded attention score into a new task and puts it into the task queue, so that PNM can retrieve the new task from the task queue and use the encoded attention score carried in the new task to perform subsequent matrix multiplication calculations.

7. The method according to claim 6, characterized in that, After the step of SIP checking whether the task queue in the shared cache is full, it further includes: If the task queue is full, SIP will conditionally wait until there is space in the task queue.

8. A data processing device for a large model, characterized in that, The device is applied to a chip, which includes a PNM (Producer-Consumer Module), a SIP (System-in-Package), and a DMA (Distribution-Directed Memory). A producer-consumer model is established between the SIP and the PNM. The PNM processes data using a single thread, while the SIP dynamically partitions and processes data using multiple threads. The device includes: The loading module is used for DMA to load model parameters and initial data from external memory into the shared cache; The pre-filling module is used in the pre-filling stage to: PNM obtains the query vector, key vector, and value vector corresponding to each token of the input sequence from HBM; performs data type conversion on the key vector and the value vector to obtain the converted key vector and the converted value vector; and places the converted key vector and the converted value vector in HBM or a shared cache. SIP performs attention calculation on the query vector, the converted key vector, and the converted value vector corresponding to each token to generate an updated context representation for each token. SIP obtains the output layer weights from HBM and performs matrix multiplication calculation on the output layer weights and the context representation to obtain the output representation for each token. The generation module is used in the generation phase to: PNM obtains the current query vector, current key vector, and current value vector corresponding to the current input token from HBM, and performs matrix multiplication on the current query vector and the current key vector to obtain an attention score; SIP performs rotational position encoding on the attention score to obtain an encoded attention score; PNM performs matrix multiplication on the encoded attention score and the current value vector to obtain a context vector of the current input token; SIP performs softmax calculation on the context vector of the current input token to obtain a probability weight vector; and SIP obtains the output layer weights from HBM and performs matrix multiplication on the output layer weights and the probability weight vector to obtain the inference result of the current input token.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the processor to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed, performs the steps of the method according to any one of claims 1 to 7.

Images